Test management provides several challenges to Quality Assurance teams. As organizations create large volumes of automated and manual tests, they often struggle to organize, categorize and prioritize them so that they can be fully utilized in an efficient way. The problem is exacerbated when organizations choose test development and test management tools that are not well integrated. AscentialTest provides an alternative; a single solution where test planning and test management are central to the test development process. Organizing tests within test plans not only assures a better perspective on test coverage but it also significantly reduces the effort to track and manage available tests. This paper provides a detailed description of the features and benefits of the AscentialTest approach to test management.

Solution Details:

It’s not uncommon for a large testing project to contain upwards of ten thousand tests, but even projects with only several hundred can be difficult to manage. One of the challenges with managing a large number of tests is distinguishing one test from the next, even when good naming conventions are in place. It’s also difficult to understand whether a set of tests provides adequate coverage. By placing the test plan at the center of the test development process, AscentialTest avoids the problems associated with traditional decentralized solutions.

The Test Plan

AscentialTest provides a specialized editor, designed for writing outline-based test plans. Documenting test requirements in outline form provides an immediate advantage that is easy to overlook. Compared to simply compiling a list of tests, the outline form provides a graphical view of test requirements, making it much easier to determine the level of test coverage that a test plan provides. Let’s look at a very simple example:

Figure 1

The test plan on the left of Figure 1 displays a simple list of tests. Notice how it repeats information like the words ‘Missing’ and ‘Required Field’. It’s not easy to figure out what is being covered without studying it. The test plan on the right is structured. A quick look informs the reader that there are three types of tests, ‘Required Fields’, ‘Confirmed Fields’ and ‘Formatted Fields’. During a peer review, it’s easy to surmise that one of the first questions that will be discussed is whether all of the required, confirmed and formatted fields are covered. Reviewing a plan written graphically encourages the test plan writer and the reviewer alike to drill down deeper to ensure comprehensive coverage. In a simple example like the one presented above, the difference in the two planning approaches may be subtle, but if you imagine a more complex test problem, the advantage of the outline-based format will quickly become evident.

Whether or not you subscribe to the outline-based format, test plans provide a way to organize tests in AscentialTest. In fact, the Test Plan Editor supports documenting test requirements in list, outline or scenario form. Regardless of format, the test plan becomes a container to store tests that are related to one another. Let’s say that a project contains a thousand tests, where tests are grouped into ten different plans based on the application module that they are written to address. If automated and/or manual tests are developed with TestToolA and then managed with a TestManagementToolB, there are several large and costly administrative tasks that must be completed to organize these thousand tests. Furthermore, once the tests are stored in TestManagementToolB, it is difficult to locate them at execution time. On the other hand, if tests are associated with test plans as they are developed, it is relatively easy to manage ten test plans where related tests are already co-located.

Figure 2

Adding Details

Unlike scenarios, where the details of a test requirement are documented in the test plan itself, the description provided in lists and outlines tends to be brief in order to keep the format clean and easy to read. The Description tab provides space for the details. Figure 3 below provides an example. The user has selected the line ‘Missing Required Fields’ in the Test Plan and then selected the Description tab.

Figure 3

The Description tab provides a rich edit field, where the user can configure the font, style, size, color, highlight and justification of the text that describes the test requirements in as much detail as required.

Locating Tests

From the list of available tests in the Project Explorer located on the left side of Figure 4 below, the user drags tests indicated by a green for automated tests and a blue for manual tests to the associated test requirement or group of requirements in the test plan. Dragging tests from the Project Explorer to the test plan makes an association (link) between the test requirements and the tests that fulfill or satisfy those requirements. The Project Explorer provides three ‘views’ to organize tests: alphabetical, by file, or by a user-defined ‘group’ to make it easy to locate them.

Figure 4

One of the challenges that organizations face is making full usage of their test assets. It can be difficult to locate tests especially on large projects where tests may number in the thousands. AscentialTest provides Test Attributes to locate tests based on an unlimited number of user defined ‘tags’ that provide a way to categorize tests.

To add attributes to a test, the user locates the Attributes section in the header of the Test Editor. Figure 5 below displays four attributes: EditType, Priority, TestLevel and TestType. The user can create an unlimited number of attributes in their AscentialTest project.

Figure 5

Attributes are input by either selecting or inputting values in the attribute fields.

To locate tests based on those attributes, the user right-clicks on the Test section in the Project Explorer and selects Query Tests:

 

Figure 6

The ‘Query Tests’ dialog displays:

 

Figure 7

While it is possible to input the query directly, it is easier to use the ‘Query Builder’, which is invoked by clicking the Edit button.

 

Figure 8

 

The user then constructs a query by selecting values from dropdowns and inputting parameters:

Figure 9

The query field displays the resulting query:

Figure 10

The query returns the names of all the tests that match the query:

Figure 11

Now that the tests have been located, they can be dragged from the listing to a Test Plan.

Test Plan Attributes

AscentialTest provides Test Plan Attributes to label tests within Test Plans so that they can be easily located and ‘marked’ for execution. Any number of user-defined attributes can be defined and applied to test requirements. At test execution time, the user submits a query against attributes to select tests to be run. For example, if a subset of tests is labeled with the attribute value ‘Smoke Test’, those tests that make up a smoke test can be marked with a single query. When combined with outline-based test plans, attributes can be applied to groups of tests with minimal effort by taking advantage of inheritance built into the outline model.

Figure 12

Figure 12 above displays a test plan that utilizes two attributes: ‘TestLevel’ and ‘TestType’. The text displayed in black indicates the fields where the user made a selection. The text displayed in blue indicates that the attribute value has been inherited from its parent. In this example, the user selected ‘Regression’ in the ‘TestLevel’ column and then overrode that value for tests to be included in a smoke test.

Factoring

Factoring is inherent in the structure of an outline-based test plan. It provides a way to reduce repetition and share what is common. Look back at Figure 1 and notice how the notation of the test requirements in the test plan on the right is more concise than that of the list-based plan on the left. The briefer, factored plan is easier to understand because the ‘noise’ from the repetition has been removed.

Likewise, test plan factoring allows tests and test data to be reused. In Figure 13 below, the test plan segment displays a group of tests that verify that an error message is displayed if a required field is omitted.

Figure 13

The icon indicates that an automated test has been linked to the group of tests parented by the node ‘Missing Required Fields’. If the test was a manual test, the icon would have been displayed. The seven tests in the group share a single re-usable test. The icon indicates that the test requirement is linked to an automated test that has been inherited from a parent node.

In this example, inheritance is used to factor both the automated test procedure and the test data. In Figure 13, the parent node, ‘Missing Required Fields’, is selected. The test data is input below in the associated ‘Test’ tab into fields which were automatically generated when the test procedure was linked to the test plan. Notice that data has been input to all of the fields. In Figure 14 below, the ‘First Name’ test requirement is selected. Note how the data displays in bright blue, indicating that the data has been inherited from a parent node. Since the ‘First Name’ field is intended to be omitted in this test, it has been overridden so that no value will be input to the field when the test is executed.

Figure 14

Likewise, in Figure 15, the ‘Last Name’ test requirement is selected. Since the intention of this test is to omit, the ‘Last Name’, the field has been overridden so that no value will be input when the test is executed.

Figure 15

Factoring not only saves time and effort to create manual and automated tests, it also significantly reduces the amount of maintenance required to keep tests up to date as the application under test changes.

Execution

AscentialTest provides an efficient way to group test plans for execution. A suite is an execution file that lists test plans to be run. To include a plan in a suite, it is simply dragged and dropped from the Project Explorer. Suites can be either ad hoc or permanent project assets. If after a suite is created, a test plan is changed, the suite is automatically updated with no user intervention required.

Reporting

Test Sets provide a way to manage the execution of tests for a single or multiple AscentialTest projects. A Test Set may be created for a build, a test cycle or a project, depending upon how your organization tests and delivers software. Although a Test Set can be used indefinitely, it is more likely that the scope will be for a software release, upon which the Test Set will be completed and archived for future reference. A Test Set is created by combining tests from available sources. Sources include test plans and suites from a single or multiple AscentialTest projects.

Each Test Set contains an Overview which provides a real-time status:

 

Figure 16

The Sources Panel provides a break-down of the overall status for each source:

Figure 17

Both the Overview and Sources Panel are updated in real-time as tests are executed. The following is a short description of the primary test management features:

• Select tests for inclusion in the Test Set using an attribute query
• Combine automated and manual tests within a Test Set
• Indicate and specify reasons for blocked tests
• Add notes to tests
• Select tests for execution using an attribute query
• Specify reasons for test failure
• Annotate test set to explain test results
• Add attachments to Test Sets for review or audit purposes
• Ignore runs or specific tests within a run
• View the results of each run
• View the results of a test across multiple runs
• Generate reports and charts
• Export reports to PDF, HTML, EXCEL, RTF, TXT or TIF formats

Below are samples of reports that can be generated for a Test Set:

Figure 18

Figure 19

In addition to standard reports, users can design and generate custom reports.

Test Sets can be run on a single local or remote machine or on a group of machines. The multiple-machine controller distributes tests to available machines to increase throughput. A customer recently shared their runtime statistics with us. Using a farm of virtual machines, they were able to reduce their testing cycle from 3 staff members over a two week period to one staff person over two days.

Benefits Summary:

AscentialTest allows customers to increase test coverage while reducing costs and the level of effort necessary to manage test assets. The outline-based format not only promotes better test planning, it also enables factoring to increase productivity when creating and maintaining tests. By tightly integrating Test Sets with automated and manual tests, the effort to manage test assets is significantly reduced. Test Sets and Attributes combine to provide a simple and effective way to locate and select tests at execution time with a single query. There is no need for expensive administration. Finally, the ability to distribute tests across multiple machines dramatically increases throughput and reduces test cycles.

Zeenyx continues to enhance the test management features of AscentialTest. The next release will provide a way to track and report on the test development effort for both manual and automated tests.

For more information on AscentialTest visit Zeenyx on the web at www.zeenyx.com.