Table Of Contents

First Rule of Frameworks: Don’t Talk About Frameworks

The topic of Test Automation Frameworks gets broad-brushed by testing solution companies because they know that there is a significant level of effort involved in building them and that translates to time and cost for the user organization, reducing the return on investment (ROI). Test automation frameworks take a long time to develop – weeks, even months and require specialized skills.

As one of our competitors wrote: “Building a framework for UI test automation is complex, time-consuming, and often one of the most challenging parts of the SDLC.” 

At Zeenyx (home of AscentialTest), we couldn’t disagree with this statement more. Time should be focused on building tests that validate the target application, not on complex test frameworks. Read on, and see how a better tool can build your frameworks for you.

What is a Test Automation Framework

Broadly speaking, a test automation framework is comprised of the components that support the development and execution of automated software tests. Open source and commercial testing solutions are based on a paradigm for generating tests, either a linear model like record/playback or a modular structure that involves scripting in languages like python, javascript, ruby or C#. Script-based solutions are often data-driven and/or keyword-driven.

Linear Frameworks

Linear frameworks based on record/playback look attractive to the uninitiated novice. Turning on a recorder to generate a test seems like a good choice because all the user has to do is execute the test manually while the recorder watches and records the actions. Of course, the problem becomes evident over time as the target application changes. Then all the recordings are broken and the amount of work to get them working again is pretty much the same as it took to record them in the first place.

Script-based Solutions

Script-based solutions solve the test maintenance problem by modularizing to varying degrees the actions that make up a test. The data-driven approach goes further to separate the data from the actions to promote reuse. The keyword-driven approach is a variant that was designed to allow testers who do not have scripting skills to build tests by organizing a list of keywords that map to functions written by scripters. The level of abstraction might seem like the right idea for at least some organizations where the two different skill sets exist, but practically speaking it turns into a mess. The approach isn’t scalable. Managing the multitude of keywords in spreadsheets is time-consuming and complex and when a test fails, debugging it is nearly impossible.

Other components of the test framework include test setup, test data management, error handling, test cleanup, logging and reporting and integration with test planning.

The Costs of Building a Custom Test Automation Framework

Does a Single Engineer Hold The Keys to Your Custom Framework?

It is true, test frameworks take a long time to develop – weeks, even months before the first automated test can be created and, without a lot of technical management oversight, the framework is apt to be over-engineered. A test framework comes at a high price because it takes a higher skill level to create it. The organization then becomes dependent upon one associate or consultant to maintain the framework and the rest of the team has to mount a learning curve that is higher than it needs to be.

Does Your Tool Create Your Framework Automatically?

Selenium Provides No Framework Support

When choosing a testing solution, look for degree of test framework support that is provided. Open source tools like Selenium, while free of charge, provide nothing. You get what you pay for so be ready to spend on building the framework from scratch or cobbling together pieces from the online user base. The price of a paid-for solution will easily cover the development costs incurred. Most commercial tools provide some of the components of a test framework like logging and reporting and integration with version control, defect tracking and continuous integration but many require a lot of coding to build test setup, test data management, error handling and test recovery/cleanup.

Commercial Tools Aren’t So Easy Either

The cost of ownership of the testing solution needs consideration over the life of the application under test (AUT). Linear-based solutions like Ranorex and Eggplant have a high cost of maintenance because tests will likely break with every change cycle. Self-healing from AI sounds great but those technologies are in their infancy, many years away from being truly effective. Script-based solutions like Test Complete and Tosca produce more robust tests but require an expensive skill set to build.

AscentialTest: Frameworks That Build Themselves

AscentialTest is unique in that the components of the automated testing framework are either pre-built or generated automatically using a combination of wizards, information available from the window manager and drag & drop. Users don’t need to build any of the framework components. They’re all provided.

Our founders spent years as test automation engineers. Their experience in the trenches taught them about the value of applying concepts like abstraction and encapsulation to our testing solution and its built-in test framework including:

• Test class
• Base states
• Object class library
• Step-based tests
• Test plans and test management
• Test data management
• Logging and reporting

The Test Class

While invisible to the user, every test created in AscentialTest derives and inherits from a class called ‘Test’ as displayed in this image:

Here is a brief explanation: When a test is executed in AscentialTest, the ‘Run’ method is called. The ‘Run’ method starts a timer and then calls another built-in method called ‘OnStart’, which gets configured by information captured from the AUT and a wizard that dialogs with the user. Next the ‘Main’ gets called. The ‘Main’ includes all the steps and actions that make up the body of the test. If an exception is raised during the ‘Main’, another method called ‘OnException’ gets called. Its job is to log the exception. Finally ‘OnFinish’ gets called to return the application to the starting state which we refer to as the ‘base state’, typically the state of the application when it is first invoked, after any login has been completed.

Why This Matters

Building an infrastructure like our Test Class from scratch is an expensive endeavor. There is a lot of internal code that makes this engine run smoothly. Creating an infrastructure like this would take a highly skilled test engineer months to build. Why reinvent the wheel when AscentialTest provides an out of the box solution.

App States

To ensure that tests run reliably, they should be designed to run independent of each other. Each test should start and stop at the ‘base state’ as defined above.

For web-based applications running in a browser or terminal-based applications running in a terminal emulator, no configuration is required to ensure that each test will start at its ‘base state’. To achieve the ‘base state’ for other application types, AscentialTest provides a wizard to capture information from the user to configure the ‘On Start’ that was described above.

A well-designed test includes actions that return the target application to the ‘base state’, which usually entails closing dialogs that were opened during the test. If a test fails before reaching those actions, the ‘OnFinish’ will recover the application to its base state.

Why This Matters

We designed a system of ‘app states’ into the AscentialTest framework to prevent what is often called the ‘domino effect’, where the failure of a single test cascades, impacting the execution of all subsequent tests. Losing an entire test run can waste valuable time and negatively impact testing cycles and target dates.

Object Class Library

For all of our supported platforms, a class library supports the generation of object-actions that make up a large proportion of a test. Examples of object classes include ListBox, PopupList, PushButton, RadioGroup, Table/Grid, TextField and TreeView. The class library exposes the attributes and behaviors of each object type in the form of properties and methods that are in turn used to drive and verify the AUT. AscentialTest provides a codeless process for adding actions to tests using an ‘Actions Panel’ where users set values, verify attributes and interact with application objects:

Why This Matters

The class library simplifies the process of generating automated tests and increases productivity by providing a codeless mechanism for building tests using drag and drop. The user does not need to be concerned about the way that the various GUI element types behave. By reducing the list of test actions to Set, Verify, Get, Events and Other, we’ve reduced the learning curve and the amount of time that it takes to generate tests.

Step-based Tests

The step-based paradigm is the next evolution in automated test development. Where keyword-driven testing formed a loose relationship between tests, keywords and test functions, the step-based approach builds tight links between application objects, actions and test data. It is from those relationships, that AscentialTest can generate the test framework automatically.

The basic building block of the step-based paradigm is the step. A step is a logical unit of a test. It is comprised of actions that are executed upon application objects. In the keyword-driven approach, users are required to either record or script the actions which make up tests. In the step-based approach, actions are automatically generated when the user selects an object-action from a list available for the object class.

Steps are designed to be reusable. Each step in a test project is designed for a specific purpose and no other step repeats the actions that it contains. For example, there is only one step defined for ‘User Login’. Any test that requires a ‘User Login’ will include that step

Why This Matters:

As the inventory of re-usable steps increases, the process of building tests becomes one of simply arranging steps in order and supplying the data. The approach is code-less and implemented entirely with drag and drop. To put it plainly, by avoiding duplication, there is less to build and less to maintain, which leads to a significant increase in ROI.

Test Plans and Test Management

It’s not uncommon for a large testing project to contain upwards of ten thousand tests, but even projects with only several hundred can be difficult to manage. One of the challenges with managing a large number of tests is distinguishing one test from the next, even when good naming conventions are in place. It’s also difficult to understand whether a set of tests provides adequate coverage. AscentialTest avoids the problems by placing the test plan at the center of the test development process.

AscentialTest provides a specialized editor, designed for writing outline-based test plans. Documenting test requirements in outline form provides an immediate advantage that is easy to overlook. Compared to simply compiling a list of tests, the outline form provides a graphical view of test requirements, making it much easier to determine the level of test coverage that a test plan provides. Here is a very simple example:

The test plan on the left of the image above displays a simple list of tests. Notice how it repeats information like the words ‘Missing’ and ‘Required Field’. It’s not easy to figure out what is being covered without studying it. The test plan on the right is structured. A quick look informs the reader that there are three types of tests, ‘Required Fields’, ‘Confirmed Fields’ and ‘Formatted Fields’. During a peer review, it’s easy to surmise that one of the first questions that will be discussed is whether all of the required, confirmed and formatted fields are covered. Reviewing a plan written graphically encourages the test plan writer and the reviewer alike to drill down deeper to ensure comprehensive coverage. In a simple example like the one presented above, the difference in the two planning approaches may be subtle, but if you imagine a more complex test problem, the advantage of the outline-based format will quickly become evident.

AscentialTest provides Test Plan Attributes to label tests within Test Plans so that they can be easily located for execution. Any number of user-defined attributes can be defined and applied to test requirements. At test execution time, the user submits a query against attributes to select tests to be run. For example, if a subset of tests is labeled with the attribute value ‘Smoke Test’, those tests that make up a smoke test can be marked with a single query. When combined with outline-based test plans, attributes can be applied to groups of tests with minimal effort by taking advantage of inheritance built into the outline model.

Why This Matters:

Test plans provide a way to organize tests within AscentialTest. Other testing solutions require integration with a separate component in their solution suite or with a third party test planning tool. Doing so incurs several large and costly administrative tasks that must be completed to organize potentially thousands of tests. It becomes arduous and time-consuming to locate tests at execution time.

Test Data Management

While most other testing tools require users to integrate their tests with test data stored externally in spreadsheets, AscentialTest provides a built-in test data solution. Data tables offer significant advantages over spreadsheets for maintaining test data. Instead of storing only strings, AscentialTest data tables store data in the type that is required by test actions including but not limited to integers, dates, complex lists and records. Users are not required to build data tables manually. AscentialTest automatically generates data tables from the data requirements of tests expressed as test parameters. Table columns associated with selector type elements like popuplists and radio groups provide dropdowns to select from a list of possible values, saving time and reducing input errors. The image below depicts a typical data table:

Why This Matters:

By automatically generating test data tables, AscentialTest saves the time and effort required to build them. Furthermore, since the field data types are generated to match the requirements of the associated test actions, users do not have to learn about data types or data type transformations, which require programming background. The picklists generated for selector type fields like dropdowns and radio lists eliminate input errors and reduce the time required to add test data. Finally, organizations save on the cost of spreadsheet components like Microsoft Excel.

Test Execution Management, Reporting and Metrics

Most testing tools require integration with other components within their solution suite for test execution management, reporting and metrics. AscentialTest’s Test Set component provides built-in support.

Each Test Set contains an Overview which provides a real-time status:

Included are facilities to provide reason codes for blocked and failed tests to support project management metrics. Test Sets can be annotated to explain results for audits and extended to include documents for test strategy, evidence of test completion and defect tracking.

Below is a sample reports that can be generated for a Test Set:

Test Sets can be run on a single local or remote machine or on a group of machines. The multiple-machine controller distributes tests to available machines to increase throughput. A customer recently shared their runtime statistics with us. Using a farm of virtual machines, they were able to reduce their testing cycle from 3 staff members over a two week period to one staff person over two days.

Why This Matters:

Organizations do not have to incur additional cost for test execution management and reporting since these features are built in to AscentialTest’s all in one testing solution. Furthermore users do not have to spend additional time manually integrating various components that were, in most cases, not designed to work together.

Spend Time Testing, Not Building Frameworks

The degree to which an automated testing solution provides a test framework significantly impacts the cost of building a set of automated tests and the return on investment. Building a custom test framework is costly and inefficient. Time should be focused on building tests that validate the target application, not on test frameworks.

AscentialTest automates much of the process of building automated tests by providing pre-built components like the test class, class library and test planner and through effective use of wizards to generate components including app states, steps, tests and data tables. These features combine to provide a full testing solution that reduces costs and increases tester productivity.