Other Case Studies
Executive Summary:
Autodesk selected AscentialTest for test automation of Dynamo Studio and developed a set of cross- platform tests running on both the Web and WPF clients targeted at identifying resource allocation issues in a complex multi-tiered application stack. They successfully found and fixed serious back-end issues by driving thousands of workflows through application engines and analyzing allocation of memory and other system resources to ensure that the system could handle the expected load. With AscentialTest, Autodesk was able to:
Client Objective:
When Neal Burnham of Autodesk was tasked with developing a testing solution for Dynamo Studio, he faced both organizational and technical challenges. There was a mindset in place to use open-source tools and to test at the unit level using harnesses embedded in the code itself, but the harness that he inherited did not run reliably. His intuition told him that the most efficient way to identify issues with the backend was to test through the application’s GUI using a commercial testing tool.
Before selecting AscentialTest, Neal looked at open-source alternatives including AutoIt, Selenium and Winium, but he knew from experience that given the complexity of what had to be validated, something much more powerful was required. Any of these solutions could have been used to drive use cases through the GUI, but the target was a validation of the underlying engine that converts graphical programs into other formats. This is where the rubber hits the road. To validate a multi-tiered application, Neal needed to be able to analyze XML, data accessed from the Chrome dashboard and resource allocation metrics from the OS. He also needed to verify that all file contents were correct, and that the graphics were correctly rendered.
The Dynamo GUI is available in two versions. Users can access Dynamo through either a WPF desktop application or through the Chrome browser. To reduce the level of effort to build and maintain tests, Neal wanted to create a meta layer where he could drive real world workflows through both versions of the application.
Our Approach:
Neal selected AscentialTest because he recognized the potential for designing a class of solutions to solve a class of problems. He understood the power of building small, reusable components that could be glued together and scaled up to form a broad solution.
Using AscentialTest’s snapshots, he built a matching set of object definitions to normalize the thick and thin client user interfaces. This layer allowed all of the test components to be shared across platforms. Instead of having to build separate tests for the WPF and Web versions of the Dynamo GUI, he cut his work in half by building one set that could reference either object repository.
Neal also built a validation mechanism to ensure the integrity of numerous complex file conversions. His approach involved parsing data from the Chrome developer console and XML files stored in the file system and comparing sets of data to expected results. He also used AscentialTest image capture and compare features to ensure that graphics were rendered properly.
The ability to have deployed agents on various target machines was also an important component of the solution. Neal needed to ensure that he could drive the GUI hard and fast from multiple client machines in an industrial fashion, sending thousands of workflows through the engines to confirm proper allocation of memory and other system resources necessary for the system to hold up over time.
Because Neal’s solution is not embedded in the application code, he can continue to run the regression tests without being impacted by change as the back-end technologies continue to evolve or be replaced. He believes that one of the strengths of a productized offering like AscentialTest is that he is assured that his test assets are permanent. He cites a recent announcement that the Selenium ‘Keys’ command will be deprecated soon, making it necessary to modify tests to keep them running. Because AscentialTest is a complete solution, there is no need to search for plug-ins and apply workarounds. The Zee testing language is fully formed, and aspects of the product will not be deprecated in the future.
Results Delivered:
Over the course of four months, the set of tests that Neal built with AscentialTest were used to improve the stability and performance of Dynamo Studio. Initially the system failed after only an hour of execution. The tests needed to be bullet-proofed and retry loops were added because it was important to be able to keep the tests running even as the system began to fail. At first small incremental improvements were made. Eventually an incorrect memory setting was found in AWS. It had manifested in a way that the regression caught, but it was difficult to find and fix. Resolving that problem was a big win not only for Dynamo Studio. It also highlighted how process improvement can be made by breaking out of a culture of open source that had almost become religion. Using an alternative approach proved that testing from the front end with a commercial solution can yield benefits that were not obtainable using the strategy that had been become common practice. The fix in this case was not small potatoes. Neal estimates that it was worth hundreds of thousands of dollars to the bottom line.
The Dynamo team also found that AscentialTest brought immediate value in an unexpected way when a Mongo database was hacked, and all of the test data lost. Fortunately, Neal had built ‘tests’ that were capable of restoring data. He said that capability “really got people’s attention” and even naysayers were stunned when it came to the rescue a second time several months later.
Being able to start building tests immediately without having to spend a lot of time building a test harness is something Neal appreciates about AscentialTest. He found that he can whip something up that can be used to identify a memory leak in a half hour. He considers it a huge win for development to be able to build tests that have a limited shelf life because the level of effort is so small. That works well in an Agile, CI environment where the feedback cycle is short.
When compared to an open-source solution, Neal found that AscentialTest is not limited. When the inevitable stumbling blocks occur, there is a dedicated technical support staff to call on along with a user forum in the Zeenyx eco system that is very targeted. While there is a huge base of Selenium users, the available help is un-curated. Neal says that in that environment “you have to do all the work, whereas with AscentialTest, there’s one place to go for an answer”.
Automated testing, manual testing and test management tools that are easy to learn, easy to use, and easy to maintain.
Hopkinton, MA 01748
Click here to download the Host ID generator. The download package includes full instructions. Please send us your details below to speed the process along.
Check our new Azure extension for executing AscentialTest command lines in the Azure Marketplace.
We look forward to offering additional integration with Microsoft’s Team Foundation Server and Azure through defect tracking and version control soon.
Automated testing support for OMNIS Studio is coming soon. All OMNIS elements will be supported including complex grids, list boxes, tables and tree views.
Add automated testing support easily for your custom application elements by responding to simple test API messages with json strings. This feature will open up the possibility of testing any GUI element and will be offered free of charge.
Set breakpoints in your automated tests, view global and local variables and execute statements to make it easier to debug tests running on any of our supported platforms.
We will get back to you ASAP. We aim to respond to all inquiries within 24 hours.