Over the course of four months, the set of tests that Neal built with AscentialTest were used to improve the stability and performance of Dynamo Studio. Initially the system failed after only an hour of execution. The tests needed to be bullet-proofed and retry loops were added because it was important to be able to keep the tests running even as the system began to fail. At first small incremental improvements were made. Eventually an incorrect memory setting was found in AWS. It had manifested in a way that the regression caught, but it was difficult to find and fix. Resolving that problem was a big win not only for Dynamo Studio. It also highlighted how process improvement can be made by breaking out of a culture of open source that had almost become religion. Using an alternative approach proved that testing from the front end with a commercial solution can yield benefits that were not obtainable using the strategy that had been become common practice. The fix in this case was not small potatoes. Neal estimates that it was worth hundreds of thousands of dollars to the bottom line.

The Dynamo team also found that AscentialTest brought immediate value in an unexpected way when a Mongo database was hacked, and all of the test data lost. Fortunately, Neal had built ‘tests’ that were capable of restoring data. He said that capability “really got people’s attention” and even naysayers were stunned when it came to the rescue a second time several months later.

Being able to start building tests immediately without having to spend a lot of time building a test harness is something Neal appreciates about AscentialTest. He found that he can whip something up that can be used to identify a memory leak in a half hour. He considers it a huge win for development to be able to build tests that have a limited shelf life because the level of effort is so small. That works well in an Agile, CI environment where the feedback cycle is short.

When compared to an open-source solution, Neal found that AscentialTest is not limited. When the inevitable stumbling blocks occur, there is a dedicated technical support staff to call on along with a user forum in the Zeenyx eco system that is very targeted. While there is a huge base of Selenium users, the available help is un-curated. Neal says that in that environment “you have to do all the work, whereas with AscentialTest, there’s one place to go for an answer”.