---

Table of Contents

---

Apparently, Commercial Software Testing Tools are Too Expensive

What a Thousand G2 “Dislikes” Taught Us

Over the last few weeks, we analyzed over a thousand reviews entered into the “Automated Software Testing” category on the G2 website. While we prefer relying on Gartner Reviews (they tend to be a little more comprehensive), there simply aren’t enough of them available to draw meaningful insights. Gartner only lists about 73 vendors, and only a dozen or so of the reviewed tools have more than 10 reviews. G2, on the other hand, covers about 231 vendors and contains about 13k reviews. So, while there may be a lot of debate about the value and veracity of the G2 reviews, it is great place to start.

We categorized the “dislikes” in the following buckets:

• Buying it and getting started (cost, onboarding etc.)
• Learning Curve
• Poor UI
• Lacking Features
• Test Creation (complex object recognition etc.)
• Test Execution
• Test Management & Data Management
• Test Reporting
• Buggy Releases
• Support/Learning Resources/Community
• Slow or Unstable Execution
• Resource Allocation Issues

Leading Dislike #1 – “Great software, but The license cost is just too high”

While many of the expressed dislikes were nuanced, idiosyncratic and somewhat difficult to categorize, several stood out as 1) frequent and 2) easy to categorize. They were – cost, support & learning resources, test creation issues and test object recognition/synchronization issues.

Here, we will address the cost issue, and hopefully provide the solo testing community with a full-featured testing tool at a reasonable cost.

Many of the dislikes related to pricing also mention the ready availability of Selenium, a free, open source solution for testing, albeit with severe limitations relative to commercial tools. And as best as we can tell, we have not seen another enterprise-level tool that attempts to provide anything remotely price competitive with free.

The Selenium Conundrum – When Free Isn’t Free

Why isn’t it free? Simple: the cost of labor swamps the cost of your testing software. Consider that in any test automation project, the fee for the tool is only a small part of the overall budget when compared with the cost of building the tests. So if you pay $10,000.00 for a testing tool and $100,000.00 in labor costs, have you saved money for your organization by selecting an open source tool, when it costs more than twice as much to build and maintain the tests?

Selenium Increases Labor Costs Relative to Good Testing Software

Nowhere will you read that Selenium is an awesome testing tool. You won’t read that it is easy to use or that the tests are easy to maintain. If there were a charge for Selenium, it is unlikely that it would be selected at all when compared to tools like AscentialTest that have been designed to promote productivity and provide a substantial return on investment in the first year of usage.

How Do We Know?

The AscentialTest team has been creating software testing solutions since the early days of the software testing industry (Segue, where we created Silk Test, was founded in 1988). Brian, our co-founder, was the head of R&D at Segue, and Dave, our co-Founder, was the Chief Architect at Segue. They also formed our own consulting firm (Star Quality) in 1998, specializing in automated software testing.  We have seen it all.

Don’t Believe Us? Ask Google:

Query – What are the downsides of using Selenium for automated software testing?

Answer:

• Not an All-in-One Solution
• Steeper Learning Curve
• Limited Desktop and Mobile Support
• Maintenance Challenges
• Browser Dependency
• Limited Image Testing
• Lack of Built-in Reporting
• No Technical Support

An Example: Browser Navigation

When a user navigates your target application, do they close and reopen the browser after each transaction? Probably not. But when you run a test with Selenium, each test opens its own private browser. That limitation leaves out a lot of usage cases. It also makes it very difficult to debug a test. If a test fails several minutes into the execution, there is no way to run just a part of that test, because Selenium needs to start a new browser with each test run. The productivity loss that results from just this one limitation is astounding. If AscentialTest worked that way, we’d be out of business.

The image on the left below depicts a typical test step in AscentialTest. The one on the right
displays a function that is part of a Selenium test. What do you want your resources spending time on: testing or writing more code? Do you really want to spend money on programming to create automated tests. You decide.

AscentialTest vs. Selenium: Feature, Function and Productivity Comparison:

While it’s true that Selenium is a free set of open source tools, using them is not really free after all. The costs and benefits of test automation are significant. We’ve worked hard to provide a testing solution that makes testers more productive, saving your organization time and money. If you still think Selenium is free after reading this document, we have some lake- front property in {name your location} that we would just love to sell to you.

Features	AscentialTest	Selenium	Time Cost
Tool setup and configuration	No setup or configuration	Requires knowledge of required development environments and programming concepts	Several Days
Test framework	Automatically generated. Users begin creating tests right away	Must be custom-built and then tested. Frameworks can take weeks or months to complete.	Several Weeks
Browser support	Edge, Firefox, Chrome. Handles all web application components including dynamic tables	IE, Firefox , Chrome, Opera. Does not handle all web components, e.g. popup windows, dynamic tables	Varies based on complexity
Non-browser based application support	Yes, including windows, dotNet, PowerBuilder, Terminals, Java and many others	No	Days to build (or search for and implement a thirdparty solution)
Coding experience	None required	Yes. Requires experienced staff with skills to integrate the framework and code tests	Skillset is costly
Product support	Zeenyx support provided	Open source community along with user forum	Support is not curatated so it can take hours or days to find solutions
Software cost	License and annual	Free maintenance fee
Script generation time	Drag and drop accelerates	Scripting is time-consuming test development	Ongoing costs are high, easily take 4 or 5 times longer to develop tests
Test maintenance	Abstraction and encapsulation results in high reuse and low maintenance	High, tests are difficult and costly to maintain	Ongoing costs are high, Easily take 4 or 5 times longer to maintain tests
Test Execution Time	30 to 40% faster than Selenium	Execution time is slow	Tests run 5x faster on AscentialTest
Object recognition /repository	Powerful object recognition with a multitude of attributes used to identify objects. Builtin Repository. Dynamic objects recognized	Uses xPath, id, name and others, but not very effective. Repository not available. Dynamic objects are difficult to recognize reliably	Twice as long to define objects
Image-based object recognition	Yes	Requires third-party tools and coding to implement	Days to build or search for and implement a thirdparty solution
Test Data tables	Provides built-in data tables that automatically handles data types of any level of complexity	Requires use of external data tables	Days to search for and implement a thirdparty solution
Startup and recovery	Built-in and customizable.	No. Must be called for each test.	No solution in Selenium, each test starts a new browser
Drag & Drop UI	Yes, making test development quick and easy	No, uses a code editor	Scripting takes 2 or 3 times longer than drag & drop
Distributed testing	Yes, single or multiple remote targets can be controlled	Possible, but requires skill-set and coding to configure	Days to build (or search for and implement a thirdparty solution)
Scheduling tests	Built-in	A lot of code must be written to implement scheduling	Days to build (or search for and implement a thirdparty solution)
Batch execution	Yes from Plan, Suite or Test Set	Yes, but knowledge required to run through Pom.xml	Days to build (or search for and implement a thirdparty solution)
Image capture/compare	Yes, includes masking and tolerance	Requires third-party tools and a lot of code to implement	Days to build (or search for and implement a thirdparty solution)
Integration with SVN	Built-in, including external non-AscentialTest files	A lot of custom code must be written for integration and management of files	Days to build (or search for and implement a thirdparty solution)
Integration with JIRA	Built-in	A lot of custom code must be written for integration	Days to build (or search for and implement a thirdparty solution)
Integration with Jenkins	Yes	Yes
Test reports	Standard and custom reports available for test results and metrics	No	Days to build (or search for and implement a thirdparty solution)
Test management	Built-in	No	Days to build (or search for and implement a thirdparty solution)
Dependence on access to target application	Tests can be developed using snapshots when target application is unavailable	Application must be available to develop tests
Software upgrades	Zeenyx provided installer	User must manually track upgrades for their IDE, webdrivers and any plug-ins being used

Try The Selenium Killer – AscentialTest Solo

Software testing isn’t a puzzle to be solved, it is a job to be done. Now that you have seen the true cost of using open source tools, get started with Solo Tester, AscentialTest’s Selenium killer package.