I Know Whereof I Speak

I’ve been in the field of test automation since the 1980s. I have experience in all facets of the discipline as a test engineer, a QA and Development manager, a testing tool developer and a test automation consultant, trainer and mentor. I led a large development group as Executive Vice President of R&D for Segue Software, one of the premier testing tool vendors, in the 1990s. I’ve consulted on automated testing with some of the largest multi-national companies in the world.

The Big Issue for CEOs: Expectations vs. Results

What I have seen consistently is a large disconnect between management’s expectations of a successful test automation project and what actually occurs in their organizations. It’s not surprising, because many of the executives who make decisions about how budgets are allocated for software quality assurance (QA) have never worked in QA. It is more likely that they have development experience, which is a fundamentally different skill set than QA. In this article, I’ll discuss common misconceptions and provide a background on what every manager should know.

Lesson 1: You need someone with field experience on your QA Team

For starters, the folks who are the most qualified to test a software application are the ones who have domain expertise. If that domain is in retail, then it is likely that associates from actual retail stores will be brought in to participate in the testing effort. The same applies to education, manufacturing, financial, legal, science or any other industry that relies on software. Those smart and capable people are probably not also trained as programmers. Yet in many cases, they are given tools that require coding experience to successfully develop automated tests.

One solution is to hire an associate or consultant to build a test automation framework to reduce the learning curve for the rest of the testing team. I can‘t imagine how many ‘custom’ frameworks have been written over the years and how much time and money has been spent to develop them. I’ve witnessed what happens when a bright and ambitious engineer is given carte blanche to create one, where their priorities and incentives are clearly not in line with what is needed in the organization. The test framework is usually over-engineered and fully understood only by its creator, who enjoys the development challenge and the opportunity to improve upon the resume. As managers, there are red flags that we are ignoring. We should be asking ourselves, what is it about our software that would require the development of a custom test automation framework? Hasn’t this problem already been solved?

Lesson 2: Open Source isn’t Free

Another misconception that contributes to the experience gap is that open-source tools will save the company money. While it is true that open source tools come free of charge, the cost of ownership of the eventual body of automated tests is far from free. Think about this: the consultant who recommends the use of a tool like Playwright or Selenium is not incentivized to suggest a testing tool that will reduce the time and cost to build and maintain automated tests and it would not add to their bottom line if they provided a solution that could be easily turned over to the organization’s testing team after the initial build-out. Those tools require programming skills in languages including C#, JavaScript, Java, Python and Ruby. Will your staff be able to update those tests in the future when they require maintenance? The cost of test maintenance is an aspect of test automation that only practitioners fully understand. Automated tests are built for a specific version of the target application. As the application changes over time, those tests need to be maintained if they are to continue to increase the efficiency and effectiveness of the testing effort. The capabilities of the testing tool along with its embedded approach have an outsized impact on the cost of that maintenance. Open-source tools might be adequate for throw away tests but not for a regression that is intended to be used over the life-cycle of the target application. Open-source tools were not designed with maintenance in mind.

The True Cost of Open Source

Let’s face it; an open-source testing tool would not be selected if it weren’t ‘free’. What managers fail to grasp is that they are paying much more in labor costs than they would for a commercial testing tool, the best of which will provide a solution that can be used by domain experts to build and maintain automated tests that can be used over the life-cycle of the software application. Why is that? One of the supposed benefits of an open-source tool is the large online community from which to seek answers to questions or problems, but in fact it can take days of research to find a resolution. The information is ‘out there’ but it’s not curated. A commercial testing tool company would be out of business if they didn’t respond within a couple of hours at worst.

Is Your Tool Delivering Positive ROI?

Managers need to be focused on providing tools to their staffs to promote productivity and help reduce the duration of testing cycles so that organizational objectives can be met. Think about how the usage of your current testing tool has impacted the effectiveness and duration of your test cycles. Has automated testing reduced the testing budget? Has it shortened test cycles? Has it reduced the number of issues found in post-deployment? If the answer to those questions is ‘no’ and the selected testing solution is not providing a positive ROI, then stop using it.

When we originally designed AscentialTest nearly 20 years ago, we already had more than a decade of experience in developing and implementing test automation. We have a keen understanding of problems and pitfalls associated with using testing tools that are not a good fit for the organizations who use them. AscentialTest is unique in that the components of the test automation framework are either pre-built or generated automatically using a combination of wizards, information available from the window manager and drag & drop. We’ve come to understand that selecting the right testing tool for your organization is a difficult and time-consuming process so we are now offering a free six month trial, fully featured and fully supported. We are that confident that AscentialTest will prove its worth