Customer support for software testing tools can make or break your testing program, and it desperately needs a new approach.

How do we know automated testing tool support provided by vendors is inadequate?

You don’t have to look very hard to find out why so many software testing organizations choose open source over commercial testing tools. Our survey of over a thousand G2 reviews on automated software testing tools told us all you we needed to know: the quality of support provided by testing tool companies is inadequate at best. Here are a few examples.

“[Tech] Support took a few days to respond to emails and there was no phone support even if you were prepared to pay more for it which meant if you got stuck on something there was a lag of several days in getting a response. Often when the response did arrive it felt like I was getting a 1st level support pre scripted response, and more than once their suggestion would be that I try something which my support request message had clearly stated I had already tried.”

“A bug [in the testing software] introduced in a release was not resolved for 5 months and prevented my company from using the tool. Initially, technical support was very responsive and in 2 weeks, they were able to replicate the issue and determine that they had released a bug. Over the next 4 months our test backlog grew and our inability to use [the product] damaged our ability to release stable software.”

What it is vendor lock-in and why does it exists in test automation?

The investment made in building an automated regression test is significant and, unlike application source code, tests cannot easily be migrated from one platform to another. Consequently, organizations find that they are locked into their testing solution so when they need to abandon their selected tool due to lack of support, the implications go well beyond a lost investment. What is more critical is the impact on the quality of the software that needs to be tested. With test conversion from tool to tool being limited, tests usually need to be automated from scratch when migrating to a new testing solution. This ‘lock-in’ can lead to a variety of sub-optimal vendor behaviors, and in automated testing, it manifests most often through poor customer service. The vendor is not incented to provide adequate technical support because they know that it is expensive for customers to move to another testing solution.

Summary of why testing organizations feel locked into their selected testing solution

• Difficult to migrate tests from one platform to another
• Costs of training the testing team on a new testing solution
• Software quality suffers during the transition

Why software testing is uniquely reliant on good support

Unlike software development where standardized programming languages, many taught in universities, are used to build software, test automation scripting languages, frameworks and approaches are particular to the testing tool being used. Testing tool subscribers usually need more than an instruction manual or a set of help pages to be successful. It is a misconception to think that tech support’s primary purpose is to provide bug fixes for the tool’s shortcomings. On the contrary, effective support groups should be ready to provide help with getting started, staff training, test design and verification strategies.

Summary of why testing organizations require strong testing tool support

• Lack of experience with test automation
• Changes to the test automation software are sometimes required to support custom or third-party elements that might not have been encountered in the trial
• Time waiting for bug fixes or issue resolution from tool vendors impacts the duration of testing cycles and the ability of user organizations to deliver software

Lack of support for open source solutions

While using an open-source community to access support might not be efficient, it’s certainly no worse than the 1^st level support often being provided by a third-party, where canned answers are sent by support representatives that clearly have no hands-on experience with the tools that they are ‘supporting’ and where 2^nd level support is slow, unresponsive or non-existent. It’s no wonder that test organizations decide to learn open source tools on their own and search for solutions through the online community when they get stuck.

But getting support from an online community is fraught with problems. For one, there is no curator for the knowledge base that exists. Users have to hunt for solutions, often spending days or weeks to find answers. Furthermore, since open-source solutions are not comprehensive, users have to integrate with third-party plugins to extend testing support for their target applications. This level of effort requires a skill set that is often lacking so organizations have to turn to consultants which increases the cost of labor to get the job done.

What does a successful support model look like?

A successful support model needs to be multi-tiered. First, the testing tool needs to be well-documented with detailed descriptions on how to use features and scripting languages with examples. An online academy of articles and videos must be available to supplement the product documentation. Email and video meeting support with a live agent that has expertise in the software being supported is required to orient new users and provide general automated testing guidance for users of all experience levels. Finally a user forum can supplement for those searching for different opinions and approaches for designing solutions.

Zeenyx has taken a high touch approach to customer service and customer retention rate suggests that it is working well for us and our customers.

From our beginnings Zeenyx Software has offered premium support as specified in the service level agreement that is part of our support and maintenance contract. Our customers often get a first response to questions right away and wait no more than a few hours during our busiest times. In most cases a support session will result in either problem resolution or an acceptable workaround. In rare cases when a patch is required, we endeavor to provide a fix as soon as reasonably possible. Recently a customer embedded a non-standard browser in their window-based application which did not comply with hooking standards employed by chromium based browsers. Within a few days, Zeenyx Development was ready with a patch. Another customer who used Selenium in the past commented that in the open-source world, a user could spend days searching online for a solution, but when working with Zeenyx, a single email turns an answer around in just hours. We’ve found that the benefit of providing our own in-house support is that we directly receive customer feedback and requirements that help inform new product features that users really need.

Summary of our support offering

• Responsive support team that understands AscentialTest and automated testing, using both email and video meetings to orient new users and provide general automated testing guidance for users at all levels
• Well-written documentation with detailed descriptions of how to use features and the scripting language with meaningful examples
• An online academy of articles and videos must be available to supplement the product documentation
• A user forum for those searching for different opinions and approaches for designing solutions.
• On call experts who have been pre-screened and are ready to offer consulting and tailored solutions

We understand that the problem faced by organizations implementing test automation is not about how much it costs to obtain good support. It’s about the time that is lost when that support is inadequate. We think we got it right and our reviews on G2, Gartner Peer Insights and PeerSpot agree with that notion.