Starchive: Tracking Performance, Verification of Uploads, and Maintaining Dynamic Object IDs

A Case Study

Executive Summary:
Starchive, a client of Loop Software Services and Testing, recently selected AscentialTest for its web-based application testing. Starchive made the switch for a variety of reasons and through AscentialTest, Loop was able to provide Starchive with the ability to:

• Track performance of upload, download, page load times to gain insight when new features or backend changes impact performance.
• Easily maintain the automated suite while Starchive’s object IDs dynamically change
• Verify large upload sets of 10,0000+ which was not feasible when done manually
• Shorten overall test result, test building, and maintenance time.

About Starchive:
Starchive is a cloud-based media management application designed to create organization in the midst of the media chaos and challenges. Started from their collaboration with the Bob Dylan Music Company, the web-based software application allows individuals, companies, and creative brands to solve media challenges by making it easy to find, manage, or share media within thousands of files. Starchive is constantly adapting to bring the best user experience forward so that the everyday user can utilize the application and maneuver the intricacies of media handling.

Challenges:
When Starchive first made the switch from manual to automated testing, it was not without challenges, both general and technical. When it came to the performance data of their application, there was no historical data. At that point in time it was not cost-effective to gather data on Starchive’s application performance, therefore it was not always known how a new feature or change affected the application performance. When creating new features and changes in the backend, regressions were run to ensure the changes did not create any breakage, but other than apparent positive or negative performance changes, no performance metrics could be easily tracked.

Another challenge that arose was related to Starchive’s React Framework architecture which creates code generated classes for delivering asynchronous experiences. This caused the application’s object ID’s to be dynamic, making open-source automation testing solutions extremely difficult to maintain. Automated tests consistently broke, leaving Starchive concerned about their automated product testing and the costs to maintain it.

Creation and maintenance of the regression suites presented a unique challenge, as Starchive’s highly effective development team was able to rapidly create new features and backend updates. The application’s mission, storing massive file sets and handling extremely large uploads/downloads has led to issues with verification of uploads/downloads during testing, with the manual approach unable to verify such massive file sets in a cost effective manner.

Solutions:
Starchive chose to implement AscentialTest based on the recommendation of Matt Osiecki, Head of Automation at LSTS as he saw the large benefits that Starchive would see by choosing this solution. Because of his recommendation, Starchive began utilizing AscentialTest. The initial automation build was focused on performance testing.

AscentialTest is now used to track performance times when uploading, downloading, as well as general page load times within Starchive’s application. The tests are run daily which document performance times in a client facing dashboard and the results are compared to daily, weekly, and monthly results. Any test results that are abnormal are automatically reported to the client within an “Automation” Slack channel. The performance tests are now able to run after any change in the application on both frontend and backend which allows Starchive to see the immediate performance impact of changes.

Another position in which AscentialTest is bolstering Starchive is in its ability to identify objects based on innumerable attributes such as color, position, webtext, parent objects, and others. Dynamic IDs which would normally break open sourced framework automation can be defensively defined within AscentialTest, giving it the definite advantage.

The ability for rapid development was another significant advantage of using AscentialTest. The test case creation and maintenance within AscentialTest proved to be much more efficient than when compared to manual or traditional automated frameworks.

Regarding maintenance, the tests built in AscentialTest are composed of steps and even sub-steps. These building blocks are then used across a variety of different tests. When a change occurs on the frontend, the step that is impacted can be updated, which updates all references and tests containing the step. Because Starchive consistently rolls out new features, a feature change that affects 1000+ test cases would take days to update within a manual suite, but can be done in minutes with AscentialTest’s prowess.

With reference to the creation side of things, Step, the building blocks of tests, can be reused within a test or across multiple tests. Once the steps are created within AscentialTest, it is a simple process to drag and drop them into a new test, instead of building an entire new test from scratch. This means that reusing previously created steps can  significantly reduce the hours spent on test creation.

Starchive also shows huge improvements in the verification process via AscentialTest. Before, verifying upload/download/management of thousands of files would be cost-prohibitive and nearly impossible, with the likelihood of  human error  if done manually. Now, a verification step can be created to verify a file’s image, title, created by date, along with 30+ other properties. This step can be rerun on every file uploaded/download/edited, verifying each file in seconds, while not consuming person hours.

Results:
The effects of AscentialTest were seen straight away. There was immediate validation of new features and backend change impact on performance which was directly reported to the Starchive team within Slack. Improved user experience and adoption also rose and performance pain points are now able to be identified (slow uploading time, long page load time) via automation and remedied quickly by the development team.

When compared to open-source solutions, AscentialTest’s automation suite proved that it is resilient to dynamic ID changes, saving time and money on maintenance. Less maintenance also means that Starchive is able to obtain the test results more quickly.

The Starchive team and Loop found that the regression suite can be built and maintained much faster when compared to manual testing, saving Starchive considerable monetary investment. The automated suite is faster than manual regressions, completes more test cases with each of them covering more scenarios.

The file set verification system ensures Starchive that any files set, file type, file set size, can be uploaded, edited, and downloaded within Starchive while retaining all original data. Any issues found during file set verification can be remedied before the push to production. AscentialTest has saved Starchive time and money while opening up new testing possibilities.

About Zeenyx:
Zeenyx Software is the creator of AscentialTest, the next generation testing solution that eliminates the need for programming, allowing users to build robust, highly maintainable, and reusable tests. AscentialTest is  an enterprise level testing solution that encompasses test planning, test development, test data management, test management, test execution and reporting for applications running on Windows, the web, java, dotNet, Chromium, Electron, terminals and PowerBuilder. Zeenyx is helping Fortune 1000 companies reduce the cost of testing by making testers more productive.

About Loop:
Loop Software Testing and Services is a Northern Michigan based company that recognized the need for quality software testing and decided to do something about it. Loop prides itself on providing high quality testing to a variety of clients focused on manual testing, regression testing, exploratory testing, mobile testing, and more. They have recently partnered with Zeenyx Software as an implementation partner and will offer additional onboarding, training, and engagement opportunities. To learn more about Loop, visit www.workwithloop.com