Test automation is an important component of software development.
The benefits cannot be understated. It saves time, it ensures consistency and it increases efficiency.
Needless to say, the Return On Investment for test automation is remarkable.
We cannot perform automated testing without tools, and choosing the right tool can determine whether it will be a success or a costly failure.
A wide selection
The abundance of tools on the market can make the selection process overwhelming.
Furthermore, the Marketing tactics used by some of those tools can send us on the wrong path.
Actually trying all the test automation solutions on the market and implementing a POC for each one of them is a costly experiment in terms of time and resources.
Who should be in charge of selecting the tool?
Even if Software Developers, Testers and DevOps Engineers will be the ones working with that tool, they might not take into consideration the overall cost of adopting that tool.
For example, most of them would likely suggest an open source library.
Tempted by the zero cost of an open source library, they might not realise that the time they consume to turn that library into a working solution translates into spendings for their company.
Executives are more qualified to estimate the total cost of adopting a test automation solution, but they might not take into consideration certain technical details that could turn into blockers.
A transparent collaboration is the key.
Why even consider paid tools when there are open source alternatives?
A software product can include a combination of Code, Documentation, Infrastructure, and Technical Support.
Open source solutions usually only include the first 2 components:
Code and Documentation.
A modern test automation solution will always require infrastructure, so the zero-cost illusion of the open source solution fades away.
The lack of Technical Support could easily translate into time-consuming research, debugging, digging on forums and finding hacks.
These aspects need to be taken into consideration when calculating the effort and Return on Investment, for both open source and paid tools.
The hidden cost of infrastructure
In terms of infrastructure, a good example would be the need for a cross-browser grid with different browsers on Windows and Mac machines.
In today’s CI/CD-centric world, running the tests on your own machine is considered to be an outdated and risky practice.
Attempting to minimise the infrastructure cost by using headless browsers in Linux containers is also a risky practice, since your product would not be tested in realistic conditions.
A website looks and works differently even on Chrome on Windows vs. Chrome on Mac vs. Chrome on Linux.
Cross-browser testing is not optional.
The corner cases and show-stoppers
The risk of evaluating a solution based on a POC is that your company might adopt that solution without being aware of all the limitations.
A good example is the handling of multiple browser tabs and iframes.
Another example would be the support for browsers such as Internet Explorer and Safari.
Honest Comparison Criteria
We’ve taken into account all the relevant data.
Most comparison charts that we found online were biased and included only criteria that would dishonestly place certain solutions on top.
As the saying goes:
Do not judge a fish by its ability to climb a tree.
Comparing a library with a cloud solution is possible and can be done in an honest way with the correct criteria.
- Basic Information
2. Supported Browsers
It’s good to know what browsers a test automation tool can support.
Certain tools are not able to support certain browsers.
This is mostly due to technical limitations of those products.
For example, your cannot execute automated tests with Cypress on Safari, Internet Explorer 11 and mobile browsers.
Another example is Ghost Inspector, it can only work on Chrome and Firefox, since it uses the CasperJS library.
3. Browsers in the cloud included by default
Certain tools offer you a cross-browser cloud to execute your tests on.
While other tools offer you the option to connect to an external cloud grid.
For example, Mabl allows you to execute your tests on Safari only if you connect your own external cross-browser cloud grid, which is not included in the price.
Another example is Endtest, which offers all the relevant browsers in their own cross-browser cloud.
4. Complex use cases
Any solution can handle a POC where you create a basic test for a Login page.
The real challenge tends to come when you hit complex use cases and require certain functionalities.
For example, it’s a well-known fact that Cypress does not support iframes and multiple browser tabs.
Most modern websites use iframes nowadays, just think of the payment pages, where the credit card inputs are placed in iframes.
Most modern websites also have workflows where you open a page in a new browser tab, such as when you click on the Social Media links from the footer.
The functionalities for sending API requests and connecting to databases are important for automatically cleaning the test environment or adding certain test data before or during the test.
5. Flexibility
Since a part of the listed products are codeless, it’s important to check if they offer the same flexibility as writing your own code.
Surprisingly, most codeless tools actually do offer that flexibility.
6. Advanced features
In 2021, we should expect products to have smart functionalities to help us be more efficient.
Computer Vision = the ability to perform visual testing
Self-Healing Tests = the tests automatically stay in sync with the application
Video Recordings = you get a video recording with the test execution
Scheduler = the option to schedule a test execution without using other tools
Email Testing = check an email in a test, similar to a disposable email service
It’s expected that the cloud solutions have more advanced features than the libraries, since it’s easier to offer these options in a cloud than it is to include them in a library.
7. Other relevant aspects
The value for Transparency is calculated based on several factors.
For example, Cypress got a low score because they have claims on their homepage that their solution is completely free and open source, but you later find out that you need to pay for the Cypress Dashboard service if you want parallel test executions and integrations with Slack and Jira.
They even went through the trouble of hiding their Pricing page, which doesn’t have a link anywhere on the homepage.
Selenium got a high score for Transparency, since it claims to be completely free and open source and these claims are actually true.
Mabl got a low score for Transparency because they do not admit that their product is actually using the Selenium library and for not displaying the prices on their Pricing page.
In 2021, no one wants to hear a Sales Rep saying that he calculated how much money their product will help us save and why the price is so high.
They also do not list their SubProcessors, making their product incompatible with GDPR.
Endtest and Ghost Inspector got high scores for Transparency for displaying the detailed prices on their Pricing pages.
Data Portability is also an important factor, since it prevents Vendor Lock-in and makes the products compatible with HIPAA regulations.
Endtest and Ghost Inspector are the only cloud solutions that got a high score for Data Portability, this is due to the fact that you can easily export your tests in different universal formats.
The Estimated ROI is calculated by taking into consideration the cost of the solution and the cost for compensating for what the solution lacks.
For example, Endtest has a High ROI score because it offers all the relevant functionalities and has a price which is significantly lower than its direct competitors.
Selenium has a Medium ROI score because it’s completely free and you don’t need to use other solutions to compensate for the missing functionalities.
But it does not have a High ROI because it requires a lot of man-hours.
Cypress has a Low ROI score because it lacks an important number of functionalities, which means that a team would need to use other tools in parallel.
Functionize, Testim and Mabl have Low ROI scores because their solutions are considered to be overpriced.
