Automated testing controversies
In software development, there are few topics as controversial as test automation.
The first controversy revolves around the fundamental value of automated tests. What can automated testing achieve and contribute to correct, bug-free software that works for the user? What does a valuable test look like?
The second debate is about whether the effort of testing is worth it – for the end user, for the business, the development team. Testing takes time, costs money and requires infrastructure.
Testing is a skill that developers need to learn in addition to countless other skills. Many developers, even seasoned ones, are beginners when it comes to automated testing. They often feel unproductive, blocked and frustrated when they have to write tests.
The third argument deals with the practical implementation of tests. How should we test a particular feature? Which tools and frameworks should we use? What makes a reliable test? And last but not least, what color should the bike shed in the garden be painted?
While I follow these discussions with some ironic distance, I do not want to dismiss them. We need to discuss these important questions again and again, especially with beginners. I sympathize with everyone who struggles writing tests. And I admire those who teach testing with respect and an open mind.
What do I know about testing anyway?
You would think I have a solid opinion on the subject. I sure know something about testing. But I have reached a Socratic point where I know that I know nothing certain about testing.
In a recent client project, we wrote and maintained a large code base with an extensive test suite. That changed my mind in several ways I would like to describe here.
Giving automated testing a central role
Automated testing played a central role in the project. We pursued the following goals:
- Tests should give us strong confidence that all bits and pieces work as designed and continue to work after a change.
- Tests should cover the complex logic that transforms and visualizes different kinds of data. It’s almost impossible for manual testing to cover these cases.
- Tests should cover accessibility features that are not immediately obvious for users without assistive technologies.
- Tests should be easy and straight-forward to write. They should run fast and reliably.
The initial testing setup was:
- End-to-end tests running against full web pages
- Static code analysis with linters, static types and accessibility checkers
We wanted build and test all parts on every change on a continuous integration server to give our developers quick feedback. We have successfully used GitHub Actions for this purpose. After some optimization, the builds, checks and tests take 15-20 minutes to run.
Unit and integration tests
Testing UI components with jsdom
Rendering UI components with jsdom makes sense for testing on the abstraction level of the DOM framework, Svelte or Preact in our case. We trust them to perform the proper DOM updates. We trust jsdom to implement the DOM correctly. So we test against the resulting HTML tree. But we need to keep in mind that other browser APIs might or might not be properly emulated by jsdom.
Testing UI components by simulating user interaction
Based on my experience with automated testing of user interfaces on the web, I support this statement:
The more your tests resemble the way your software is used, the more confidence they can give you. – Kent C. Dodds
The Testing Libraries provide handy utilities for testing components in React, Angular, Svelte, Preact etc. But what makes them special is the underlying testing methodology.
Consequently, the Testing Library recommends finding elements by text content, accessible name, form field label or ARIA role. This way, testing focuses on what matters to the user while promoting semantic markup and good accessibility practices.
In the client project, accessibility was a core requirement. So the Testing Library proved to be particularly beneficial: If the test finds and clicks a button with a certain accessible name, it also guarantees that the button is indeed a
button element with this very label. And not a meaningless
div that happens to have a click handler.
Switching from Jest to Vitest
Our biggest issue with Jest was the lack of proper ECMAScript modules (ESM) support. More and more packages are published as ESM only and code transformers output ESM only. It became harder and harder to test our growing code base with Jest. We did not succeed to get the tests running with Jest’s experimental ESM support.
Another issue was code parity between production and tests. For the development and production build, we used Vite, which has excellent ESM support. For Jest, we had to use a different build pipeline with different code transformers. So the tested code slightly differed from the production code.
We started evaluating Vitest as an alternative to Jest. Since we were using Vite, the Vitest test runner would tightly integrate with the existing build pipeline. Vitest offers almost the same APIs as Jest and supports jsdom as well.
While the Jest ecosystem did not make the progress we wished for, Vitest grew into a viable alternative to Jest. We evaluated Vitest three times over the course of one year. The first and second attempt, we ran into bugs and incompatibilities. Luckily, the Vitest team addressed them quickly while improving the stability and performance. The third attempt, we were finally able to migrate our tests from Jest to Vitest.
In the larger ecosystem, Jest is still nine times more popular than Vitest, but Vitest is gaining users quickly.
Our in-depth tests are written for Cypress and run on Chromium. Additional high-level tests are written for Playwright and run in Firefox. Why that?
Using the Vite build tool and @vitejs/plugin-legacy, we produce two builds:
- A modern build that requires ECMAScript Module (ESM) support. In particular,
<script type="module">plus dynamic imports plus import.meta.
- A legacy build for browsers without ESM support. This mostly targets old Chrome and Safari versions. A significant number of people have mobile devices that cannot be updated.
Because of these two builds, we used using two end-to-end testing frameworks: Cypress tested the modern build with Chrome. Playwright tested the legacy build with Firefox.
Firefox supports ESM since version 60 (May 2018). When the feature was still in beta, the config option
dom.moduleScripts.enabled switched it on. When the feature became stable, the option was enabled per default and allowed to switch it off. This is what we did to mimic an old browser without ESM support.
When your testing trick suddenly stops working
With version 117, Firefox removed the feature flag for ECMAScript Modules. That makes sense since ESM is an essential web feature today.
Unfortunately, this change made our dual setup with Cypress and Playwright obsolete. The tests still pass on Playwright with Firefox 117, but they test the modern build, not the legacy build.
We are still figuring out how to test the legacy build without much cost and effort. For now, we have pinned the Playwright version at 1.37, so it launches Firefox 115. This is possible because Playwright versions and browser versions are hard-wired.
Despite these hiccups, Cypress and Playwright are both excellent tools that served us well. Tests for Cypress and Playwright are easy to write and execute reliably. End-to-end testing as a whole improved significantly thanks to these projects.
Static checks with TypeScript and linters
For us, TypeScript has proven to be an indispensable tool for software design, for implementation and for maintaining code correctness. Our project’s modular and flexible architecture is held together by type contracts: Configuration objects, function signatures, component props and data objects. These rules are codified in TypeScript types and enforced by the TypeScript type checker.
.ts files instead of
For static code analysis, TypeScript, ESLint, svelte-check as well as axe-core are powerful, reliable and mature. They form the basis of automated testing. While we went all-in with static types in the latest project, you can decide which level of strictness you want with these tools.
Pragmatic testing insights
This project challenged my beliefs about software testing in a good way. Here are my main insights:
- Testing fundamentally means struggling with tools – for better and for worse. Improving the software, improving the tests and improving the tools go hand in hand.
- Tools determine your testing practice and also shape how you reason about testing conceptually. It is also well-known that testing shapes and improves your design decisions as well as the code implementation. In the best case, this leads to an upwards spiral of doing, learning and understanding.
- Testing is trying out, going back and forth. Throwing out your plans, making pragmatic decisions.
Thanks to my 9elements teammates Daniel Hölzgen, Julian Laubstein, Leif Rothbrust, Matthias von Schmettow and Philipp Doll who contributed to this project.
Work with us!
Thanks for reading this article! If you are planning an ambitious web project and need a strong design and development partner, 9elements is the right agency for you.