A JavaScript Testing Journey
Automated testing controversies
In software development, there are few topics as controversial as test automation.
The first controversy revolves around the fundamental value of automated tests. What can automated testing achieve and contribute to correct, bug-free software that works for the user? What does a valuable test look like?
The second debate is about whether the effort of testing is worth it – for the end user, for the business, the development team. Testing takes time, costs money and requires infrastructure.
Testing is a skill that developers need to learn in addition to countless other skills. Many developers, even seasoned ones, are beginners when it comes to automated testing. They often feel unproductive, blocked and frustrated when they have to write tests.
The third argument deals with the practical implementation of tests. How should we test a particular feature? Which tools and frameworks should we use? What makes a reliable test? And last but not least, what color should the bike shed in the garden be painted?
While I follow these discussions with some ironic distance, I do not want to dismiss them. We need to discuss these important questions again and again, especially with beginners. I sympathize with everyone who struggles writing tests. And I admire those who teach testing with respect and an open mind.
What do I know about testing anyway?
In my professional work, I have delved deeply into automated testing of websites and web applications. I have published the online books Robust JavaScript and Testing Angular in which I tried to share my experience.
You would think I have a solid opinion on the subject. I sure know something about testing. But I have reached a Socratic point where I know that I know nothing certain about testing.
In a recent client project, we wrote and maintained a large code base with an extensive test suite. That changed my mind in several ways I would like to describe here.
Giving automated testing a central role
For our client, we developed a data visualization framework written in HTML, SVG, CSS and JavaScript. The data experiences are rendered both on the server and the client. The visualized data is mostly pulled from HTTP APIs.
Automated testing played a central role in the project. We pursued the following goals:
- Tests should give us strong confidence that all bits and pieces work as designed and continue to work after a change.
- Tests should cover the complex logic that transforms and visualizes different kinds of data. It’s almost impossible for manual testing to cover these cases.
- Tests should cover accessibility features that are not immediately obvious for users without assistive technologies.
- Tests should be easy and straight-forward to write. They should run fast and reliably.
The initial testing setup was:
- Unit tests of plain JavaScript functions and UI components
- Integration tests of public HTML/JavaScript APIs and Node.js HTTP services
- End-to-end tests running against full web pages
- Static code analysis with linters, static types and accessibility checkers
We wanted build and test all parts on every change on a continuous integration server to give our developers quick feedback. We have successfully used GitHub Actions for this purpose. After some optimization, the builds, checks and tests take 15-20 minutes to run.
Unit and integration tests
We started using the Jest test runner for tests of plain JavaScript code, Node.js services as well as Svelte and Preact components. Jest is the de facto standard for testing JavaScript code. It was originally developed at Facebook alongside React and later React Native and released in 2014.
Testing UI components with jsdom
While Jest runs under Node.js, it emulates browser JavaScript APIs with jsdom. This enables web developers to test their client-side JavaScript code – for example, Svelte components – in a pure Node environment. Such tests run faster than spinning up a fully-fledged web browser.
Rendering UI components with jsdom makes sense for testing on the abstraction level of the DOM framework, Svelte or Preact in our case. We trust them to perform the proper DOM updates. We trust jsdom to implement the DOM correctly. So we test against the resulting HTML tree. But we need to keep in mind that other browser APIs might or might not be properly emulated by jsdom.
Since jsdom merely emulates JavaScript running in the browser, there a numerous large and small differences. Most importantly, the generated HTML and CSS is never rendered. Elements do not have a size, a visibility or other computed styles. Therefore, features that revolve around visual rendering need to be tested in real browsers.
Testing UI components by simulating user interaction
Based on my experience with automated testing of user interfaces on the web, I support this statement:
The more your tests resemble the way your software is used, the more confidence they can give you. – Kent C. Dodds
This is the motto of the Testing Library, which is a whole family of libraries for testing HTML DOM and JavaScript frameworks that render HTML.
The Testing Libraries provide handy utilities for testing components in React, Angular, Svelte, Preact etc. But what makes them special is the underlying testing methodology.
The Testing Libraries promote high-level tests that do not test implementation details. The tests should interact with the HTML document as it is presented to the user: Text, buttons, links, form fields, elements with certain ARIA roles. Tests should deal with these HTML elements rather than component instances. Quite like end-to-end tests which by nature know nothing about JavaScript frameworks and component organization.
Consequently, the Testing Library recommends finding elements by text content, accessible name, form field label or ARIA role. This way, testing focuses on what matters to the user while promoting semantic markup and good accessibility practices.
In the client project, accessibility was a core requirement. So the Testing Library proved to be particularly beneficial: If the test finds and clicks a button with a certain accessible name, it also guarantees that the button is indeed a button element with this very label. And not a meaningless div that happens to have a click handler.
Switching from Jest to Vitest
In the last years, Facebook (Meta) pulled out of the maintenance of Jest, transferred the legal ownership to the OpenJS foundation and started a community funding. Despite being one of the pillars of the JavaScript ecosystem, Jest has become a project a few volunteers are thanklessly maintaining in their free time.
Our biggest issue with Jest was the lack of proper ECMAScript modules (ESM) support. More and more packages are published as ESM only and code transformers output ESM only. It became harder and harder to test our growing code base with Jest. We did not succeed to get the tests running with Jest’s experimental ESM support.
Another issue was code parity between production and tests. For the development and production build, we used Vite, which has excellent ESM support. For Jest, we had to use a different build pipeline with different code transformers. So the tested code slightly differed from the production code.
We started evaluating Vitest as an alternative to Jest. Since we were using Vite, the Vitest test runner would tightly integrate with the existing build pipeline. Vitest offers almost the same APIs as Jest and supports jsdom as well.
While the Jest ecosystem did not make the progress we wished for, Vitest grew into a viable alternative to Jest. We evaluated Vitest three times over the course of one year. The first and second attempt, we ran into bugs and incompatibilities. Luckily, the Vitest team addressed them quickly while improving the stability and performance. The third attempt, we were finally able to migrate our tests from Jest to Vitest.
In the larger ecosystem, Jest is still nine times more popular than Vitest, but Vitest is gaining users quickly.
End-to-end tests
For end-to-end (E2E) tests, we used Cypress and Playwright right from the start. Both are “second generation” end-to-end testing frameworks that orchestrate Chromium, Firefox and WebKit browsers.
Our in-depth tests are written for Cypress and run on Chromium. Additional high-level tests are written for Playwright and run in Firefox. Why that?
Using the Vite build tool and @vitejs/plugin-legacy, we produce two builds:
- A modern build that requires ECMAScript Module (ESM) support. In particular,
<script type="module">plus dynamic imports plus import.meta. - A legacy build for browsers without ESM support. This mostly targets old Chrome and Safari versions. A significant number of people have mobile devices that cannot be updated.
Because of these two builds, we used using two end-to-end testing frameworks: Cypress tested the modern build with Chrome. Playwright tested the legacy build with Firefox.
Firefox supports ESM since version 60 (May 2018). When the feature was still in beta, the config option dom.moduleScripts.enabled switched it on. When the feature became stable, the option was enabled per default and allowed to switch it off. This is what we did to mimic an old browser without ESM support.
You might wonder, why not use Cypress for testing the legacy build as well? After all, Cypress does allow running tests in Firefox and does allow passing launch options. But the Cypress in-browser test runner itself is written in JavaScript using ESM syntax! Disabling ESM support would paralyse the test runner.
When your testing trick suddenly stops working
With version 117, Firefox removed the feature flag for ECMAScript Modules. That makes sense since ESM is an essential web feature today.
Unfortunately, this change made our dual setup with Cypress and Playwright obsolete. The tests still pass on Playwright with Firefox 117, but they test the modern build, not the legacy build.
We are still figuring out how to test the legacy build without much cost and effort. For now, we have pinned the Playwright version at 1.37, so it launches Firefox 115. This is possible because Playwright versions and browser versions are hard-wired.
Despite these hiccups, Cypress and Playwright are both excellent tools that served us well. Tests for Cypress and Playwright are easy to write and execute reliably. End-to-end testing as a whole improved significantly thanks to these projects.
Recently, Cypress and Playwright gained the experimental ability to test JavaScript UI components. They became a “one-stop shop” for UI testing in real browsers. We have not compared this approach to the existing Vitest-based tests in this project, but we will definitely do so in the future.
Static checks with TypeScript and linters
In his influential articles on testing JavaScript, Kent C. Dodds extends the familiar Testing Pyramid so it becomes a Testing Trophy: Static code checks form the foundation. Unit, integration and end-to-end tests rest on this foundation.
These checks are especially important for JavaScript, a dynamically-typed language with many pitfalls. Static analysis preserves API contracts, checks for code slips that may lead to runtime errors and enforces coding guidelines for robustness, performance and accessibility.
For us, TypeScript has proven to be an indispensable tool for software design, for implementation and for maintaining code correctness. Our project’s modular and flexible architecture is held together by type contracts: Configuration objects, function signatures, component props and data objects. These rules are codified in TypeScript types and enforced by the TypeScript type checker.
Many developers understand TypeScript as a new language with new syntax on top of JavaScript. This is true if you write .ts files instead of .js files. In our recent project, we wrote only a few .ts files with type declarations. The implementation itself used plain JavaScript with type annotations in JSDoc. This way, we got strict type checks without the need to compile TypeScript to JavaScript code.
For static code analysis, TypeScript, ESLint, svelte-check as well as axe-core are powerful, reliable and mature. They form the basis of automated testing. While we went all-in with static types in the latest project, you can decide which level of strictness you want with these tools.
Pragmatic testing insights
This project challenged my beliefs about software testing in a good way. Here are my main insights:
- Testing fundamentally means struggling with tools – for better and for worse. Improving the software, improving the tests and improving the tools go hand in hand.
- Tools determine your testing practice and also shape how you reason about testing conceptually. It is also well-known that testing shapes and improves your design decisions as well as the code implementation. In the best case, this leads to an upwards spiral of doing, learning and understanding.
- The tooling landscape progresses quickly, especially for testing UI components implemented JavaScript. You cannot stick with one setup over the course of years. We had to find a way to keep up with these changes and benefit from them.
- Testing is trying out, going back and forth. Throwing out your plans, making pragmatic decisions.
Credits
Thanks to my 9elements teammates Daniel Hölzgen, Julian Laubstein, Leif Rothbrust, Matthias von Schmettow and Philipp Doll who contributed to this project.
Work with us!
Thanks for reading this article! If you are planning an ambitious web project and need a strong design and development partner, 9elements is the right agency for you.
