LOG IN
Find MentorsFind FreelancersCommunity PostBlog
SIGN UP
LOG IN
Codementor Events
Vitali Zaidman
Follow

An Overview of JavaScript Testing in 2018

Published Mar 12, 2018Last updated Sep 08, 2018
An Overview of JavaScript Testing in 2018

TL;DR; Use Jest for unit and integration tests and TestCafe for UI tests.

This guide is intended to catch you up with the most important reasoning, terms, tools, and approaches to JavaScript testing in 2018. It combines information from many great articles, that are linked at the bottom, and adds from our own experience in Welldone Software Solutions where we implemented different testing solutions for different products for years.

Whoever reads and understands this guide, can safely assume they know the big picture of the state of JavaScript testing in the web development community for 2018. This is a great reason to share it with all your co workers, family and friends 😃

  • I did a pretty serious research for this article. If you see anything I missed or got wrong, comment about it and I’ll address it in no-time.
  • Notice the links at the bottom of the page. Reading them will take you from just understanding the big picture, to be an expert (in theory).
  • The best way to implement this guide is to choose whatever testing types you need, select several tools that looks suitable for you, and test them all. We live in a world of a great abundance of tools and practices. Your goal is to filter them and to get to the best set up for your individual cases.

Enjoy 😃

Intro

Look at the logo of Jest, a testing framework by Facebook:

As you can see, their slogan promises a “painless” JavaScript Testing, but as “some guy from the comments” pointed out:

And indeed, Facebook have an excellent reason to use this slogan. In general, JS developers are not too happy with website testing. JS tests tend to be limited, hard to implement, slow and sometimes expensive.

Nevertheless, with the right strategy and the right combination of tools a nearly full coverage can be achieved and tests can be very organized, simple, and relatively fast.

Test Types

You can read about different test types in more depth here and here and here.
In general, the most important test types for a website are:

  • Unit Tests - Testing of individual functions or classes by supplying input and making sure the output is as expected.
  • Integration Tests - Testing processes or components to behave as expected, including the side effects.
  • UI Tests - (A.K.A Functional Tests) Testing scenarios on the product itself, by controlling the browser or the website, regardless of the internal structure to ensure expected behavior.

Test Tools Types

Test tools can be divided into the following functionalities. Some provide us with only one functionality, and some provide us with a combination.

In order to get a more flexible functionality, it’s common to use a combination of tools even if one can achieve relatively the same.

  1. Provide a testing structure (Mocha, Jasmine, Jest, Cucumber)
  2. Provide assertions functions (Chai, Jasmine, Jest, Unexpected)
  3. Generate, display, and watch test results (Mocha, Jasmine, Jest, Karma)
  4. Generate and compare snapshots of component and data structures to make sure changes from previous runs are intended (Jest, Ava)
  5. Provide mocks, spies, and stubs (Sinon, Jasmine, enzyme, Jest, testdouble)
  6. Generate code coverage reports (Istanbul, Jest, Blanket)
  7. Provide a browser or browser-like environment with a control on their scenarios execution (Protractor , Nightwatch, Phantom , Casper)

Let’s explain some of the terms mentioned above:

Testing structure refers to the organization of your tests. Nowdays, tests are usually organized in a BDD structure that supports behavior-driven development (BDD). It often looks like this:

Gist

Assertion functions are functions that make sure that tested variables contain the expected value. They usually looks like this, where the most popular are the first two:

Gist

TIP: Here is a nice article about advanced Jasmine assertions.

Spies provide us with information about functions- How many times were they called, in what cases, and by whom?

They are used in integration tests to make sure that the side effects of a process are as expected. For example, how many times was a calculation function called during some process?

gist

Stubbing or dubbing **(**like doubles in movies) replaces selected functions with selected functions to ensure an expected behavior on selected modules.

If we want to ensure user.isValid() will always return true during a test, where you test a different component, for example, you can do this:

Gist

It also works with promises:

gist

Mocks or Fakes are faking certain modules or behaviors to test different parts of a processes.

Sinon can, for example, fake a server to ensure offline, fast and expected responses when testing a process.

Gist

Snapshot Testing is when you compare a data structure to an expected one.

The following example, from the official Jest documentation, shows a snapshot test of a certain Link component.

Gist

It won’t actually render and take a picture of the component, but it would save its internal data in a separate file like this:

Gist

When the test runs, and the new snapshot is different from the last one, the developer is prompted to agree that the change is intended.

Notice: Snapshots are usually made to compare component representation data but they can also compare other types of data like redux stores and inner structure of different units in the application.

Browser or browser-like environment can be one of the three:

  • jsdom —A Pure JavaScript environment that simulates a real browser. It has no UI and it renders nothing- It provides you with window, document, body, location, cookies, selectors and whatever you get when you run your JS inside the browser.
  • Headless Browser Environment — A browser that runs without a UI for the purpose of making the browser respond faster.
  • Real Browser Environment  — An actual browser that opens and runs your tests.

Putting it All Together

We suggest using the same tools for all the test types if possible:
The same testing structure and syntax (1), assertion functions (2), result reporting, and watching mechanism (4).

We also suggest creating two different processes. One for running unit and integration tests and another for UI tests. This is because UI tests takes a longer time, especially if tested on different browsers and usually use an external service to provide you with different devices and browsers (this will be discussed later) and can cost money so you would probably prefer to run it much less then the first process. For example: only before a merge of a feature branch.

Unit Tests

Should cover all small pure units of the application- utils, services and helpers. Provide all these units with simple and edge case inputs and make sure their outputs are as expected using the assertion functions (3). Also make sure to use a coverage reporting tool (6) to know what units are covered.

Unit tests are one of the reasons to use functional programming and pure functions as much as possible- > The purer your application is, the easier you can test it.

Integration Tests

Old school tests were focused on unit testing and resulted in applications where many small parts are working but the processes as a whole keeps on failing.> Integration tests (including snapshots), on the other hand, can detect many unexpected failures where you fix one thing and break the other.> It is also important to remember that in the real world , for the reasons of imperfect design and the widespread use of black boxes, not all units are pure and not all units are testable- Some units can be tested only as part of a bigger process.

Integration tests should cover important cross-module processes. Comparing to unit tests, you would probably use spies (5) to ensure expected side effects instead of just asserting the output and stubs (5) to mock and modify the parts of the process that are not in the specific test.

Also, as opposed to unit tests, a browser or browser-like environment (7) could help with processes that are dependent on the window and when part of the process is to render certain components or interact with them.

Component snapshot tests (4) fall into this category as well. They provide us with a way to test how processes affect selected components without actually render them or using a browser or browser-like environment.

UI Tests

Sometimes the quick and effective unit and integration tests are not enough.

UI tests are always running inside a browser or browser-like environment (7) that were discussed earlier.

They simulate user behavior on these environments (clicking, typing, scrolling etc…) and make sure these scenarios actually work from the point of view of an end user.

It is important to remember that these tests are the hardest to set up. Imagine yourself creating an environment to run a test on different machines, devices, browser types and versions. This is why there are many services that provide this service for you. And even more can be found here.

List of General Prominent Testing Tools

jsdom

jsdom

jsdom is a JavaScript implementation of the WHATWG DOM and HTML standards. In other words, jsdom simulates a browser’s environment without running anything but plain JS.

As mentioned before, in this simulated browser environment, tests can run really fast. The drawback of jsdom is that not everything can be simulated outside a real browser (you can’t take a screenshot for example) so using it will limit your test’s reach.

It is worth mentioning that the JS community rapidly improves it and the current version is very close to a real browser.

Istanbul

Istanbul, a JavaScript test coverage tool

Istanbul will tell you how much of your code is covered with unit tests. It will report on statement, line, function and branch coverage in percentages so you will understand better what is left to cover.

Karma

Karma

Karma lets you run tests in browser and browser like environments including jsdom.

Karma hosts a test server with a special web page to run your tests in the page’s environment. This page can be run across many browsers.

This also means tests can be run remotely using services like BrowserStack.

Chai

chai

Chai is the most popular assertion library.

Unexpected

Unexpected

Unexpected is an assertion library with a slightly different syntax from Chai. It is also extensible so assertions can be more advanced with libraries that are based on it like unexpected-react that you can read about more in depth here.

Sinon.JS

Sinonjs

Sinon is a very powerful standalone test spies, stubs and mocks for JavaScript that works with any unit testing framework.

testdouble.js

testdoublejs

testdouble is a less popular library that does what Sinon does, and claims to do it better. With a few differences in design, philosophy, and features that could make it useful in many cases. you can read about it here, hereand here.

Wallaby

Wallaby

Wallaby is another tool worth mentioning. It is not free, but many users recommend buying it. It runs on your IDE (it supports all major ones) and runs relevant to your code changes tests and indicates if anything fails in real time just alongside your code.

Cucumber

Cucumber

Cucumber help with writing tests in BDD by dividing them between the acceptance criteria files using the Gherkin syntax and the tests that are correspondent to them.

Tests can be written in a variety of languages that are supported by the framework, including JS that we are focusing on:

gist

Many teams will find this syntax more convenient then TDD.

Choose Your Unit and Integration Tests Framework

The first choice you should probably make is what framework do you want to use and libraries to support it. It is recommended to use the tools your framework provides until a need for unique tools arises.

  • In short, if you want to “just get started” or looking for a fast framework for large projects, go with Jest.
  • If you want a very flexible and extendable configuration, go with Mocha.
  • If you are looking for simplicity go with Ava.
  • If you want to be really low-level, go with tape.

Here is a list of the most prominent tools with some of their charactaristics:

mocha

mocha

Mocha is currently the most used library. Unlike Jasmine, it is used with third party assertion, mocking, and spying tools (usually Enzyme and Chai).

This means Mocha is a little harder to set up and divided into more libraries but it is more flexible and open to extensions.

For example, if you want special assertion logic, you can fork Chai and replace only Chai with your own assertion library. This can also be done in Jasmine but in Mocka this change will be more obvious.

  • Community - Has many plugins and extension to test unique scenarios.
  • Extensibility - Plugins, extensions and libraries such as Sinon includes features Jasmine does not have.
  • Globals - Creates test structure globals by default, but obviously not assertions, spies and mocks like Jasmine- some people are surprised by this seemingly inconsistency of globals.

Jest

Jest

Jest is the testing framework recommended by Facebook. It is based on Jasmine that we will discuss later. By today Facebook replaced most of its functionality and a added a lot of features on top of it.

After reading a huge amount of articles and blog posts, it’s incredible how people are impressed by Jest’s speed and convenience.

  • Performance - First of all Jest is considered to be faster for big projects with many test files by implementing a clever parallel testing mechanism (For example by us from our experience and in these blog posts: here, here, here, here).

  • UI - Clear and convenient.

  • Ready-To-Go - Comes with assertions, spies, mocks that are equivalent to libraries that do the same like Sinon. Libraries still can easily be used in case you need some unique features.

  • Globals - Like in Jasmine, it creates test globals by default so there is no need to require them. This can also be considered bad since it makes your tests less flexible and less controllable, but in most cases it just makes your life easier:

  • Snapshot testing - jest-snapshot is developed and maintained by Facebook, although it can be used in almost any other framework as part of the framework’s integration of the tool or by using the right plugins.

  • Improved modules mocking - Jest provides you with an easy way to mock heavy modules to improve testing speed. For example a service can be mocked to resolve a promise instead of making a network request.

  • Code coverage - Includes a powerful and fast built-in code coverage tool that is based on Istanbul.

  • Reliability - Although this is a relatively young library, throughout 2017 Jest stabilized and is now considered reliable. It is currently supported by all the major IDEs and tools.

  • Development- jest only updates the files updated so tests are running very fast in watch mode.

jasmine

jasmine

Jasmine is the testing framework that Jest is based on. Why would you still use Jasmine? It has been around for a longer time and has a huge amount of articles and tools that were created by the community.

Also, Angular still suggests using it over Jest, although Jest is perfectly suitable to run Angular tests as well, and many people do it.

  • Ready-To-Go - Comes with everything you need to start testing.
  • Globals - Comes with all the important testing features in the global scope as well.
  • Community- It has been on the market since 2009 and gathered a vast amount of articles, suggestions and tools that are based on it.
  • Angular - Has widespread Angular support for all it’s versions and it is what recommended in the official Angular documentation.

AVA

AVA

Ava is a minimalistic testing library that runs tests in parallel.

  • Ready-To-Go - Comes with everything you need to start testing (besides spying and dubbing that you can add in no-time). Using the following syntax for test structure and assertions, and runs in Node.js:

gist

  • Globals - As seen above, it does not create any test globals thus you have more control over your tests.
  • Simplicity - Simple structure and assertions without a complex API while supporting many advanced features.
  • Development- Ava only updates the files updated so tests are running fast in watch mode.
  • Speed - Runs tests in parallel as separate Node.js processes.
  • Snapshot testing is supported as part of the framework.

tape

tape

Tape is the simple of them all. It’s just a JS file you run with node with a very short and “to-the-point” API.

  • Simplicity - Minimalistic structure and assertions without a complex API. Even more than Ava.
  • Globals - Does not create any test globals thus you have more control over your tests.
  • No Shared State between tests- Tape discourages the use of functions like “beforeEach” to ensure test modularity and maximum user control over the tests cycle.
  • No CLI is needed- Tape is simply run anywhere JS can be run.

UI Testing Tools

First of all, as mentioned above, here and here you can find great articles about service providers that would host the machines where tests would run and help you run these tests on different devices and browsers.

The number of permanent tools for the purpose of UI testing differs very much from each other both in their implementation, philosophy and API, so it is strongly suggested to invest time in understanding the different solutions and testing them on your product.

  • In short, if you want to “just get started” with a reliable and simple to set-up cross-browser all-in one tool, go with TestCafe.
  • If you want to go with the flow and have maximum community support, WebdriverIO is the way to go.
  • If you don’t care about cross-browser support, use Puppeteer.
  • If your application has no complex user interactions and graphics, like a system full of forms and navigations, use cross browser headless tools like Casper.

selenium

selenium

Selenium, automates the browser to simulate user behavior. It is not written specifically for tests and can control a browser for many purposes by exposing a server that simulates user behavior on a browser using an API.

Selenium can be controlled in many ways and using a variety programming languages, and with some tools even without any real programming.

To our needs, however, Selenium server is controlled by a Selenium WebDriver that serves as a communication layer between our NodeJS and the server that operates the browser.

Node.js <=> WebDriver <=> Selenium Server <=> FF/Chrome/IE/Safari

The WebDriver can be imported into your testing framework and tests can be written as part of it:

gist

The WebDriver itself might be sufficient for you and indeed some people suggest using it as it is, but various libraries were created to extend it ether by forking and altering it or by wrapping it.

And indeed wrapping the WebDriver might add redundant code and could make debugging harder, whereas forking it might diverge it from the very active (as for 2018) ongoing development of WebDriver.

Still, some people prefer to not use it directly. Let’s look at some of libraries for selenium operating:

Appium

gist

Apium provides an API similar to Selenium for testing websites on a mobile device using the following tools:

So if you use Selenium or Selenium based tools, you can also use Apium to test on mobile devices.

Protractor

gist

Protractor is a library that wraps Selenium to add it an improved syntax and special in-built hooks for Angular.

  • Angular- Has special hooks, although can be successfully used with other JS frameworks too. Angular official documentation suggests using this tool.
  • Error reporting - Good mechanism.
  • Support - TypeScript support is available and the library is operated and maintained by the huge Angular team.

WebdriverIO

gist

WebdriverIO has it’s own implementation of the selenium WebDriver.

  • Syntax - very easy and readable.
  • Flexible - A very simple and agnostic from even being used for tests, flexible and extensible library.
  • Community - It has good support and enthusiastic developer community that makes it reach with plugins and extensions.

Nightwatch

gist

Nightwatch has it’s own implementation of the selenium WebDriver. And provides it’s own testing framework with a test server, assertions, and tools.

  • Framework - Can be used with other frameworks too, but can be especially useful in case you want to run functional tests not as part of other framework.
  • Syntax - looks the easiest and the most readable.
  • Support - No typescript support and in general, this library seems to be slightly less supported than the others.

TestCafe

gist

TestCafe is a great alternative to Selenium-Based tools. It was rewritten and open-sourced in the end of 2016.

There is still a paid version that offers non programming testing tools like a test recorder and a customer support, this is important because many outdated articles mistakenly state that it’s code is closed and count it like the libraries disadvantage.

It injects itself into the website as JavaScript scripts instead of controlling the browsers themselves like Selenium does. This allows TestCafe to run on any browser, including on mobile devices , and have a full control over the JavaScript execution loop.

TestCafe is JavaScript and test oriented. It is under massive development right now, although it is already considered stable and full of features.

gist

Cypress

gist

Cypress is a direct competitor of TestCafe. They are doing relatively the same, which is injecting tests into a website, but they try to do it in a more modern, flexible and convenient way.

They are newer, and just moved from closed beta to public beta (At October 2017) but they already have many adopters.

  • No Cross-Browser Support - Only support chrome right now (and not headless). They are working on it as this article is being created. (And we will update the article once they do it)
  • Lacks Advanced Features - Parallel testings and several testing tools are still missing comparing to TestCafe but they are on the road-map of the product’s diligent team.
  • Documentation - Solid and clear.
  • Debugging Tools - Easy debugging and logging of the test process.
  • Using Mocha as it’s test structure provider makes it’s use pretty standard and can make your UI tests be built in the same structure as the rest of your tests.

gist

Puppeteer

gist

Puppeteer is a Node.js library, developed by Google. It provides a convenient Node.js API to control Headless Chrome .

Headless Chrome is just a regular Chrome v59+ that is launched with the
--headless flag. When chrome is run in headless mode, it exposes an API to control it, and as said before, Puppeteer is the JavaScript tool that Google provides to control it.

Here it is worth mentioning, that Firefox has also released their headless mode in the end of 2017.

Notice that different testing tools can also use Headless Chrome and Firefox. For example: TestCafe, Karma, .

  • Puppeteer is relatively new, but it has a big community that uses and develops tools and wrappers around it.
  • Since it is native, it is faster, it is native and uses the latest Chrome engine , unlike PhantomJS that is built on an older fork of WebKit. (We will discuss it in the next section)
  • One major drawback of Headless Chrome (thus of Puppeteer as well) is that it doesn’t supports extensions like flash and probably wouldn’t in the near future.

PhantomJS

gist

Phantom implements the chromium engine to create a controllable Chrome-like headless browser.

Since Google announcement of theirPuppeteer, It’s creator and maintainer Vitaliy Slobodin no longer works on it, so it is maintained and developed much slower since mid 2017, although it is still somewhat maintained.

Why would you use Phantom over Puppeteer?

  • First of all because it is much more mature and has many great guides and tools.
  • It also used by many useful tools like CasperJS that we will discuss later.
  • It uses older WebKit so it can simulate older Chrome browsers.
  • Also, as mentioned before, Phantom supports extensions like Flash as opposed to Headless Chrome.

Nightmare

gist

Nightmare is a great UI testing library that offers a very simple test syntax.

It uses Electron which is similar to Phantom but uses a newer Chromium and actively maintained and developed, since Electron’s main purpose is to build cross platform desktop apps with JavaScript, HTML, and CSS.

Also, they are now [discussing(https://github.com/segmentio/nightmare/issues/1092) and experimenting with Headless Chrome as well.

Here how Nightmare code looks like vs Phantom code.

gist

Casper

gist

Casper is written on top of PhantomJS and SlimerJS(The same as Phantom but written using Firefox’s Gecko) to provide navigation, scripting and testing utilities and abstracts away a lot of the complicated, asynchronous stuff when creating Phantom and Slimer scripts.

Slimer was in widespread use for a long time although considered experimental, but in the end of 2017 they released their beta version:
1.0.0-beta.1 that uses the new Headless Firefox and currently working on stabilizing it and releasing version 1.0.0.

Casper would probably migrate from PhantomJS to Puppeteer in the near future with their expected release of version 2.0 and become a great tool to test on both Headless Chrome and Headless Firefox. Stay tuned.

CodeceptJS

gist

Like CucumberJS that was discussed above, Codecept provides another abstraction over different libraries’ API’s to make your interactions with tests using a slightly another philosophy that focuses on user behaviour.

Here is how it looks like:

gist

And here is the list of libraries that can be executed using this code. All discussed above.

WebDriverIO, Protractor, Nightmare, Appium, Puppeteer.

If you believe this syntax is better for your needs, give it a shot.

Conclusion

We saw the most trending testing strategies and tools in the web development community and hopefully made it easier for you to test your sites.

In the end, the best decisions regarding application architecture today are made by understanding general solution patterns that are developed by the very active community of developers, and combining them with your own experience and by taking in account the characteristics of your application and it’s special needs.

Oh, and writing, and rewriting, and rewriting, and rewriting, and testing different solutions 😃

Happy testings 😃

Thanks 😃

Suggested Articles

Best

General

spies and stubs

Testing Frameworks Comparison

Jest

Ava

Tape

UI Tests

TestCafe

Cypress

ReactTestingProgrammingJavaScriptAngular
Report

Enjoy this post? Give Vitali Zaidman a like if it's helpful.

14
post comments1
Vitali Zaidman
Follow
Discover and read more posts from Vitali Zaidman
get started
post commentsBe the first to share your opinion
MarkdownGitHub flavored markdown supported
submit
Lubega Simon
6 years ago

its such a nice article …thanks vitali

Reply
Show more replies