Tests as tools

Automated testing can be a highly useful facility for software developers.

Applied appropriately, tests can help us to discover design, work incrementally, verify correct execution, prevent regression, identify and fix bugs and understand code that we are unfamiliar with.

However, the increasing range of options, techniques, frameworks and philosophies in the field of automated testing have led to an increase in complexity. Many developers I have spoken with describe feeling overwhelmed, confused and undecided, when trying to decide how and where to apply automated testing.

A mind-set I have been adopting, which has proven useful, is "tests as tools". That is, from a broad knowledge of different kinds of automated tests, combined with thinking about what I am trying to build in the present moment, I selectively "cherry-pick" the appropriate kind of test. This article summarises how you might go about this.

What are you building?

Begin by asking this question, to yourself and/or to other team members and stakeholders, before writing any code. What, specifically, is your task at the present moment?

From the answers to this question, carefully search for statements of expectation. That is, what would you expect to be the case, had you successfully completed the present task?

Based on the answer to this question, you can then proceed to selecting an appropriate kind of test.

Selecting a test

Depending on what kind of expectation you are building for, and thus, what kind of thing you're building, different kinds of tests will be appropriate or inappropriate.

No test

Are you writing code that is essentially configuration? Code that simply defines how something should look or behave, without, itself, containing any behaviour?

If this is the case, perhaps no unit test is needed.

An example of code that requires no test might be a simple CSS block:

  .alert {
    background-color: maroon;
    color: white;
  }

Snapshot test

Are you building an isolated, stateless unit of code (e.g. a function), which is essentially a straightforward mapping of input(s) to output(s)? For example, a mapping from one static key-value pair to another?

In this case, a very minimal "snapshot test" may be all that you need. This kind of test only exists to alert you to changes to the mapping, without forcing you to re-iterate the same configuration, as would be necessary in another kind of test.

An example might be a very simple, stateless React component:

  var Pane = function (props) {
    return <div>{props.children}</div>;
  };

Unit test

Are you building a unit of code (e.g. a function), with clearly defined inputs and outputs, and some behaviour?

If so, before writing the code, you could write a unit test, which asserts your expectations, and then write the code to satisfy those expectations.

An example might be a function that validates a credit-card number:

Test:

  define('credit card validator', function () {
    it('returns true for a valid number', function () {
      ...
    });
    it('returns false for an invalid number', function () {
      ...
    });
  }

Function:

  function isValidCreditCardNumber(number) {
    ...
  }

A unit-test affords you an easy, quick way of executing the code as you write it, until it works the way you expect it to. Afterwards, you have a nice unit test that serves as documentation to future users (including yourself!) about how your function may be properly used.

Integration test

Are you writing code that will work with collaborators – either internal (e.g. calling other functions, perhaps importing them from other modules), or external systems, via some kind of tranfer mechanism (e.g. HTTP)?

In this case, you might want to write a test that verifies that this collaboration works correctly. This test will set up an environment in which it is possible for your code to talk to collaborators (either real or mocked), and then run the code to call them, and assert that the expected result was returned.

An example might be a function that makes an HTTP request:

Test:

  define('log in api client', function () {
    it('returns true for correct credentials', function () {
      ...
    });
    it('returns false for incorrect credentials', function () {
      ...
    });
  }

Function:

  function logIn(username, password) {
    ...
    request({
      uri: 'https://login-service.com/login',
      type: 'post',
      body: {
        username,
        password
      })
    ...
  }

System test

Have you written enough code to deploy to a real environment, accessible to users (either the public, or a subset of early "beta-testers")? If so, do you wish to verify that your product behaves as expected, from a user's perspective?

If so, you could manually verify this by accessing the environment and manipulating the user interface, including creating dummy credentials, etc. Manual testing is an established discipline, and there may be good reasons to employ qualified testers to verify the interface in this manner.

However, to save manual testers (or anyone else) from having to conduct mundane, repetitive tests, such as verifying that the basic, fundamental functionality of your product is working, you may wish to automate this testing.

In this case, an automated system test (also known as an "end-to-end" or "automation test") is useful.

This test may be executed manually or automatically (e.g. by a monitoring system built for the purpose). The test is configured to point to a particular environment (e.g. a URL on the web), and it accesses this environment, mimicking the actions of a regular user, and verifies that the behaviour is correct, as communicated through the user interface.

Test coverage

What does test coverage mean? Is it meaningful at all?

Ensuring that each and every piece of code has its own associated test seems to me to be an unnecessary pedantry. Not all code manifests the same kind and degree of complexity. So not all code requires the same level of testing, if any testing at all.

What might be more meaningful is ensuring that each and every piece of expected behaviour has its own associated test.

If there is no behaviour (e.g. a simple definition of rules, as in CSS), no test is needed. If there is a trivial input-output mapping, a simple alert that something has changed, such as a snapshot test, may be useful. If there is behaviour, and it can be isolated, a unit-test is certainly useful. If the behaviour cannot be isolated, an integration test, which merely verifies the collaboration of the components, is useful. Finally, if the whole system needs to be verified to be working correctly, a system test can automate this process, saving a human tester from tedious and/or repetitive manual effort.

Conclusion

Rather than trying to write every kind of test for every kind of scenario, or haphazardly writing tests based on personal style, preference or time-constraints, I highly recommend a "tool-based" approach to tests.

That is, through discussions with team members and stakeholders, and careful thought, identify the essential problem you are trying to solve and how you would expect a correct solution to behave.

Based on this information, decide what kind of code to write, and thus, what kind of test will verify this code.

Testing should not be performed blindly or dogmatically. Testing is a set of tools, which are highly useful, if applied correctly in the appropriate use cases.

Credits

The ideas presented in this article were heavily influenced by the Test Driven Development technique, developed by Kent Beck, as well as the Unix Philosophy, originated by Ken Thompson.