Understanding software testing

Developers make mistakes all the time. We can misunderstand requirements, miss semicolons, overlook edge case scenario or just forget about certain business requirements. Mistakes happen and it’s almost impossible to avoid bugs.

Software testing is an investigation conducted to provide stakeholders with information about the quality of the software product or service under test. Software testing can also provide an objective, independent view of the software to allow the business to appreciate and understand the risks of software implementation. Test techniques include the process of executing a program or application with the intent of finding software bugs (errors or other defects), and verifying that the software product is fit for use. [ Source Wikipedia]

There are different types of bugs and different types of testing. Testing can be done by developers if it has to involve the source code, QA professional for some specialized tests or can be done by anyone on the team if it’s just executing software as a user.

In my experience, some developers tend to skip testing and just throw the code “over the wall” to QA testers and just wait for issues to be reported. Often times it creates a bottleneck in the process and teams become very slow. It takes a lot of time to go back and forth, deliver a feature, test, report a bug, fix it, test again, repeat. Most companies have low tester-to-developer ratios (or no dedicated testers at all), usually QA testers can’t keep up with the workload.

Developers have to have a good testing strategy and be confident in the code they ship. Ideally QA testers should be either optional or be working on creating, running and maintaining automated and integration testing. A good rule of thumb is when QA tester finds a bug — developer should write a unit test before fixing the bug and then make it pass with the fix.

First, let’s cover the most common types of bugs (defects in the software).

Types of bugs:

• Business logic (something isn’t right according to business requirements)
• Security bugs (the code is vulnerable to some security exploits)
• Regression (some code updates caused existing features to break)
• Performance (the code is slow or some actions execute extra functions)
• Accessibility (the code doesn’t meet aria spec for accessibility)
• UI bugs (user interface doesn’t meet the design specification)
• Integration (two or more components don’t work together as expected)
  Two of the most common ways to approach testing as a user and as a developer are white box and black box testing.

White box testing is when we inspect the source code and verify it works according to the spec.

Black box testing is when we don’t have access to the source code and only test external interface produced by the code.

Another form of testing could be static typing can prevent some bugs too and can simplify unit testing. This especially applies in JavaScript world (flow or typescript). For example, we don’t have to check function breaks if a number was passed instead of a string. ESlint also helps to prevent some typos and other bugs.

// @flow
function square(n: number): number {
  return n * n;
}
square("2"); // Catching the error even before unit test!

Test pyramid

When we’re talking about testing software in general we have to mention the test pyramid. The point of it is to show that unit tests are easy to write and we should have many of them, integration tests are more expensive and we should fewer of them, and end-to-end tests are the most involved and we should have very few of them. If an end-to-end test catches a bug it means something in the unit or integration test is missing.

Tests Levels and Types

Test Levels (proximity to the source code and the footprint of the test)

• Unit testing. Fast, low-level tests with small footprint written by developers to test stability of isolated units of code. For example, given an argument function should return the expected result. The goal is to test a unit of code (component, module, function) in isolation and in a meaningful way. It’s the least time consuming test, we should have many unit tests and few end-to-end tests. When writing unit tests, make sure assertions are running, starting with failing test and then working on the test to make it pass.

• Integration testing. Verifying if and how two or more components or modules integrate (talk to each other). Usually this kind of testing can be done by QA tester if it’s a black box testing or a developer if it involves more complex and technical integration like a database.

• System testing (or end-to-end testing). Is a complete system testing. When unit tests and integration tests are testing parts of the system, this one is targeting the system as a whole. This testing is performed by QA testers. For example testing an entire application from login to checkout and making sure emails are delivered. This is done from user perspective, and shouldn’t be automated with data mocks or fake requests. This type of testing is the most involved and time consuming. If a bug was discovered during E2E testing, that means something was missing in unit or integration testing.

• Acceptance Testing is a level of the software testing where a system is tested for acceptability. The purpose of this test is to evaluate the system’s compliance with the business requirements and assess whether it is acceptable for delivery.

Test types

• Functional testing is using software to check if its behavior matches the expectations. Expectations should be documented in the form of a technical spec which can be written in the form of a documentation or a user story. Functional testing applies to all levels of tests from unit to end-to-end. If it answers the “what” question, it checks if a function returned the expected value.
  Nonfunctional testing focuses on usability, reliability, maintainability and other attributes. It answers the “how” question, for example function’s performance when it returns a specific value.

• Performance testing is self explanatory. It checks how fast the application loads, how responsive it is, and its reliability under high load. This kind of test is necessary to make sure the system can scale under high traffic. Stress testing and spike testing are different types of performance testing to check if system can withstand higher than the expected load.

• Security testing should be done by security specialists. It can also be performed as an audit. It’s highly likely for e-commerce websites to undergo security testing on the regular basis and have random audits. Usually security testing is focused on data integrity, confidentiality and availability. E-commerce companies can also be checked for PCI compliance. The most common security bug found in web applications is XSS (cross site scripting), which can inject JavaScript into web applications and allows “hackers” to perform some malicious actions (access to users’ personal data, display any content on hijacked webpage, redirect users to other websites, etc).

• Regression testing checks if new changes to the system have not broken existing functionality or caused old bugs to reappear. It’s done by re-running all tests on the system.

• Smoke testing focuses only on the most important functions work. This testing is performed to make sure the system is stable enough for users to do basic actions (like search for hotels or log in with email/password). It’s usually done right after production deploy to check if the server is running and everything looks okay. Smoke testing can be part of continuous deploy cycle.
  Static analysis is a type of testing where code is analyzed for any errors without running or executing the code. The most common form of static analysis are linters.

Summary

You can’t always avoid making a mistake but there are ways to catch that mistake before it goes to production. If you, as a developer, understand different types of testing and terminology, it’s easier to plan your testing strategy and collaborate with QA engineers.

This was a very high level overview on software testing concepts. We learned about types of bugs, test pyramid, test levels and test types.

I’ll be covering more about testing in the next few weeks. Diving into test-driven development (TDD) and behavior-driven development (BDD). I will also cover automation testing tools and frameworks.