Authoring Automated Tests even without UI

Automation is typically developed on a lag of at least one sprint.
The dev code that is written for a feature now may only be considered for Automation in an upcoming sprint.

There will only be say a few weeks for Development after which the build will be handed off to the testing team. The testers start writing test scripts or code to identify bugs and other affected areas, report them back to the dev team, have it fixed and returned back again for Testing(Regression Testing). All of this in one sprint cycle which should happen within a week or two as planned.

As the release cycles get shorter and the requirements keep changing in Agile and DevOps, it becomes necessary to automate Regression Testing.
Also, as the dev team adds new functionality to the software, tester might still be working on an old feature and may not be able to advance at the same speed as development. This delay can also be caused due to delay in application availability in some cases.

Performing regression tests after build deployment is not the best approach to match Agile/CI/CD methodologies and the regression testing backlogs should not be carried forward to the next sprint.

But, Agile leaves no time between dev-complete and testing.
Then, how do we keep pace with Agile while delivering Quality at Speed?

Introducing In-Sprint Automated Testing!

What we could do is automate user stories within the sprint alongside development by automating Unit tests, API tests and part of the UI tests and leave the UI mapping. In short, perform In-Sprint Automated Testing.

Once the UI is close to done, you can fill in the incomplete fields. This ensures that the tests are completely automated along with development and ready to run through the Continuous Integration pipeline without any delay.

The biggest challenges in implementing In-Sprint test automation is to automate unit, API testing and UI tests for different application types like Web mobile and native apps and API services depending on your application needs.

Let's look at a few challenges of Regression Testing:

1. Identifying relevant test cases with changing code

2. Team members execute sprint as mini-waterfalls which is not the best approach since you can't do things in parallel.

It is particularly daunting to perform Regression Testing across browsers and devices of different screen resolutions and sizes. Managing environment specific datasets and variables that are likely to change in multiple execution environments at each stage like Development, Testing, Staging, Production is also crucial but would be better if these could be stored in an object repository that can be shared across environments.

3. Regression tests need to be run after every development iteration.
The delivery pipeline should trigger regression tests automatically whenever a successful build is made to reduce the dev/test complete gap.

4. The Agile methodology allows for a change in requirements even late in the development. The Scrum teams need to be prepared to accept the late change in requirements.
When the requirements change, especially towards the end of the sprint and a particular functionality hasn’t been tested well, the team should make informed decisions whether to release the feature or push it to the next sprint based on the criticality of that function/feature.

5. The lack of a Unified solution is also a big challenge in automating multiple applications as part of the same project. It takes a lot of time and effort to work on multiple tools to automate different application types.

With In-sprint, with the new feature being automated along with code-complete, there will be good enough coverage right from the first build. Since the tests are almost ready at the early stages, regression tests can be run earlier and often.

Checklist for In-sprint Test Automation and DevOps Implementation

1. Test automation framework that supports end-to-end testing including unit, API and UI testing

2. Team with the skill set that matches the framework and the chosen tool
DevOps infrastructure setup with the right set of tools (i.e. Jenkins, Docker, etc)

3. Build automation is in place and the continuous integration pipeline is ready for continuous integration.

Here’s a practical guide you can refer that explains how In-Sprint Test Automation is done without depending on the application’s availability.

Related References

In-sprint test automation for Regression Testing at DevOps speed
In-Sprint Test Automation Challenges for Regression Testing
Overcoming Automated Regression Testing Challenges in Agile