Run Cucumber Script with Jenkins

In this post, I will show you how to configure Jenkins to run Cucumber automated testing script and display report on Jenkins UI.

Content

• Cucumber's command line mode
• CukeTest's command line mode
• Jenkins continuous integration configuration
• Configuration Jenkins test report plugin
• Summary

Automation scripts created with Cucumber framework have the advantage of easy to understand, easy to communicate, and have good maintainability, therefore it is popular among test automation teams.

DevOps is widely adopted by engineering teams, integrating automated testing into a continuous integration (CI) process is an important part of implementing DevOps. Once integrated, automated testing runs automatically and generate execution reports, which can update entire team the healthy state of the project. Below is a report of the Cucumber project shown on Jenkins:

As we know for any tools to integrate with Jenkins, it needs to support command line mode, and it’s also true for Cucumber.

Cucumber's command line mode

We use Cucumber.js to illustrate Cucumber's command line. Those who are already familiar with this tool know that it means running "Cucumber-js" command directly in the automation script directory. Or if you don't have Cucumber installed globally, run "node_modules/.bin/cucumber-js" (or the one with .cmd extension if you are on Windows) and it will use the locally installed Cucumber. After execution, it prints the output of the text on the command line.

If you run it as a job under Jenkins, the command line is similar. In order for Jenkins to render visual report, just add the "format" parameter, indicating the output file in “json” format. The command line is as follows:

node_modules\.bin\cucumber-js --format json:report.json

It will output report to report.json and then this file will be used by Jenkins job to show the report.

CukeTest's command line mode

CukeTest is a development tool of Cucumber.js. It is available in Windows Desktop, Windows Store, MacOS, etc., all these editions support command line mode. It can be downloaded from http://cuketest.com. CukeTest is used to develop Cucumber.js scripts, it can also be used to execute Cucumber.js scripts from the command line. The advantage of using CukeTest is you don't have to install Cucumber.js for each project, CukeTest already has built-in Cucumber.js, which saves your disk space and time, other advantage are it can run Windows automation with the built in library, record video etc.

The Windows Store version of CukeTest has different mechanism for command line, therefore the command line is a little different from the desktop version. Luckily you don’t need to remember all the command line details. In CukeTest, when your project is opened, you can open a run profile to get the command line, by clicking the drop-down arrow besides Run project button, and then clicking “Edit Profiles…”.

In “Run Profiles” dialog, the corresponding command line is automatically generated in the command line bar for this profile:

Click the “Copy to clipboard” button to copy the commands to the clipboard, click the “Open console window” button, and paste the command line to the newly opened window.

Here are the command lines on different platforms:

CukeTest Windows Store version

Here is the command line for Windows Store version:

chcp 65001 && start cuketest://?profile=Profile1

"chcp 65001" changes the command line encoding to UTF8 because CukeTest's command line output is encoded in UFT8.

CukeTest desktop version
The following is the command to run CukeTest on the Windows desktop:

chcp 65001 && cuke --run --format html

Jenkins continuous integration configuration

After we know how to run the project in command line, we can configure Jenkins to run it. Below explains the steps to configure the Jenkins job.

We are using the source code in https://github.com/CnodejsTest for demo. This project can be used to automate test cnodejs.org community site. Feel free to use your own Cucumber project to configure the job.

Here are the steps to configure the job:

1. Create a new job in Jenkins task:

   

2. Select "Windows Batch Command" in the item bar.

   Copy and paste the corresponding command line from CukeTest to the Jenkins batch command box. For example, I installed the CukeTest desktop version natively. Here is my command line:

   npm install
   chcp 65001 && cuke --run --format html
   

   The first command installs npm dependency packages that are used by the automation script. For example, for a Web project, it may includes selenium-webdriver, chromedriver, etc.

   The following figure is the job configuration UI with the commands entered:

   

3. Save the project configuration. Click [Build Now] and the project will start executing.

You can see the execution progress of the job in the build history:

You can also find the running log of the job in [Build History]. At the same time, the report file is also generated in the Jenkins Workspace directory. Our next step is to configure Jenkins job to show the test report on the Jenkins interface, in short, we will use the Jenkins plugin to format and show the test report.

Configure Jenkins to show the test report

CukeTest is using Cucumber.js, therefore will generate the same json reporting format as Cucumber. In Jenkins we can use the Cucumber reports plugin of Jenkins to show the report for both of them.
Plugin introduction: https://wiki.jenkins.io/display/JENKINS/Cucumber+Reports+Plugin

The following are the steps:

1. Install the Jenkins reports plugin
   Open Jenkins [System Management] - [Manage Plugin], and filter "Cucumber reports" in [Optional Plugin].

Restart Jenkins after this plugin is installed.

2. Reconfigure the Jenkins task

   The Cucumber reports plugin generates reports by parsing the json report file that Cucumber or CukeTest create. Therefore we need to first configure running the project in order to generate json log data files.

   Select "json" in the report format in the run configuration file.

   

   The command line execution output is changed to json:

   chcp 65001 && cuke --run --format json
   

3. Display [Cucumber reports]

   In the [Add post-build operation steps] select the [Cucumber reports] option.

   

   Open the Cucumber reports "Advanced..." option. Enter "reports" in "Json Reports Path":

   

   Click "Save" to complete the project configuration.

4. Generate test report

   Click "Build Now" and the test report shown at the beginning of this article will be generated automatically after the job finishes running. In addition, the test report provides a link to each scenario, and when clicked, it shows the detail of the scenarios.

   

   The report not only can show the execution result of each step, but also shows the screenshot captured in by the scenario steps or hooks:

   

Summary

This blog introduces how to configure Jenkins to run Cucumber or CukeTest for automation. After integrating with CI process, project stakeholders can all read the Cucumber test report and understand the quality status. Jenkins also has many other functions, such as setting up timed operation, mail sending and other functions, you can configure according to the project needs, to further customize the report notification, for example, send html report as an email to key members.

Related readings:
here are some further readings on creating Cucumber test:

• BDD Web Automation Series
• Windows Automation Series
• Create and Run API Automation Script with Cucumber