Setup & Configure Jenkins for Your Team
Background
Nowadays, continuous integration is an important part of the agile software development life-cycle. There is a number of tools on the market: Atlassian Bamboo, Jenkins, Jetbrains TeamCity.
In my opinion, Jenkins has the most optimal product community and set of really useful plugins that suits most of your software projects: you can build software, deploy software, websites, portals to various places (e.g including AWS, DigitalOcean, bare metal servers) or to run unit tests. It can be integrated with communication tools of your choice, like Slack, HipChat or email.
If you haven't had a chance to try Jenkins earlier, feel free to use the tutorial below to get started.
Manual installation
In order to install Jenkins, we will need:
- Unix system. I would recommend a Debian-based machine, like Ubuntu server LTS
- Java runtime environment installed. I usually use Java 8
- Get base Jenkins setup
- Install necessary plugins
- Put everything behind your web server.
Install Java
The easiest way to install Java is through the apt-get
package manager:
sudo apt-get install python-software-properties
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
Once you added ppa
above, you can install java with the following command:
sudo apt-get install oracle-java8-installer
Get base Jenkins Setup
You will need to execute a series of the following commands, namely: add the Jenkins signing key, register Jenkins apt sources, update package lists, and install Jenkins package.
wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
sudo echo deb http://pkg.jenkins-ci.org/debian binary/ > /etc/apt/sources.list.d/jenkins.list
sudo apt-get update
sudo apt-get install jenkins
By default, it will install the base Jenkins setup, which is insecure. You will need to go to the host where your Jenkins is installed, for example: http://jenkins-host:8080/
.
Navigate to Manage Jenkins (on the left) and choose the "Configure Global Security" item on the page loaded.
Now look below on the Matrix-based security (select it, if it is not selected previously), and make sure Anonymous only has the "Read" right under the View group.
Click Save at the bottom of the page. After the page reloads, you'll see a login form, but simply ignore that and
go to the home page (like, for example, http://jenkins-host:8080/). You'll see this signup form, and the first signed up account will be the administrator.
The Power of Plugins
Jenkins would not be so powerful without plugins. Usually, I install these plugins by default:
-
Bitbucket
:
The BitBucket plugin is designed to offer integration between BitBucket and Jenkins. BitBucket offers a Jenkins hook, but this one just triggers a build for a specific job on commit, nothing more. The BitBucket plugin, like the GitHub plugin, uses the POST hook payload to check which job has to get triggered based on the changed repository/branch.
Plugin URL: https://wiki.jenkins-ci.org/display/JENKINS/BitBucket+Plugin -
bitbucket-pullrequest-builder
:
This plugin builds pull requests from Bitbucket.org. This is a must-have plugin if you perform QA deploy for each submitted pull request.
Plugin URL: https://wiki.jenkins-ci.org/display/JENKINS/Bitbucket+pullrequest+builder+plugin -
build-pipeline-plugin
:
This plugin provides a Build Pipeline View of upstream and downstream connected jobs that typically form a build pipeline. In addition, it offers the ability to define manual triggers for jobs that require intervention prior to execution, e.g. an approval process outside of Jenkins. Provides nice visualization of the paths & flows.
Plugin URL: https://wiki.jenkins-ci.org/display/JENKINS/Build+Pipeline+Plugin -
copyartifact
:
Adds a build step to copy artifacts from another project. The plugin lets you specify which build to copy artifacts from (e.g. the last successful/stable build, by build number, or by a build parameter). You can also control the copying process by filtering the files being copied, specifying a destination directory within the target project, etc.
Plugin URL: https://wiki.jenkins-ci.org/display/JENKINS/Copy+Artifact+Plugin -
credentials
:
Adds a build step to copy artifacts from another project. The plugin lets you specify which build to copy artifacts from (e.g. the last successful/stable build, by build number, or by a build parameter). You can also control the copying process by filtering the files being copied, specifying a destination directory within the target project, etc.
Plugin URL: https://wiki.jenkins-ci.org/display/JENKINS/Credentials+Plugin -
delivery-pipeline-plugin
:
Visualisation of Delivery/Build Pipelines, renders pipelines based on upstream/downstream jobs. When using Jenkins as a build server it is now possible with the Delivery Pipeline Plugin to visualise one or more Delivery Pipelines in the same view even in full screen.
Plugin URL: https://wiki.jenkins-ci.org/display/JENKINS/Delivery+Pipeline+Plugin -
environment-script
:
Environment Script Plugin allows you to have a script run after SCM checkout, before the build. If the script fails (exit code isn't zero), the build is marked as failed.
Any output on standard out is parsed as environment variables that are applied to the build. It supports "override syntax" to append paths to PATH-like variables.
Plugin URL: https://wiki.jenkins-ci.org/display/JENKINS/Environment+Script+Plugin -
git
:
Supports popular git version control system -
ghprb
:
This plugin builds pull requests in GitHub. It's another must-have plugin if your software development life cycle includes deploying pull requests to PR environment to test.
Plugin URL: https://wiki.jenkins-ci.org/display/JENKINS/GitHub+pull+request+builder+plugin -
greenballs
: The funniest plugin - changes Jenkins to use green balls instead of blue for successful builds.
Plugin URL: https://wiki.jenkins-ci.org/display/JENKINS/Green+Balls -
hipchat
:
This plugin allows your team to setup build notifications to be sent to HipChat rooms.To enable notifications, add "HipChat Notifications" as a post-build step.
Plugin URL: https://wiki.jenkins-ci.org/display/JENKINS/HipChat+Plugin -
junit
:
Allows JUnit-format test results to be published. Note: number of tools, including Karma, PhpUNIT & other tools allow to publish test results in a JUnit format. Thus, this is a must-have plugin for unit test flows.
Plugin URL: https://wiki.jenkins-ci.org/display/JENKINS/JUnit+Plugin -
matrix-auth
:
Offers matrix-based security authorization strategies (global and per-project). This is quite handy if you have shared build server across several teams.
Plugin URL: https://wiki.jenkins-ci.org/display/JENKINS/Matrix+Authorization+Strategy+Plugin -
parameterized-trigger
:
This plugin lets you trigger new builds when your build has completed, with various ways of specifying parameters for the new build.
You can add multiple configurations: each has a list of projects to trigger, a condition for when to trigger them (based on the result of the current build), and a parameters section.
Plugin URL: https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Trigger+Plugin -
rebuild
:
This plays nice with the parameterized-trigger plugin. The plug-in allows the user to rebuild a parametrized build without entering the parameters again.
Plugin URL: https://wiki.jenkins-ci.org/display/JENKINS/Rebuild+Plugin -
ssh
: You can use the SSH Plugin to run shell commands on a remote machine via ssh.
Plugin URL: https://wiki.jenkins-ci.org/display/JENKINS/SSH+plugin -
s3
: Allows uploading artifacts to S3 with multiple options.
Plugin URL: https://wiki.jenkins-ci.org/display/JENKINS/S3+Plugin -
throttle-concurrents
:
This plugin allows for throttling the number of concurrent builds of a project running per node or globally.
Unfortunately, this is also a must-have plugin for Node (0.10-0.12) projects with NPM - two concurrent npm installs often fail.
Plugin URL: https://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin
Plugins are installed using Plugin manager on a Manage Jenkins Section.
Putting Jenkins Behind a Web Server
Usually I hide Jenkins behind nginx. A typical configuration looks like the one below:
server {
listen 443 ssl;
server_name jenkins.vagrant.dev;
ssl_certificate /etc/nginx/jenkins_selfsigned.crt;
ssl_certificate_key /etc/nginx/jenkins_selfsigned.key;
location / {
proxy_pass http://127.0.0.1:8080;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_redirect off;
proxy_connect_timeout 150;
proxy_send_timeout 100;
proxy_read_timeout 100;
}
...
}
Automated installation
Do I install Jenkins manually each time? Of course not, I do it often for my customers.
With ansible, and sa-box-jenkins role new Jenkins installation can be deployed while you drink your coffee.
Let's prepare a basic bootstrap project that can be used by you in the future.
It includes following files:
- bootstrap.sh - installs ansible alongside with dependencies.
- init.sh - initializes 3rd party dependencies
- .projmodules - fully compatible with
.gitmodules
git syntax, specifies list of the dependencies
that will be used by the playbook.
In particular, it includes ansible- by default developer_recipes (repository with set of handy deployment recipes)
and ansible role called sa-box-bootstrap responsible for box securing steps (assuming you plan to put Jenkins on remote hosts).
[submodule "public/ansible_developer_recipes"]
path = public/ansible_developer_recipes
url = git@github.com:Voronenko/ansible-developer_recipes.git
[submodule "roles/sa-box-bootstrap"]
path = roles/sa-box-bootstrap
url = git@github.com:softasap/sa-box-bootstrap.git
[submodule "roles/sa-box-jenkins"]
path = roles/sa-box-jenkins
url = git@github.com:softasap/sa-box-jenkins.git
- hosts - list here the initial box credentials that were provided to you for the server. Note: jenkins-bootstrap assumes you have the fresh box with root access only. If your box already secured, adjust credentials appropriately
[jenkins-bootstrap]
jenkins_bootstrap ansible_ssh_host=192.168.0.17 ansible_ssh_user=yourrootuser ansible_ssh_pass=yourpassword
[jenkins]
jenkins ansible_ssh_host=192.168.0.17 ansible_ssh_user=jenkins
- jenkins_vars.yml - set here specific environment overrides, like your preferred deploy user name and keys.
- jenkins_bootstrap.yml - First step - box securing. Creates jenkins user, and secures the box using sa-box-bootstrap role.
See more details about the sa-box-bootstrap role
In order, to override params for sa-box-bootstrap - pass the parameters like in the example below:
- hosts: all
vars_files:
- ./jenkins_vars.yml
roles:
- {
role: "sa-box-bootstrap",
root_dir: "{{playbook_dir}}/public/ansible_developer_recipes",
deploy_user: "{{jenkins_user}}",
deploy_user_keys: "{{jenkins_authorized_keys}}"
}
- jenkins.yml provisioning script that configures jenkins with set of plugins and users.
- jenkins_vars.yml configuration options for jenkins deployment.
- setup_jenkins.sh shell script that invokes deployment in two steps: initial box bootstraping & jenkins setup
#!/bin/sh
ansible-playbook jenkins_bootstrap.yml --limit jenkins_bootstrap
ansible-playbook jenkins.yml --limit jenkins
Configuration Options for Automated Installation
You need to override:
jenkins_authorized_keys
(this is list of the keys, that allow you to login to Jenkins box under Jenkins)jenkins_domain
- your agency domainjenkins_host
- name of the Jenkins host (Site will be bound to jenkins_host.jenkins_domain)java_version
- your Java choice (6,7,8 supported)
jenkins_user: jenkins
jenkins_authorized_keys:
- "{{playbook_dir}}/components/files/ssh/vyacheslav.pub"
jenkins_domain: "vagrant.dev"
jenkins_host: "jenkins"
java_version: 8
-jenkins_users
list of users with passwords to create. Admin and deploy are required users.
Admin is used to manage instance, deploy is used to access the artifacts via deployment scripts.
If you won't override passwords, the default one will be used (per role), which is not the best, for public deployments.
jenkins_users:
- {
name: "Admin",
password: "AAAdmin",
email: "no-reply@localhost"
}
- {
name: "deploy",
password: "DeDeDeDeploy",
email: "no-reply@localhost"
}
jenkins_plugins
Your choice of plugins to install. By default:
jenkins_plugins:
- bitbucket # https://wiki.jenkins-ci.org/display/JENKINS/BitBucket+Plugin
- bitbucket-pullrequest-builder
- build-pipeline-plugin
- copyartifact # https://wiki.jenkins-ci.org/display/JENKINS/Copy+Artifact+Plugin
- credentials # https://wiki.jenkins-ci.org/display/JENKINS/Credentials+Plugin
- delivery-pipeline-plugin # https://wiki.jenkins-ci.org/display/JENKINS/Delivery+Pipeline+Plugin
- environment-script # https://wiki.jenkins-ci.org/display/JENKINS/Environment+Script+Plugin
- git
- ghprb # https://wiki.jenkins-ci.org/display/JENKINS/GitHub+pull+request+builder+plugin
- greenballs # https://wiki.jenkins-ci.org/display/JENKINS/Green+Balls
- hipchat # https://wiki.jenkins-ci.org/display/JENKINS/HipChat+Plugin
- junit # https://wiki.jenkins-ci.org/display/JENKINS/JUnit+Plugin
- matrix-auth # https://wiki.jenkins-ci.org/display/JENKINS/Matrix+Authorization+Strategy+Plugin
- matrix-project #https://wiki.jenkins-ci.org/display/JENKINS/Matrix+Project+Plugin
- parameterized-trigger #https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Trigger+Plugin
- rebuild # https://wiki.jenkins-ci.org/display/JENKINS/Rebuild+Plugin
- ssh
- s3 # https://wiki.jenkins-ci.org/display/JENKINS/S3+Plugin
- throttle-concurrents #https://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin
The Code in Action
You can download this template code from this repository. In order to use it - fork it, adjust parameters to your needs, and use.
Running is as simple as
./setup_jenkins.sh
Welcome to the world of continuous integration & deployment!
Nice Tutorial. One thing I wish someone would write a good tutorial about (now that I think about it this might be a good topic for me to write about) is a tutorial for getting a Jenkins instance up and running with Chef. Not just the base Jenkins install but a really honed and capable system.
I think, they have ready to use package already.
I used to use CHEF previously, but find that bootstrapping box for CHEF (both provisioned box & provisioner) takes more efforts than ansible requires. Thus last 2 years most of mine deployments are ansible based.
and, btw using recipe above you get fully operational Jenkins automatically.
I am working on backup / restore scripts to save user data in case of failure reinstall, but haven’t finished yet
Nice, I’ll check it out when you’re finished. I agree that Chef can be a beast, however, many companies I’ve worked with in the past have been using it for some time, as before Chef there was little in the way as far as change management tools, and thus I am constrained with their work. However, my personal preference is quickly becoming Salt Stack. How does Ansible compare with that tool? I have heard great things about it, but have only briefly used it in the past. I’d be curious to see what benefits you’re seeing there.
Yep, CHEF is considered “enterprise level”, usually you can target with one chef playbook multiple os-es w/o specific issues + Amazon likes CHEF too.
I would say Salt Stack & Ansible are same level players.
Major difference I see - Salt is built around a Salt master and multiple Salt minions , while Ansible is master less (if you need master => minions you implement smth on your own). Can’t say if it is pro or cons.
From other hand, my recent customers use openStack, Amazon - usually deployment happen using platform api, and master is not really needed. - Ansible goes pretty well for me as for now.
Cool - thanks for the info. I have found with Chef, that while yes it is very ‘enterprisey’ with lots of big solutions for problems, many people choose to use alternative tools to accomplish things better than Opscode can (because opscode has a obvious incentive to give companies solutions).
One of those is Berkshelf which is a better dependency and Cookbook architecture in my opinion. It allows for a more graceful solution than what you would get by using Chef + all the chef methods.
Chef solo or Chef zero is another tool that many people don’t realize is extremely valuable either. By baking cookbooks into an AMI or any disk image for that matter and using Chef zero, the dependency on a Chef server is removed, and thus you are free to not have per server costs. Many of these tricks exist but of-course large companies want the confidence of an SLA and all that, so they tend to be not well known.