This is how to achieve multiple job concurrency with GitLab

Gitlab is used by indie hackers but also by big companies with hundreds of devs.
A single dev hacking on her project can push changes every now and then, but when that is multiplied tenfold, quickly becomes overwhelming for a build server.

In some environments there are just too many devs working on so many projects pushing changes every second of the day!
You seem familiar with what I am describing. Tired of waiting for those build jobs to finish, right?
The solution is simple, increase the concurrency ... :]

Note: In this post, we will take into account multiple projects that are served by one runner with docker as executor. That runner will run the build jobs, based on the concurrency we will set. Different setups for different case studies are naturally feasible.

For the sake of completeness, we will set it all up from scratch.

For this guide we'll use:

• GitLab Enterprise Edition 10.8.4-ee

• gitlab-runner 10.6.0

Just a side note...:
I love automation so I am writing a book about it!
It is packed with comprehensive guides on the concepts of CI/CD, implemented with techs like Kubernetes, Helm, Docker, Gitlab, Draft, Scaffold etc.
Do you want to be notified as soon as it is available? Get notified!
Do you know a friend/colleague that will be interested? Share it!

Step 1: Create the gitlab-runner container

Note: you may need to be root

docker run --name runner --restart always -d -v $HOME/gitlab-runner:/etc/gitlab-runner -v /var/run/docker.sock:/var/run/docker.sock gitlab/gitlab-runner:v10.6.0

Attention: this is NOT a registered runner. It is only the gitlab-runner binary running inside a docker container!

Step 2: Register a runner to GitlabCI

gitlab-runner register --non-interactive --description "docker-runner" --url "http://www.your-gitlab.com/" -registration-token "something_weird" --executor "docker" --docker-image "docker:latest" --run-untagged --locked=false --docker-privileged=true --docker-volumes /var/run/docker.sock:/var/run/docker.sock

Note: the --locked=false argument does not work. The runner will still be registered as specific. I include it in case a future upgrade fixes it.

Optionally, you can register more than one runners with the same [or different] name, if you want to lock them per project.

Step 3: Increase the concurrency in the config.toml

The concurrent setting, controls how many jobs can be run concurrently.
Globally. That means, it applies to all the runners independently of the executor [docker, ssh, kubernetes etc]

You should open the /etc/gitlab-runner/config.toml file with an editor like nano or vim.

And change the concurrent = 1 to something like concurrent = 5 [or as many concurrent builds you like]

Check here for more info.

Step 4: Restart gitlab-runner

Just to be safe and not end up banging our head...

gitlab-runner restart

Step 5: Run some builds

Now you can run as many jobs you set the concurrency to be [i.e n]! Try to overwhelm it with n+1, just to watch the +1 job to wait patiently for a slot to open.

Note: when the docker executor is used, every new build job will run inside a new docker container that will spawn automagically. The same logic applies with the kubernetes executor too.

Optional step: Clean up

There will be a LOT of (depending on your projects) cache containers that remain exited when the build jobs succeed.

Check it out below!

2ca3a04e44dd  88a04ddd0898  "gitlab-runner-cache " 10 minutes ago  Exited (0) 10 minutes ago  runner-707174de-project-38-concurrent-0-cache-3c3f0...
8beb199e0e1f  88a04ddd0898  "gitlab-runner-cache " 10 minutes ago  Exited (0) 10 minutes ago  runner-707174de-project-38-concurrent-0-cache-d6b5f...
a7f0bf9ba8b1  88a04ddd0898  "gitlab-runner-cache " 10 minutes ago  Exited (0) 10 minutes ago  runner-f2bd384a-project-46-concurrent-0-cache-3c3f0...
66bb3ef98879  88a04ddd0898  "gitlab-runner-cache " 10 minutes ago  Exited (0) 10 minutes ago  runner-f2bd384a-project-46-concurrent-0-cache-d6b5f...
3689d386af1b  88a04ddd0898  "gitlab-runner-cache " 18 minutes ago  Exited (0) 18 minutes ago  runner-8df2f9cc-project-19-concurrent-0-cache-3c3f0...
1391db607410  88a04ddd0898  "gitlab-runner-cache " 18 minutes ago  Exited (0) 18 minutes ago  runner-8df2f9cc-project-19-concurrent-0-cache-d6b5f...

These containers have to remain after the build, so that the next build can use the cache.
Removing it after a build would defeat its purpose. However, if you notice that you run out of space and strange behaviour occurs...or you have just a tiny OCD, run this:

docker run -d -e LOW_FREE_SPACE=10G -e EXPECTED_FREE_SPACE=20G -e LOW_FREE_FILES_COUNT=1048576 -e EXPECTED_FREE_FILES_COUNT=2097152 -e DEFAULT_TTL=10m -e USE_DF=1 --restart always -v /var/run/docker.sock:/var/run/docker.sock --name=gitlab-runner-docker-cleanup quay.io/gitlab/gitlab-runner-docker-cleanup

Note: for more info check here