This is how to achieve multiple job concurrency with GitLab
Gitlab is used by indie hackers but also by big companies with hundreds of devs.
A single dev hacking on her project can push changes every now and then, but when that is multiplied tenfold, quickly becomes overwhelming for a build server.
In some environments there are just too many devs working on so many projects pushing changes every second of the day!
You seem familiar with what I am describing. Tired of waiting for those build jobs to finish, right?
The solution is simple, increase the concurrency ... :]
Note: In this post, we will take into account multiple projects that are served by one runner with
docker
as executor. That runner will run the build jobs, based on the concurrency we will set. Different setups for different case studies are naturally feasible.
For the sake of completeness, we will set it all up from scratch.
For this guide we'll use:
-
GitLab Enterprise Edition 10.8.4-ee
-
gitlab-runner 10.6.0
Just a side note...:
I love automation so I am writing a book about it!
It is packed with comprehensive guides on the concepts of CI/CD, implemented with techs like Kubernetes, Helm, Docker, Gitlab, Draft, Scaffold etc.
Do you want to be notified as soon as it is available? Get notified!
Do you know a friend/colleague that will be interested? Share it!
Step 1: Create the gitlab-runner container
Note: you may need to be
root
docker run --name runner --restart always -d -v $HOME/gitlab-runner:/etc/gitlab-runner -v /var/run/docker.sock:/var/run/docker.sock gitlab/gitlab-runner:v10.6.0
Attention: this is NOT a registered runner. It is only the gitlab-runner binary running inside a docker container!
Step 2: Register a runner to GitlabCI
gitlab-runner register --non-interactive --description "docker-runner" --url "http://www.your-gitlab.com/" -registration-token "something_weird" --executor "docker" --docker-image "docker:latest" --run-untagged --locked=false --docker-privileged=true --docker-volumes /var/run/docker.sock:/var/run/docker.sock
Note: the
--locked=false
argument does not work. The runner will still be registered asspecific
. I include it in case a future upgrade fixes it.
Optionally, you can register more than one runners with the same [or different] name, if you want to lock them per project.
config.toml
Step 3: Increase the concurrency in the The concurrent
setting, controls how many jobs can be run concurrently.
Globally. That means, it applies to all the runners independently of the executor [docker, ssh, kubernetes etc]
You should open the /etc/gitlab-runner/config.toml
file with an editor like nano
or vim
.
And change the concurrent = 1
to something like concurrent = 5
[or as many concurrent builds you like]
Check here for more info.
Step 4: Restart gitlab-runner
Just to be safe and not end up banging our head...
gitlab-runner restart
Step 5: Run some builds
Now you can run as many jobs you set the concurrency to be [i.e n]! Try to overwhelm it with n+1, just to watch the +1 job to wait patiently for a slot to open.
Note: when the
docker
executor is used, every new build job will run inside a new docker container that will spawn automagically. The same logic applies with thekubernetes
executor too.
Optional step: Clean up
There will be a LOT of (depending on your projects) cache containers that remain exited when the build jobs succeed.
Check it out below!
2ca3a04e44dd 88a04ddd0898 "gitlab-runner-cache " 10 minutes ago Exited (0) 10 minutes ago runner-707174de-project-38-concurrent-0-cache-3c3f0...
8beb199e0e1f 88a04ddd0898 "gitlab-runner-cache " 10 minutes ago Exited (0) 10 minutes ago runner-707174de-project-38-concurrent-0-cache-d6b5f...
a7f0bf9ba8b1 88a04ddd0898 "gitlab-runner-cache " 10 minutes ago Exited (0) 10 minutes ago runner-f2bd384a-project-46-concurrent-0-cache-3c3f0...
66bb3ef98879 88a04ddd0898 "gitlab-runner-cache " 10 minutes ago Exited (0) 10 minutes ago runner-f2bd384a-project-46-concurrent-0-cache-d6b5f...
3689d386af1b 88a04ddd0898 "gitlab-runner-cache " 18 minutes ago Exited (0) 18 minutes ago runner-8df2f9cc-project-19-concurrent-0-cache-3c3f0...
1391db607410 88a04ddd0898 "gitlab-runner-cache " 18 minutes ago Exited (0) 18 minutes ago runner-8df2f9cc-project-19-concurrent-0-cache-d6b5f...
These containers have to remain after the build, so that the next build can use the cache.
Removing it after a build would defeat its purpose. However, if you notice that you run out of space and strange behaviour occurs...or you have just a tiny OCD, run this:
docker run -d -e LOW_FREE_SPACE=10G -e EXPECTED_FREE_SPACE=20G -e LOW_FREE_FILES_COUNT=1048576 -e EXPECTED_FREE_FILES_COUNT=2097152 -e DEFAULT_TTL=10m -e USE_DF=1 --restart always -v /var/run/docker.sock:/var/run/docker.sock --name=gitlab-runner-docker-cleanup quay.io/gitlab/gitlab-runner-docker-cleanup
Note: for more info check here
Hi I’m using gitlab enterprise edition 15.6, with gitlab-runner installed on amazon linux 2 instance and I have around 6 individual project runners registered on the same instance. I’ve left the config.toml to default opitons including concurrent = 1 globally. but in the builds directory, i can see 4 builds. But the issue is these runners take a lot of time to pick up the jobs. The jobs stay pending for anywhere between 2 to 10 minutes. Any idea what could be the reason?