Few months before we came up with an idea to scale our Jenkins infrastructure by dynamically launching new containers on AWS cloud in order to build and verify pull requests. It would help our developers to avoid long waiting time (on average 4-6 h!!!) before their pull requests could be merged, since we had limited number of Jenkins slaves and every time we wanted to add a new slave, our Ops have to go all over again to set up a new Jenkins slave.
From developer’s point of view, it means packing one really big and fat monolithic app (along with all its dependencies like messaging queues, database server, configuration management, backend API, validation API, etc.) on a docker container. All those services need to run locally inside a container, that way we could scale on demand, i.e. launch N number of such containers depending upon the number of pull requests. Hence, no waiting time (as soon as the pull request arrives, Jenkins can start building it).
We first started development locally on our laptops and after battling for weeks, everything worked like a charm on developer’s machines.
Then we tried to launch our docker container on AWS cloud. We thought that everything would work, since we had already built and tested docker containers locally on multiple flavors of Linux versions running different kernel versions, but the reality was quite the opposite, after launching the container it just halted for some strange reasons.
After investigating, we found the culprits were kernel parameters, to be precise, too small shmmax and shmall parameters . You may ask, why a docker container required more shmmax and shmall. Remember our big and fat monolithic application ? It required a lot of dependencies to be installed inside the container, few of those dependencies required access to set bigger values for shmmax and shmall in init-scripts.
In order to run those applications, we were forced to launch docker in privileged mode like this
docker run --privileged -it someDockerImage
After that, we could run those applications without modifying init-scripts.
Note: Although the above solution works, if possible one should really avoid launching containers in privileged mode because root inside docker container == root on host.