Scaling Jenkins on Kubernetes

-

Scaling Jenkins on Kubernetes


Jenkins running on K8s

Why do we want to run Jenkins in kubernetes? That might be your first question. You might not find a requirement to run Jenkins in K8s strait away. When your codebase grows larger or run many jobs parallelly, Jenkins will grow very largely. This will slow down your builds and lead to unnecessary resource utilization. Let’s assume that you have 3 Jenkins slaves and each can run 3 job parallelly. Then you can run maximum 9 jobs parallelly, other jobs have to wait. Solution for these is scaling Jenkins.
Scaling Jenkins is vey easy. Jenkins has scaling feature out-fo-the-box. Jenkins comes with master/slave mode. Master is responsible for maintaining jobs, users, configurations, scheduling jobs in slaves and etc... Slaves are Jenkins agents, their primary task is executing jobs scheduled by the master.
Every one knows K8s is container orchestration platform. So I’m going to implement this solution in K8s using its features.

Setting-up Jenkin in K8s

  • Jenkins Master
First we have to create a Jenkins master. In order to do this we are going to use official Jenkins docker image. This docker only provides Jenkins server so we have to create a custom docker image from official image.
FROM jenkins/jenkins:2.176.1
# Distributed Builds plugins (1)
RUN /usr/local/bin/install-plugins.sh ssh-slaves
#Install GIT (2)
RUN /usr/local/bin/install-plugins.sh git
# Artifacts
RUN /usr/local/bin/install-plugins.sh htmlpublisher
# UI (3)
RUN /usr/local/bin/install-plugins.sh greenballs
RUN /usr/local/bin/install-plugins.sh simple-theme-plugin
# Scaling (4)
RUN /usr/local/bin/install-plugins.sh kubernetes
VOLUME /var/jenkins_home (5)
USER jenkins
You can find this docker file here. So what we are doing here is installing distribution plugin, git plugin, UI and K8s plugin. And one important is (5), docker volume for Jenkins home. I’ll explain this later.
Now we are going to deploy this docker image in to K8s. In order to do that we are going to create a K8s Deployment, Service and Ingress.
Below is the Deployment for Jenkins master. ( My Jenkins master image is keaz/jenkins-master, you can use your own image in the deployment).
kind: Deployment
metadata:
name: "jenkins-master-deployment"
labels:
app: jenkins-master
version: "latest"
group: "jenkins"
namespace: "jenkins"
spec:
replicas: 1
selector:
matchLabels:
app: "jenkins-master"
version: "latest"
group: "jenkins"
template:
metadata:
labels:
app: jenkins-master
version: "latest"
group: "jenkins"
spec:
containers:
- name: "jenkins-master"
image: "keaz/jenkins-master:latest"
imagePullPolicy: "IfNotPresent"
env:
- name: JAVA_OPTS
value: -Djenkins.install.runSetupWizard=false
ports:
- name: http-port
containerPort: 8080
- name: jnlp-port
containerPort: 50000
volumeMounts:
- name: jenkins-home
mountPath: "/var/jenkins_home"
volumes:
- name: jenkins-home
persistentVolumeClaim:
claimName: "jenkins-volume-claim"
readOnly: false
Important thing of this Deployment is the volumeMounts. Here we are using a pvc( Persistent Volume Claim) for the volume we created in Docker file. All the Jenkins configurations, jobs, user detail etc.. are stored in Jenkins home. It is important to mount this into a pvc, otherwise we will lose everything when the Jenkins master pod get restarted.
Then we are going to create Service and an Ingress rule for Jenkins.
K8s Service
apiVersion: "v1"
kind: "Service"
metadata:
labels:
app: "jenkins-master"
version: "latest"
group: "jenkins"
namespace: "jenkins"
name: "jenkins-master-service"
spec:
ports:
- name: "http"
port: 80
targetPort: 8080
- name: "jnlp"
port: 50000
targetPort: 50000
selector:
app: "jenkins-master"
version: "latest"
group: "jenkins"
type: "ClusterIP"
K8s Ingress
apiVersion: "extensions/v1beta1"
kind: "Ingress"
metadata:
labels:
app: "jenkins-master"
version: "latest"
group: "jenkins"
namespace: "jenkins"
name: "jenkins-master-ingress"
spec:
rules:
- host: "jenkins"
http:
paths:
- backend:
serviceName: "jenkins-master-service"
servicePort: 80
path: "/"
If you use minikube or Docker for Mac you have to add a host entry in /etc/hosts file like below.
127.0.0.1 jenkins
Now you can access Jenkins via http://jenkins .

Jenkins running in K8s

  • Jenkins Slave
Then we have to create Jenkins salve Image. We can use the official Docker Jenkins image for this. But it lacks very important thing that is ability to execute docker commands. Because of that we are going to create a custom docker image for Jenkins slave.
We can run docker inside a docker container. But it is not recommended, this article explains it. What we are going to do is accessing the Docker deamon in the host system via Docker container. We can achieve this by mounting Docker socket to our container. I have created a Dockefile for this. You can find it here. What I’m doing in this Dockerfile is installing docker and then setting up Jenkins slave. Jenkins slave setting up part is taken from two official Jenkins salve Dockerfiles.
First we have to configure Jenkins Kubernetes plugin. In order to do this, navigate to “Manage Jenkins -> Configure System” and then scroll to the “Cloud” section.


Click the “Add a new Cloud” and select “Kubernetes”.

Jenkins Cloud in Configuration

In the cloud configuration you have to set the Kubernetes URL, Kubernetes Namespace, Jenkins URL. You can get the K8s url from below command.
Kasuns-MacBook-Pro:~ kasunranasinghe$ kubectl cluster-info | grep master
Kubernetes master is running at https://kubernetes.docker.internal:6443
Kasuns-MacBook-Pro:~ kasunranasinghe$
For Jenkins URL set Jenkins K8s service name. Set “Kubernetes Namespace” as Jenkins master namespace.

Kubernetes configurations

Next we are going to create “Pod Template”. Click “Add Pod Template” and select “Kubernetes Pod Template”.


Set the “Name, Namespace, Labels and Usage” as below image.


Then we are going to add our custom image as Jenkins slave. Select “Add Container” and then select “Container Template”.


Setup the container configuration as below image.

Container Template

It is important to set Name as jnlp. Because Jenkins always creates a container called jnlp and executes jobs. We do this to override Jenkins default behaviour.
Then we are going to mount host docker socket into Jenkins slave container. Go to “Volumes” click “Add Volume” and then select “Host Path Volume”. Set the /var/run/docker.sock both of Host path and Mount path. Then save the changes.

Mount Host Docker into container

Now we have completed the Jenkins setup.
  • Testing the Jenkins Setup
We are going to create a simple job that executes docker image ls in a Shell script.


When you execute the job Jenkins put that job into a queue until a slave pod starts.


You can see the Jenkins slave pod running in the K8s using this command kubectl get pods -n jenkins.

Jenkins slave running in K8s

When you check the Console Output of the job, you can see Kubernetes template for the slave pod.

Jenkins slave pod template generated by Jenkins

We can see the output of the docker command we executed in the job below Kubernetes pod template.

The output of the build

One other important thing is you can notice is that as soon as job completes, the Jenkins slave get removed from the cluster. Jenkins only spins up slaves when needed, this will avoid unnecessary resource utilization.

Conclusion

Running Jenkins in a K8s cluster has huge advantages.
  • Automatically spin up and remove slaves, this reduce unnecessary resource utilization and save cost.
  • Able to run many jobs parallelly.
But creating a Jenkins Docker images according to your requirements is not straightforward, like in this example I had to create a custom Jenkins slave image to add docker support and I did not talk about how we can clone git project using ssh (Will update once I complete that).
keaz/jenkins

Comments

Post a Comment

Popular posts from this blog

Docker sizing Java application

-
Image
Many developers put java in docker yet they face many issues from development to running containers. Docker sizing Java application As  a Java developer, I faced many issues when dockerizing and running java applications. Running dockrized java application When we start our development we used java 8 as our runtime environment. We ran jars as separate processes (yes using java -jar ) in VM, at that time there was no requirement to dockerize applications and everything worked without any major issues. Later we experienced the need for dockerizing applications. Then we faced many performance issues. Our initial docker environment was just docker containers running in a VM without any container orchestration. When we compared dockerized application with normal java application, the performance of dockerized application was not good. When we set the memory limits for docker application, most of the applications threw  java.lang.OutOfMemoryError: Java heap space .   When we inves
Read more

Spinnaker Authentication with Keycloak

-
Image
Spinnaker Authentication with Keycloak In my previous  story  I explained how I deployed Spinnaker onto Kubernetes and enabled public access via ingress. In this short story I’m going to explain how I secure Spinnaker by enabling authentication with Keycloak. For this demo I used Kubernetes running on my Mac and also I’m not going to explain OAuth 2.0, OpenID Connect, all the features of Keycloak. Keycloak Keycloak is an Open Source software for Identity and Access Management. It is developed on the top of  Wildfly   server. It provides support for standard protocols like OpenID Connect, OAuth 2.0, and SAML. Authentication Spinnaker authentication involves three main components. They are, Deck  : Sinnaker UI. That is what you see when you access spinnaker. Gate : API gateway for Spinnaker. All the requests go through the Gate and check whether the request is coming from authenticated user. Identity Provider:  In our case this is Keycloak. Spinnaker Authen
Read more

Four Event Driven Patterns

-
Image
In this story I would like to explain my understanding of the Event Driven Architecture(EDA) and its patterns. I was assigned to introduce the CQRS pattern in my previous company. That is how I got to know about EDA. EDA comes with a lot of concepts like  Events, Commands, Event sourcing, Aggreate, Evet Reaplay  etc.. . I’ll briefly explain what EDA is and what common patterns are. Event Driven Architecture(EDA) EDA is producing, captureing , processing and persisting of Events . The concept of the EDA is not new. When we talk about EDA, we can identify two types of objects, those are  Event  and  Command.  We cannot do a proper implementation without knowing these two. Command Command is a request to a system to do something and sometimes expects a response for that. Command is very specific and it has a known destination. That means we know what is going to process and who is going to execute our command. Let’s take an Ordering System as an example. When we want to
Read more