Set Up a CI/CD Pipeline with Kubernetes
Part 1: Overview
Start GCP K8s Cluster
export CLOUD_SDK_REPO="cloud-sdk-$(lsb_release -c -s)"
echo "deb http://packages.cloud.google.com/apt $CLOUD_SDK_REPO main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
sudo apt-get update && sudo apt-get install google-cloud-sdk -y
gcloud init
gcloud container clusters create admatic-cluster --num-nodes=3
sudo snap install kubectl --classic
kubectl 1.11.3 from 'canonical' installed
Add cluster-admin permissions to service accounts
cat << EOF > admin.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: serviceaccounts-cluster-admin
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:serviceaccounts
EOF
kubectl create -f admin.yaml
clusterrolebinding.rbac.authorization.k8s.io/serviceaccounts-cluster-admin created
Create a Local Image Registry
While Docker Hub is great for public images, setting up a private image repository on the site involves some security key overhead that we don’t want to deal with. Instead, we’ll set up our own local image registry. We’ll then build, push, and run a sample Hello-Admatic app from the local registry.
Set up the cluster registry by applying a .yml manifest file.
https://github.com/admatic/kubernetes-ci-cd/blob/master/manifests/registry.yml
git clone https://github.com/admatic/kubernetes-ci-cd.git
cd kubernetes-ci-cd
kubectl apply -f manifests/registry.yml
persistentvolume "registry" created
persistentvolumeclaim "registry-claim" created
service "registry" created
service "registry-ui" created
deployment.extensions "registry" created
Wait for the registry to finish deploying. Note that this may take several minutes.
kubectl rollout status deployments/registry
Waiting for rollout to finish: 0 of 1 updated replicas are available...
deployment "registry" successfully rolled out
View the registry user interface in a web browser. Right now it’s empty, but you’re about to change that.
kubectl get svc | grep registry-ui
registry-ui NodePort 10.63.247.190 <none> 8080:31105/TCP 36s
kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
gke-admatic-cluster-default-pool-0b7b8b74-58q9 Ready <none> 2m v1.9.7-gke.6 10.168.0.2 35.236.125.7 Container-Optimized OS from Google 4.4.111+ docker://17.3.2
gke-admatic-cluster-default-pool-0b7b8b74-b3p6 Ready <none> 2m v1.9.7-gke.6 10.168.0.4 35.236.2.204 Container-Optimized OS from Google 4.4.111+ docker://17.3.2
gke-admatic-cluster-default-pool-0b7b8b74-p7gh Ready <none> 2m v1.9.7-gke.6 10.168.0.3 35.236.29.14 Container-Optimized OS from Google 4.4.111+ docker://17.3.2
Go to http://35.236.125.7:31105/ to view the registry UI
Let’s make a change to an HTML file in the cloned project.
vim applications/hello-admatic/index.html
Now let’s build an image, giving it a special name that points to our local cluster registry.
curl -sSL http://get.docker.com/ | sh
sudo docker build -t 127.0.0.1:30400/hello-admatic:latest -f applications/hello-admatic/Dockerfile applications/hello-admatic
Sending build context to Docker daemon 70.14kB
Step 1/4 : FROM nginx:latest
---> bc26f1ed35cf
Step 2/4 : COPY index.html /usr/share/nginx/html/index.html
---> Using cache
---> 288577935e8f
Step 3/4 : COPY DockerFileEx.jpg /usr/share/nginx/html/DockerFileEx.jpg
---> da0d66d1f6a5
Step 4/4 : EXPOSE 80
---> Running in 0957cb472401
Removing intermediate container 0957cb472401
---> 1d280a0770b5
Successfully built 1d280a0770b5
Successfully tagged 127.0.0.1:30400/hello-admatic:latest
We’ve built the image, but before we can push it to the registry, we need to set up a temporary proxy. By default the Docker client can only push to HTTP (not HTTPS) via localhost. To work around this, we’ll set up a container that listens on 127.0.0.1:30400 and forwards to our cluster.
sudo docker stop socat-registry
sudo docker rm socat-registry
sudo docker run -d -e "REGIP=35.236.125.7" --name socat-registry -p 30400:5000 chadmoon/socat:latest bash -c "socat TCP4-LISTEN:5000,fork,reuseaddr TCP4:35.236.125.7:30400"
Unable to find image 'chadmoon/socat:latest' locally
latest: Pulling from chadmoon/socat
627beaf3eaaf: Pull complete
f2bcbd47243c: Pull complete
f4175031eafb: Pull complete
35070a1aa40d: Pull complete
Digest: sha256:14bfffdd5fbcec8e30b263dc29fc2b48c73a09c5eed9a00dc0d5efb8e83eba94
Status: Downloaded newer image for chadmoon/socat:latest
d5d7427371b0b13a49383270bfa0666e8ba16723a4de6e6d5b1da1f12aaf2a1f
With our proxy container up and running, we can now push our image to the local repository.
sudo docker push 127.0.0.1:30400/hello-admatic:latest
The push refers to repository [127.0.0.1:30400/hello-admatic]
eacf7afb5aa5: Pushed
cd204b10686f: Pushed
e8916cb59586: Pushed
3bbff39fa30b: Pushed
8b15606a9e3e: Pushed
latest: digest: sha256:5778ccd6aa0484920ff5721a910ae329c91cedf17813d366656878debdac109e size: 1364
Refresh the browser window with the registry UI and you’ll see the image has appeared.
The proxy’s work is done, so you can go ahead and stop it.
sudo docker stop socat-registry;
socat-registry
With the image in our cluster registry, the last thing to do is apply the manifest to create and deploy the hello-admatic service based on the image.
kubectl apply -f applications/hello-admatic/k8s/deployment.yaml
service/hello-admatic created
deployment.extensions/hello-admatic created
Launch a web browser and view the service.
kubectl get svc | grep admatic
hello-admatic NodePort 10.63.255.133 <none> 80:30182/TCP 4s
curl 35.236.125.7:30182
<p><h2 style="font-family:sans-serif">Welcome! God is Great!</br>Hello from Admatic!!!</br> You've successfully built and run the Hello-Admatic app.</h2> </p>
<p style="font-family:sans-serif">The Hello-Admatic app is a modified version of the <a href="https://hub.docker.com/_/nginx/">nginx web server image</a>. If you open up the <b>kubernetes-ci-cd/part1/hello-admatic/DockerFile</b>, you will note several things:</p>
<img src="DockerFileEx.jpg">
Part 2: Jenkins
Creating and Building a Pipeline in Jenkins
Install Jenkins, which we’ll use to create our automated CI/CD pipeline. It will take the pod a minute or two to roll out.
kubectl apply -f manifests/jenkins.yml
persistentvolume "jenkins" created
persistentvolumeclaim "jenkins-claim" created
service "jenkins" created
deployment.extensions "jenkins" created
kubectl rollout status deployment/jenkins
Waiting for rollout to finish: 0 of 1 updated replicas are available...
deployment "jenkins" successfully rolled out
kubectl get svc | grep jenkins
jenkins NodePort 10.63.252.52 <none> 80:31975/TCP 1m
Go to http://35.236.125.7:31975 to see the Jenkins Web UI
kubectl get pods | grep jenkins
jenkins-774bf687f9-htzr4 1/1 Running 0 1m
kubectl exec -it jenkins-774bf687f9-htzr4 bash
cat /root/.jenkins/secrets/initialAdminPassword
adf8f3799afa4cd5bfce697fb8559814
Paste the Jenkins admin password in the box and click Continue. Click Install suggested plugins and wait for the process to complete.
- Create an admin user and credentials
- We now want to create a new pipeline for use with our
Hello-Admaticapp. On the left, click New Item. - Enter the item name as
Hello-Admatic Pipeline, select Pipeline, and click OK. - Under the Pipeline section at the bottom, change the
Definitionto bePipeline script from SCM. - Change the
SCMtoGit. - Change the Repository URL to
https://github.com/admatic/kubernetes-ci-cd - Click Save. On the left, click
Build Nowto run the new pipeline. You should see it run through the build, push, and deploy steps in a few seconds.
Now view the Hello-Admatic application.
kubectl get svc | grep admatic
hello-admatic NodePort 10.63.255.133 <none> 80:30182/TCP 7m
curl 35.236.125.7:30182
<p><h2 style="font-family:sans-serif">Welcome! God is Great!</br>Hello from Admatic!!!</br> You've successfully built and run the Hello-Admatic app.</h2> </p>
<p style="font-family:sans-serif">The Hello-Admatic app is a modified version of the <a href="https://hub.docker.com/_/nginx/">nginx web server image</a>. If you open up the <b>kubernetes-ci-cd/part1/hello-admatic/DockerFile</b>, you will note several things:</p>
<img src="DockerFileEx.jpg">
You might notice that you’re not seeing the change you previously made to index.html. That’s because Jenkins wasn’t using your local code. Instead, Jenkins pulled the code from your forked repo on GitHub, used that code to build the image, pushed it, and then deployed it.
Pushing Code Changes Through the Pipeline
Commit the changed index.html to your Git repo (you’ll need to enter your GitHub credentials)
cd ~/kubernetes-ci-cd
git add .
git commit -m "Added message to index.html"
git push
In the Jenkins UI, click Build Now to run the build again.
View the updated Hello-Admatic application. You should see the message you added to index.html.
curl 35.200.162.24:31857
<p><h2 style="font-family:sans-serif">Welcome! God is Great!</br>Hello from Admatic!!!</br> You've successfully built and run the Hello-Admatic app.</h2> </p>
<p style="font-family:sans-serif">The Hello-Admatic app is a modified version of the <a href="https://hub.docker.com/_/nginx/">nginx web server image</a>. If you open up the <b>kubernetes-ci-cd/hello-admatic/DockerFile</b>, you will note several things:</p>
<img src="DockerFileEx.jpg">