Saturday, January 18, 2020

Automatically Rebuild Image on Docker Hub

This post focuses on me being lazy. In the previous post, I talked about building a custom image and posting it to the Docker Hub. I have also talked about creating a Git repo and storing everything in it thus far. What if we could make a commit rebuild our image for us? As luck would have it, you can do this!

This post is going to focus on making that very simple change to your Docker Hub repository so that every commit causes the image to be rebuilt to the latest. How fun!

Connecting Docker Hub to Your Git Account

The major thing to accomplish here is configuring Docker Hub to monitor Git. In order to do that, you'll need to first sign into your Docker Hub account. This should bring you to the main page where you see the list of repos you maintain:


From there, click on the repo that you plan to configure. In my case, it's the testnginximage repo. On the resulting screen, click on the Builds link to reveal the below page:


Click on the Link to GitHub button, to open your preferences to configure linked accounts.


Click the Connect link on this screen, to link to your GitHub account. If you are already signed into GitHub, Docker Hub will automatically connect to whatever account you are signed in with. If you are not already signed into GitHub, you'll see the below login to GitHub screen:


Login to the GitHub account you used to store your Dockerfile we created in the previous post. Once connected, you'll return to your Docker Hub profile with your GitHub account connected and the account name used listed:


At this point, you now have your Docker Hub and GitHub accounts connected. The next step will be to enable automatic builds.

Enabling Automatic Builds in Docker Hub

With Docker Hub and GitHub connected, the next step is to tell Docker Hub which repo to use and where the Dockerfile is located. In order to do that, go back to your repo and once again, click on the Build link. Within the Build screen, again, click on the Link to GitHub button. This time, the button should say "Connected" on it as shown below:


On the resulting page, configure the username and repo you would like to use as your source. Since I have been building everything in my mysamplerepo repo, I'm choosing this from the drop down:


In my prior examples, I created the Dockerfile in the nginxdocker directory within my mysamplerepo. Assuming you have done the same, scroll down the page and set the Build Context to be the nginxdocker in the Build Rules. This Build Context would be the path from the root of your repo that contains the Dockerfile. If you've placed your Dockerfile in a different path within your repo, make sure you have Build Context configured for that particular path.


Once you have this all configured, click on the Save and Build button at the bottom of the page. This should take you back to the Build page where you can monitor the status of the build.


Monitor the progress to make sure everything builds correctly. Once done, you should see a success status for the build.


Use a Commit to Generate a Build

Now that we have everything connected and working, let's see if we can do a commit to our repo and confirm that the commit makes a build trigger. Let's just make a simple change and no longer expose port 443 from for the image:


FROM ubuntu

MAINTAINER Scott Algatt

RUN apt-get update \
    && apt-get install -y nginx libnginx-mod-http-lua libnginx-mod-http-subs-filter software-properties-common\
    && add-apt-repository -y universe \
    && add-apt-repository -y ppa:certbot/certbot \
    && apt-get update \
    && apt-get -y install certbot python-certbot-nginx \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
COPY ./conf/nginx.conf /etc/nginx/nginx.conf
COPY ./conf/site.conf /etc/nginx/sites-available/default

EXPOSE 80
CMD ["nginx"]

With that change, let's do a commit and push:


$ git commit -a
[master 0e01193] Removing port 443
 Committer: Scott <scott@iMacs-iMac.local>

 2 files changed, 2 deletions(-)

$ git push origin master
Counting objects: 6, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 499 bytes | 499.00 KiB/s, done.
Total 6 (delta 3), reused 0 (delta 0)
remote: Resolving deltas: 100% (3/3), completed with 3 local objects.
To github.com:algattsm/mysamplerepo.git
   1d4d448..0e01193  master -> master

After performing the commit, refresh your Build page in Docker Hub and you should see a build trigger:


This means that you'll be able to simply use your GitHub to generate a new image anytime you like! This also means that with every commit, you'll be exposing the latest version of your image on Docker Hub.

Referenced File

In case you want to make sure you have the correct file, here would be the only file I referenced in this post:

Sunday, January 5, 2020

Posting a Custom Image to Docker Hub

Welcome to 2020! I hope the new year finds everyone in good spirits and ready to continue listening to me babble about my struggles with technology.

So far, the focus has been on using default Docker images for our builds. This is great if you plan to deploy stock instances and only need to serve custom content with some minor configuration tweaks. Note that we were able to make configuration changes using a configMap yaml. What if you needed Nginx modules that weren't already installed in the base image? Sure, you could come up with some funky CMD statement in your yaml file that tells Kubernetes to install the modules. Of course, that'll take some time for the pod to be available while it boots up and runs through the install steps. This will also defeat the purpose of what I'm attempting to show you too :)

The focus of this article is simple. We're going to setup a Docker Hub account and build a custom Nginx image to post there. From there, there are some future articles to help us use this new found knowledge to do some cool stuff.

Let's stop the babble and start the fun!

Creating a Docker Hub Account

This is pretty straight forward so we'll cover it briefly.

  1. Go to https://hub.docker.com/
  2. Click the Sign up for Docker Hub button
  3. Enter your information
  4. Sign up
  5. Wait for the verification Email from Docker
  6. Verify your email via the verification email
  7. Sign in
Done....

Create a Docker Hub Repository

Now that you have a Docker Hub account, you'll want to create a repo to be able to store your custom docker image. Assuming that you are still signed in from the steps above, you should see a Create a Repository button:


If you don't see the Create a Repository button, worry not, you can get there by clicking on the Repositories link on the top menu and then the Create Repository button:


On the resulting Create Repository screen, let's add in some details such as below:


You may call the repo whatever you want and feel free to give it a description. For now, we're going to make this a public repo. Once you have this information filled out, scroll to the bottom and click Create. You should now see something similar to the below:


Make note of the docker push command in the black background box on the right. In my case, it is

docker push algattblog/testnginximage:tagname

We'll need this later when we build our custom image.

Configuring Our Custom Nginx Docker Image

In order to keep everything in one place and keep things backed up, we'll be building this out within our previously defined Git Repo. It's a private repo so reasonably protected and Github is a really nice place to maintain our backup. So the first step will be to make sure we're in the root of our repo and we'll make a new directory to store this image.


$ mkdir nginxdocker

From there, we'll change into the directory so that we can start with our Dockerfile


$ cd nginxdocker/

Now let's create a new Dockerfile that looks like the following:

FROM ubuntu
  
MAINTAINER Scott Algatt

RUN apt-get update \
    && apt-get install -y nginx libnginx-mod-http-lua libnginx-mod-http-subs-filter software-properties-common\
    && add-apt-repository -y universe \
    && add-apt-repository -y ppa:certbot/certbot \
    && apt-get update \
    && apt-get -y install certbot python-certbot-nginx \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
COPY ./conf/nginx.conf /etc/nginx/nginx.conf
COPY ./conf/site.conf /etc/nginx/sites-available/default

EXPOSE 80
EXPOSE 443

CMD ["nginx"]

Let's see what this does...First, we're going to build this new Docker using ubuntu as our base image. From there, we're going to install nginx, libnginx-mod-http-lua, libnginx-mod-http-subs-filter, and software-properties-common. We're installing software-properties-common so that we can add the certbot repo and then add certbot. We're going to also copy over some custom Nginx configuration files so we won't need to leverage our configMap anymore. We make sure ports 80 and 443 are exposed to the running container. Finally, the container should run the "nginx" command to start the nginx server.

Next, we'll want to create those files referenced by the COPY commands. We start by creating the conf directory with and then change into the directory:


$ mkdir conf
$ cd conf

Create the nginx.conf file with the following (Basically, we're defining a custom log format):

user www-data;
worker_processes auto;
pid /run/nginx.pid;

events {
        worker_connections 768;
}

http {

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        include /etc/nginx/mime.types;
        default_type text/html;
        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;
        log_format  graylog2_format  '$remote_addr $request_method "$request_uri" $status $bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" "$http_if_none_match"';
        gzip on;
        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}


daemon off;

The MOST important item at the bottom of this file would be the daemon off; statement. Without this, our container will start run nginx and then COMPLETE and stop. We want nginx to run in the foreground and not background as a daemon so this is why this is here. Now create the referenced site.conf file.

server {
    listen       80;
    server_name  localhost;
    access_log /var/log/nginx/access.log graylog2_format;
    error_log /var/log/nginx/error.log graylog2_format;
    location / {
        root   /usr/share/nginx/www/html;
        index  index.html index.htm;
    }
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/www/html;
    }
  location ~ .php$ {
      root /usr/share/nginx/www/html;
      try_files $uri =404;
      fastcgi_split_path_info ^(.+.php)(/.+)$;
      fastcgi_pass phpfpm:9000;
      fastcgi_index index.php;
      include fastcgi_params;
      fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
      fastcgi_param PATH_INFO $fastcgi_path_info;
  }

}

This looks good so let's first save our changes and commit them to our repo.

$ cd ../..
$ git add .
$ git commit -a
[master ade43da] Adding in our stuff
 Committer: Scott <scott@iMacs-iMac.local>
 3 files changed, 23 insertions(+), 37 deletions(-)
 rewrite nginxdocker/conf/site.conf (99%)
 delete mode 100644 nginxdocker/conf/test
$ git push origin master
Enter passphrase for key '/Users/scott/.ssh/id_rsa': 
Counting objects: 6, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 754 bytes | 754.00 KiB/s, done.
Total 6 (delta 2), reused 0 (delta 0)
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
To github.com:algattsm/mysamplerepo.git
   97b7a1c..ade43da  master -> master

$ 

Now that we've got that squared away, it's onto the next step!

Building our Docker Image and Publishing it

This is the easy part as we just watch it run. Make sure we're in the directory that contains our Dockerfile and then we'll run the build command:


$ cd nginxdocker/
imacs-imac:nginxdocker scott$ docker build -t algattblog/testnginximage:latest .

Sending build context to Docker daemon  6.144kB
Step 1/8 : FROM ubuntu
latest: Pulling from library/ubuntu
2746a4a261c9: Pull complete 
4c1d20cdee96: Pull complete 
0d3160e1d0de: Pull complete 
c8e37668deea: Pull complete 
Digest: sha256:250cc6f3f3ffc5cdaa9d8f4946ac79821aafb4d3afc93928f0de9336eba21aa4
Status: Downloaded newer image for ubuntu:latest
 ---> 549b9b86cb8d
Step 2/8 : MAINTAINER Scott Algatt
 ---> Running in ff6d8459f56b
Removing intermediate container ff6d8459f56b
 ---> 666acba43494
Step 3/8 : RUN apt-get update     && apt-get install -y nginx libnginx-mod-http-lua libnginx-mod-http-subs-filter software-properties-common    && add-apt-repository -y universe     && add-apt-repository -y ppa:certbot/certbot     && apt-get update     && apt-get -y install certbot python-certbot-nginx     && apt-get clean     && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
 ---> Running in acfefd676a08
Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:2 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]
...
Setting up python-certbot-nginx (0.31.0-1+ubuntu18.04.1+certbot+1) ...
Removing intermediate container acfefd676a08
 ---> 09301061d312
Step 4/8 : COPY ./conf/nginx.conf /etc/nginx/nginx.conf
 ---> c82b8d22e6a0
Step 5/8 : COPY ./conf/site.conf /etc/nginx/sites-available/default
 ---> 841e6ecfc3d9
Step 6/8 : EXPOSE 80
 ---> Running in f2f36c350457
Removing intermediate container f2f36c350457
 ---> 79af7e01f9c0
Step 7/8 : EXPOSE 443
 ---> Running in 9d6a8dcdba31
Removing intermediate container 9d6a8dcdba31
 ---> a534c821c51b
Step 8/8 : CMD ["nginx"]
 ---> Running in 82ceccd20644
Removing intermediate container 82ceccd20644
 ---> 6728616336a3
Successfully built 6728616336a3
Successfully tagged algattblog/testnginximage:latest

Next, we need to publish it on Docker Hub (remember that push command from earlier?):


$ docker push algattblog/testnginximage:latest
The push refers to repository [docker.io/algattblog/testnginximage]
010c4615edf3: Pushed 
98c06aef3fd3: Pushed 
229f4ffc7b88: Pushed 
918efb8f161b: Mounted from library/ubuntu 
27dd43ea46a8: Mounted from library/ubuntu 
9f3bfcc4a1a8: Mounted from library/ubuntu 
2dc9f76fb25b: Mounted from library/ubuntu 
latest: digest: sha256:4545730a7dd5b5818f0ce9a78666f40ea9a864198665022dc29000a34cc4b402 size: 1778

Testing the New Image in our Cluster

At this point, we've got our newly created image uploaded to Docker Hub. The next step is to test it out and see if it works in our cluster. In order to do that, we'll want to change the image in our webserver.yaml file to reference the newly created image:


        image: algattblog/testnginximage:latest

Now let's apply the updated deployment:


# kubectl apply -f webserver.yaml 
configmap/webserver-config unchanged
service/webserver unchanged
deployment.apps/webserver configured

Wait for it to build the new webserver pod by keeping an eye on kubectl:


# kubectl get pod
NAME                         READY   STATUS    RESTARTS   AGE
phpfpm-7b8d87955c-rps2w      2/2     Running   0          3d1h
webserver-6b577db595-wgwwb   2/2     Running   0          25s

Looks like it's up and running so now let's check and see if we can test to make sure everything looks good. We'll start by connecting to the webserver, installing curl, and making sure everything works again.


# kubectl exec -it webserver-6b577db595-wgwwb /bin/bash -c webserver
groups: cannot find name for group ID 65533
root@webserver-6b577db595-wgwwb:/# apt update     
Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:2 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]           
Setting up libcurl4:amd64 (7.58.0-2ubuntu3.8) ...
Setting up curl (7.58.0-2ubuntu3.8) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
root@webserver-6b577db595-wgwwb:/# curl localhost
<html>
<body>
hello world! Everything must be cleaned up at this point
</body>
</html>
root@webserver-6b577db595-wgwwb:/# curl localhost/index.php
hello world from php

We can ignore the groups error for now. Some idiot left out instructions regarding fixing that but not a biggie. Looks we're serving content properly. Let's check our nginx configuration to confirm it's also running the correct config:


root@webserver-6b577db595-wgwwb:/# cat /etc/nginx/nginx.conf 
user www-data;
...
log_format  graylog2_format  '$remote_addr $request_method "$request_uri" $status $bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" "$http_if_none_match"';
gzip on;
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}

daemon off;
root@webserver-6b577db595-wgwwb:/# 

Confirmed! This looks great! You've now got a custom Docker image that you can use and expand upon. I know I left you hanging here but if I continued to build my attention span couldn't tolerate typing anymore. More to come....

Monday, December 30, 2019

Deploying Nginx + PHP + git-sync on Kubernetes

In my previous post, I explained how to setup a simple nginx instance that could be used to sync to a private Git repo. The only drawback is that this setup will only serve static pages. What if you wanted to be able to run a server with dynamic code like PHP? I'm glad you asked! In this post, we'll update our config to include a php-fpm instance to allow us to serve PHP pages.

I have planned these articles out so that they build on each other. With that in mind, I'm assuming you have followed my articles to date and therefore we'll be simply extended the current deployment.

If you're impatient like me, just scroll to the bottom and download the full files.

Setting Up The PHP-FPM Instance

First we need to get our PHP-FPM yaml setup. By default, php-fpm runs on port 9000. This means we need a service definition to expose this to the cluster. This will also need access to the git repo we created so we'll add in the git container spec. Instead of running the nginx image, we'll run the php-fpm image. In order to make life easy on ourselves, I'm going to use the webserver.yaml from my previous post as a template. I'm going to make the following changes to it:

  1. Replace any reference of "webserver" with "phpfpm".
  2. Change the following in the service definition
    1. change the port name from http to phpfpm
    2. change the port number from 80 to 9000
  3. Remove the ConfigMap
    1. Remove the definition of it from the top of the file
    2. Remove the references to it in the spec volumes and the container volumeMounts
  4. Change the image of the second container from nginx:latest to php:fpm
  5. Change the containerPort from 80 to 9000
If we've done this all correctly, we should have a yaml that looks similar to the below:

apiVersion: v1

kind: Service
metadata:
  name: phpfpm
  labels:
    tier: backend
spec:
  selector:
    app: phpfpm
    tier: backend
  ports:  
  - name: phpfpm
    port: 9000
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: phpfpm
  labels:
    tier: backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: phpfpm
      tier: backend
  template:
    metadata:
      labels:
        app: phpfpm
        tier: backend
    spec:
      securityContext:
        fsGroup: 65533 # to make SSH key readable
      volumes:
      - name: dir
        emptyDir: {}
      - name: git-secret
        secret:
          secretName: github-creds
          defaultMode: 288
      containers:
      - env:
        - name: GIT_SYNC_REPO
          value: git@github.com:<some user>/mysamplerepo.git
        - name: GIT_SYNC_BRANCH
          value: master
        - name: GIT_SYNC_SSH
          value: "true"
        - name: GIT_SYNC_PERMISSIONS
          value: "0777"
        - name: GIT_SYNC_DEST
          value: www
        - name: GIT_SYNC_ROOT
          value: /git
        name: git-sync
        image: k8s.gcr.io/git-sync:v3.1.1
        securityContext:
          runAsUser: 65533 # git-sync user
        volumeMounts:
        - name: git-secret
          mountPath: /etc/git-secret
        - name: dir
          mountPath: /git
      - name: phpfpm
        image: php:fpm
        ports:
        - containerPort: 9000
        volumeMounts:
        - name: dir
          mountPath: /usr/share/nginx

We can now save this yaml and apply it to our cluster:

# kubectl apply -f phpfpm.yaml 
service/phpfpm unchanged

deployment.apps/phpfpm configured

Assuming all went well, we should now have our webserver and phpfpm containers up and running:

# kubectl get pod
NAME                         READY   STATUS    RESTARTS   AGE
phpfpm-b46969c5f-zzh6d       2/2     Running   0          103s

webserver-8fb84dc86-7xw4w    2/2     Running   0          10s

That's just lovely but what next?

Configuring Nginx for PHP

At this point, we basically have two unassociated containers that are living independently in the same cluster. The only common bond is that they have the same set of files synched from the Git Repo. Next, we need to tell nginx to handle PHP requests and where to send them. This will require us to update our Nginx configMap. We do this by adding a location statement to handle php files like so:


      location ~ .php$ {
          try_files $uri =404;
          fastcgi_split_path_info ^(.+.php)(/.+)$;
          fastcgi_pass phpfpm:9000;
          fastcgi_index index.php;
          include fastcgi_params;
          fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
          fastcgi_param PATH_INFO $fastcgi_path_info;
      }

There's lots going on here in this file but some important items to note. Nginx acts like a reverse proxy when handling PHP files. It simply takes the request and sends to php-fpm. The php-fpm service finds the request file locally, executes PHP on it, and sends the resulting processed output from PHP back to Nginx. Here is the full updated configMap:


apiVersion: v1
kind: ConfigMap
metadata:
  name: webserver-config
  labels:
    tier: backend
data:
  config :
    server {
        listen       80;
        server_name  localhost;

        location / {
            root   /usr/share/nginx/www/html;
            index  index.html index.htm;
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/share/nginx/www/html;
        }

      location ~ .php$ {
          root /usr/share/nginx/www/html;
          try_files $uri =404;
          fastcgi_split_path_info ^(.+.php)(/.+)$;
          fastcgi_pass phpfpm:9000;
          fastcgi_index index.php;
          include fastcgi_params;
          fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
          fastcgi_param PATH_INFO $fastcgi_path_info;
      }

    }

Let's apply this to our cluster:


# kubectl apply -f configmap.yaml 
configmap/webserver-config configured

With the new configuration running, we'll need Nginx to reload it. There's a number of different ways we could do this but I'm going to use a hack that will allow us to test the config and then restart. First step, I want to make sure the new config will work for us:


# kubectl exec -it webserver-8fb84dc86-7xw4w -c webserver -- /usr/sbin/nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

It looks like the configuration is acceptable so let's reload Nginx.


# kubectl exec -it webserver-8fb84dc86-7xw4w -c webserver -- /usr/sbin/nginx -s reload
2019/12/28 14:01:32 [notice] 2804#2804: signal process started

We should now be ready to commit a PHP file to our repo and test.

Testing Our Configuration

Let's create a simple PHP file in the html directory of our. 


We'll jump onto the web server, install curl and test:


# kubectl exec -it webserver-8fb84dc86-7xw4w -c webserver -- /bin/bash
root@webserver-8fb84dc86-7xw4w:/# apt update
Hit:1 http://deb.debian.org/debian buster InRelease
Hit:2 http://deb.debian.org/debian buster-updates InRelease
Hit:3 http://security-cdn.debian.org/debian-security buster/updates InRelease
Reading package lists... Done
Building dependency tree       
Reading state information... Done
All packages are up to date.
root@webserver-8fb84dc86-7xw4w:/# apt install curl
Reading package lists... Done
Building dependency tree       
Reading state information... Done
curl is already the newest version (7.64.0-4).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
root@webserver-8fb84dc86-7xw4w:/# curl localhost/index.php
hello world from php

A great question to ask is how does php-fpm know which file and where that file exists? Like I said, great question.

This is handled by the fastcgi_param SCRIPT_FILENAME entry. This means that Nginx is going to tell php-fpm that it should try to load the $document_root$fastcgi_script_name file for the request. If you look at our configMap, we define document root as /usr/share/nginx/www/html. Assuming a request comes for index.php into Nginx, Nginx will tell php-fpm to also load /usr/share/www/html/index.php. In an environment where Nginx + PHP live on the same host, this doesn't appear to be a problem because that file will exist for sure. In our configuration, we running two separate hosts aka containers. So we need to make sure the file exists on both servers in the same location. That's the easy part! It does! Reason being, we're using gitsynch on both containers and mounting that synched directory to the same location!

Full Working Configs

In case you want to just cheat and load the configurations, feel free to download them and play around:

Thursday, December 26, 2019

Building a Kubernetes Container That Synchs with Private Git Repo

My previous post explained how to create a private git repo. On its own, that post is roughly useless unless you planned to maintained some private copy of your project so nobody can see it. In this post, we're going to put that private repo to use in a Kubernetes environment. A basic assumption is that you already have a Kubernetes environment setup.

Adding Another SSH Key to the Repo

The first step would be to add another SSH Key to our repo. The purpose of this key is to be used to configure access from the container to the repo. We'll load the SSH key into Kubernetes as a secret. We can't set a password on this key or we might get prompted for the password during container build and that's not useful. Also, since the key will not have a password, we won't give it Read / Write access to our repo.

Generate the SSH Key

As before, we're going to run the ssh-keygen command but we'll specify the file where to save the key and just simply hit enter at the password prompt so that it's not password protected.


imacs-imac:~ scott$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/scott/.ssh/id_rsa): /Users/scott/.ssh/GH_RO_key_rsa
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /Users/scott/.ssh/GH_RO_key_rsa.
Your public key has been saved in /Users/scott/.ssh/GH_RO_key_rsa.pub.
The key fingerprint is:
SHA256:0v0koHVNHdJbt4j2PaNorHa25dXgNl0sQjJB8R3ClPA scott@imacs-imac.lan
The key's randomart image is:
+---[RSA 2048]----+
|         .===+o. |
|           *o+o.o|
|        o + E ooo|
|       + + * ..o |
|      o S + + + o|
|       .   + + Bo|
|          . o.=.=|
|         . *oo.. |
|        ..=...   |
+----[SHA256]-----+
imacs-imac:~ scott$ 

Upload the Key to our Git Repo

With our new SSH Key created, we'll want to once again take the contents of the .pub file aka GH_RO_key_rsa.pub if you're following along and paste that into our repo's Deploy Keys like below:


Be sure that Allow write access is NOT selected and paste in the contents of the pub file to the Key box. Next, click Add Key. You should now have two keys listed:


Configuring Kubernetes

Now that we have our new Read Only key added to the repo, it's time to setup Kubernetes. This is going to be a simple configuration so that we can display static HTML pages on our Kubernetes cluster.

Add SSH Key to Kubernetes

In order to have Kubernetes be able to use the SSH key, we need to add it as a secret that we'll reference in our pod deployment. The first step is to create a known hosts file to be used along with the key so we don't have to worry about acknowledging any new key messages.


~# ssh-keyscan github.com > /tmp/known_hosts
# github.com:22 SSH-2.0-babeld-778045a0
# github.com:22 SSH-2.0-babeld-778045a0
# github.com:22 SSH-2.0-babeld-778045a0
~# 

This copies the ssh key from github into the /tmp/known_hosts file. Next, we need to get the contents of our private key file. When we pasted the key into GitHub, we were working with the public key file..aka the .pub file...Since Kubernetes will need to authenticate using this key, it'll need the private key file...aka the GH_RO_key_rsa file. We'll use the kubectl command to add the key into Kubernetes:


~# kubectl create secret generic github-creds --from-file=ssh=.ssh/GH_RO_key_rsa --from-file=known_hosts=/tmp/known_hosts
secret/github-creds created
~# 

Creating the Web Server Deployment

Now we're going to create a YAML file to configure and setup everything. The start of that YAML file will be to configure Kubernetes to open a port that directs traffic to port 80 of our resulting pod. From there, we'll need to setup a pod that runs two separate containers. One container will be our git-synch application and the other will be nginx. We could get into some "complex" discussions and added costs of running a PVC or some other Kubernetes shared storage but we're only dealing with a small web site that is synched with github so we're gonna simply leverage local storage on each node by defining two volumes:


      volumes:
      - name: dir
        emptyDir: {}
      - name: git-secret
        secret:
          secretName: github-creds
          defaultMode: 288

This creates two volumes dir and git-secret. The dir is simply an empty directory volume that we'll be filling with our files that we synch from Github. The git-secret is the SSH Key we added above. This needs to be made available to our git-synch container. 

In the nginx container, we're going to mount the dir volume as /usr/share/nginx. The default nginx image looks for web content, aka document root, in /usr/share/nginx/html. Therefore, we're going mount the repo as /usr/share/nginx. We mount the dir volume to /git as this is where we're going to write our synched data.

You can see all of these configurations in the git-synch container configuration such as the target location for our synched files as well as the secret to use.


      containers:
      - env:
        - name: GIT_SYNC_REPO
          value: git@github.com:<some user>/mysamplerepo.git
        - name: GIT_SYNC_BRANCH
          value: master
        - name: GIT_SYNC_SSH
          value: "true"
        - name: GIT_SYNC_PERMISSIONS
          value: "0777"
        - name: GIT_SYNC_DEST
          value: www
        - name: GIT_SYNC_ROOT
          value: /git
        name: git-sync
        image: k8s.gcr.io/git-sync:v3.1.1
        securityContext:
          runAsUser: 65533 # git-sync user
        volumeMounts:
        - name: git-secret
          mountPath: /etc/git-secret
        - name: dir
          mountPath: /git

You'll want to make sure you change the GIT_SYNC_REPO to match the value of your clone/download link in Github. The GIT_SYNC_DEST should match the name of your repo.

Here is the full config for reference:


apiVersion: v1
kind: Service
metadata:
  name: webserver
  labels:
    tier: backend
spec:
  selector:
    app: webserver
    tier: backend
  ports:  
  - name: http
    port: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webserver
  labels:
    tier: backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: webserver
      tier: backend
  template:
    metadata:
      labels:
        app: webserver
        tier: backend
    spec:
      securityContext:
        fsGroup: 65533 # to make SSH key readable
      volumes:
      - name: dir
        emptyDir: {}
      - name: git-secret
        secret:
          secretName: github-creds
          defaultMode: 288
      containers:
      - env:
        - name: GIT_SYNC_REPO
          value: git@github.com:<some user>/mysamplerepo.git
        - name: GIT_SYNC_BRANCH
          value: master
        - name: GIT_SYNC_SSH
          value: "true"
        - name: GIT_SYNC_PERMISSIONS
          value: "0777"
        - name: GIT_SYNC_DEST
          value: www
        - name: GIT_SYNC_ROOT
          value: /git
        name: git-sync
        image: k8s.gcr.io/git-sync:v3.1.1
        securityContext:
          runAsUser: 65533 # git-sync user
        volumeMounts:
        - name: git-secret
          mountPath: /etc/git-secret
        - name: dir
          mountPath: /git
      - name: webserver
        image: nginx:latest
        ports:
        - containerPort: 80
        volumeMounts:
        - name: dir
          mountPath: /usr/share/nginx

With out configuration file all ready to go, we'll use kubectl to apply the file:


~# kubectl apply -f webserver.yaml 
service/webserver created
deployment.apps/webserver created
~# 

After some time, we should be able to check the status and see the pod is online and the service is setup:


~# kubectl get pod
NAME                         READY   STATUS    RESTARTS   AGE
webserver-686854f667-cwq5f   2/2     Running   5          3m46s
~# kubectl get svc
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.152.183.1     <none>        443/TCP   149m
webserver    ClusterIP   10.152.183.195   <none>        80/TCP    5m28s
~# 

Testing the Deployment

With everything deployed, we should have a web server up and running that is serving our git repo from the previous post. Without getting into deploying an ingress server and such, let's take a short cut to test out our deployment. We can do this by connecting to the web server and doing a curl. First, we connect to the web server container:


# kubectl exec -it webserver-686854f667-cwq5f -c webserver /bin/bash

The above command will connect you to a shell in the container. By default, the nginx image does not have curl installed so we'll need to install this to test further. Install curl using the below commands:


root@webserver-686854f667-cwq5f:/# apt update;apt -y install curl

With curl installed, let's connect to the local web server:


root@webserver-686854f667-cwq5f:/# curl localhost
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx/1.17.6</center>
</body>
</html>

That does not seem right...I broke something...didn't I? Oh wait, I know let's try...


root@webserver-686854f667-cwq5f:/# curl localhost/html/
<html>
<body>
hello world!
</body>
</html>

That works better. Looks like we need to fix something here but first let's see if making a change to the repo works. Let's cheat and use the github file editor and make a change to the index.html file like the below:


If we run our curl again, survey says....


root@webserver-686854f667-cwq5f:/# curl localhost/html/
<html>
<body>
hello world! Test #2
</body>
</html>

Boom! Just like that it's working. Kinda...

Fixing Our Deployment

In case the problem isn't quite obvious, we are attempting to mount the git repo in a location that nginx isn't quite looking for. It's a bad idea to mount the entire git repo as the document root since it could allow people to look at your .git directory and possibly other files that you didn't consider. In order to fix our deployment and secure just a little further we're going to first adjust the nginx configuration with a Kubernetes configmap:


apiVersion: v1
kind: ConfigMap
metadata:
  name: webserver-config
  labels:
    tier: backend
data:
  config :
    server {
        listen       80;
        server_name  localhost;
    
        location / {
            root   /usr/share/nginx/www/html;
            index  index.html index.htm;
        }
    
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/share/nginx/www/html;
        }
   
    }

This configmap supplies nginx with a new configuration for the default site that tells nginx that the document root is now located in /usr/share/nginx/www/html. We also made some changes to the original webserver.yaml to add this new configuration as well as changing the mount point for git and nginx. The full configuration is here.


apiVersion: v1
kind: ConfigMap
metadata:
  name: webserver-config
  labels:
    tier: backend
data:
  config :
    server {
        listen       80;
        server_name  localhost;
        location / {
            root   /usr/share/nginx/www/html;
            index  index.html index.htm;
        }
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/share/nginx/www/html;
        }
    }
---
apiVersion: v1
kind: Service
metadata:
  name: webserver
  labels:
    tier: backend
spec:
  selector:
    app: webserver
    tier: backend
  ports:  
  - name: http
    port: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webserver
  labels:
    tier: backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: webserver
      tier: backend
  template:
    metadata:
      labels:
        app: webserver
        tier: backend
    spec:
      securityContext:
        fsGroup: 65533 # to make SSH key readable
      volumes:
      - name: dir
        emptyDir: {}
      - name: git-secret
        secret:
          secretName: github-creds
          defaultMode: 288
      - name: config
        configMap:
          name: webserver-config
          items:
          - key: config
            path: default.conf
      containers:
      - env:
        - name: GIT_SYNC_REPO
          value: git@github.com:<some user>/mysamplerepo.git
        - name: GIT_SYNC_BRANCH
          value: master
        - name: GIT_SYNC_SSH
          value: "true"
        - name: GIT_SYNC_PERMISSIONS
          value: "0777"
        - name: GIT_SYNC_DEST
          value: www
        - name: GIT_SYNC_ROOT
          value: /git
        name: git-sync
        image: k8s.gcr.io/git-sync:v3.1.1
        securityContext:
          runAsUser: 65533 # git-sync user
        volumeMounts:
        - name: git-secret
          mountPath: /etc/git-secret
        - name: dir
          mountPath: /git
      - name: webserver
        image: nginx:latest
        ports:
        - containerPort: 80
        volumeMounts:
        - name: dir
          mountPath: /usr/share/nginx
        - name: config
          mountPath: /etc/nginx/conf.d

Let's apply this updated configuration using kubectl:


root@do-nyc04:/tmp# kubectl apply -f webserver.yaml 
configmap/webserver created
service/webserver unchanged
deployment.apps/webserver configured

Let's now reconnect and test our configuration:


root@do-nyc04:/tmp# kubectl get pod -o wide
NAME                         READY   STATUS    RESTARTS   AGE    IP             NODE              NOMINATED NODE   READINESS GATES
webserver-8fb84dc86-5chm5    2/2     Running   0          17s    10.244.1.53    pool-sfo01-ssy1   <none>           <none>
root@do-nyc04:/tmp# kubectl exec -it webserver-8fb84dc86-5chm5 -c webserver /bin/bash
root@webserver-8fb84dc86-5chm5:/# apt update;apt -y install curl
Get:1 http://deb.debian.org/debian buster InRelease [122 kB]
Get:2 http://deb.debian.org/debian buster-updates InRelease [49.3 kB]             
Get:3 http://security-cdn.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:4 http://deb.debian.org/debian buster/main amd64 Packages [7908 kB]
Get:5 http://deb.debian.org/debian buster-updates/main amd64 Packages [5792 B]
Get:6 http://security-cdn.debian.org/debian-security buster/updates/main amd64 Packages [167 kB]
Fetched 8317 kB in 2s (3534 kB/s)                         
Reading package lists... Done
Building dependency tree       
Reading state information... Done
All packages are up to date.
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
...
128 added, 0 removed; done.
Setting up libgssapi-krb5-2:amd64 (1.17-3) ...
Setting up libcurl4:amd64 (7.64.0-4) ...
Setting up curl (7.64.0-4) ...
Processing triggers for libc-bin (2.28-10) ...
Processing triggers for ca-certificates (20190110) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
root@webserver-8fb84dc86-5chm5:/# curl localhost
<html>
<body>
hello world! Test #2
</body>
</html>

Great news! It looks like it's fixed. Just to make sure things are working still, let's make another change and see if it publishes.


root@webserver-8fb84dc86-5chm5:/# curl localhost
<html>
<body>
hello world! Everything must be cleaned up at this point
</body>
</html>

W00t! Looks like everything is working and as we expect. Although, this configuration is mostly useless unless you are actually within the Kubernetes cluster. For the next article, I'll provide some options and a hack for exposing this web server to the world.

Automatically Rebuild Image on Docker Hub

This post focuses on me being lazy. In the previous post , I talked about building a custom image and posting it to the Docker Hub. I have a...