#! /bin/env sh

This page

Work in progress notes and views of kubernetes and setting it up on bare metal. it works on everything from your own raspberry-pi cluster, as well as anything else you can compile golang, I will use systmd, but you should be able to be up and running by just running programs in a screen-session.


Intro

Not very long ago I started looking at kubernetes for my job at the university and found a shitload of howtos. Many of them were good, some just gave me parts of the information I needed, and some led me on wrong tracks I got a lot of infomation from when I got back on the right track.

My goal was from the start to have a general cluster recipe, as one of the great joys about docker is that you can run systems everywhere. One of the great things about staticly compiled golang is that you can run the same binary on all machines with the right architecture. Kubernetes is written in go, and so is etcd (storage), and flannel (network). I will not describe using other network-plugins, and I will probably describe it different than how your OS wants it to be for the sake of plattform-independency and simplicity (e.g. "on fedora you do this, on ubuntu you do that").

I assume you know docker in this blog, so I won't say much about it, it will in many ways be an underlying technology and not prominent in this work.



Certificate config-files

We'll start by creating a certificate authority for authentication and encryption so we know that other people can't connect to the cluster. Here comes a lot of files, that we will use later on for creating certificates for etcd, kubernetes, kubelet, and posibly others.

ca-config.json:

{
  "signing": {
    "default": {
      "expiry": "43800h"
    },
    "profiles": {
      "server": {
        "usages": ["signing", "key encipherment", "server auth"],
        "expiry": "43800h"
      },
      "client": {
        "usages": ["signing", "key encipherment",  "client auth"],
        "expiry": "43800h"
      },
      "peer": {
        "usages": ["signing", "key encipherment", "server auth", "client auth"],
        "expiry": "43800h"
      }
    }
  }
}
	  

ca-csr.json:

{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  }
}
	  

client-csr.json

{
  "CN": "client",
  "hosts": [ "" ],
  "key": {
      "algo": "ecdsa",
      "size": 256
  }
}
	  

For each server a server-csr.json

{
  "CN": "< hostname >",
  "hosts": [
      "localhost",
      "{{hostname}}"
  ],
  "key": {
      "algo": "ecdsa",
      "size": 256
  }
}
	  

Storage (etcd)

I find it easiest to start with etcd, as it is not a part of kubernetes, but is beeing used by it for storage, and is also a prerequisite for flannel (overlay network for docker). Here we will start setting up one secure etcd instance, and later on in another chapter we'll add two more nodes to have High Availability for storage. We will start by generating certificates with cfssl.

cfssl gencert -initca ca-csr.json | cfssljson -bare etcd-ca -

cfssl gencert \
 -ca=etcd-ca.pem \
 -ca-key=etcd-ca-key.pem \
 -config=ca-config.json \
 -hostname=localhost,{{hostname}} \
 -profile=server server-csr.json \
 | cfssljson -bare etcd-{{hostname}}-server

cfssl gencert \
 -ca=etcd-ca.pem \
 -ca-key=etcd-ca-key.pem \
 -config=ca-config.json \
 -hostname=localhost,{{hostname}} \
 -profile=peer server-csr.json \
 | cfssljson -bare etcd-{{hostname}}-peer

cfssl gencert \
 -ca=etcd-ca.pem \
 -ca-key=etcd-ca-key.pem \
 -config=ca-config.json \
 -hostname=localhost,{{hostname}} \
 -profile=client client-csr.json \
 | cfssljson -bare etcd-{{hostname}}-client

# this is the area we want to save data to, change at will, and change it to the same
# in the command under.
mkdir -p /var/lib/etcd 
	  

You should now have a server and peer for each server, and a client-cert. And we can start using etcd in a secure authenticated way. I am using relative paths and am standing where all certs are created, you can use absolute paths, bit for brevity this looks better. You could create a separate peer-ca-certificate and use that for peering. You can run this as a normal command:

/path/to/etcd \
  --name {{hostname}} \
  --cert-file=etcd-{{hostname}}-server.pem \
  --key-file=etcd-{{hostname}}-server-key.pem \
  --trusted-ca-file=etcd-ca.pem \
  --client-cert-auth \
  --peer-cert-file=etcd-{{hostname}}-peer.pem \
  --peer-key-file=etcd-{{hostname}}-peer-key.pem \
  --peer-ca-file=etcd-ca.pem \
  --peer-client-cert-auth \
  --listen-peer-urls=https://0.0.0.0:2380 \
  --listen-client-urls=https://0.0.0.0:2379,http://127.0.0.1:4001 \
  --advertise-client-urls=https://0.0.0.0:2379,http://127.0.0.1:4001 \
  --data-/var/lib/etcddir=/var/lib/etcd \
  --initial-advertise-peer-urls=https://{{hostname}}:2380 \
  --initial-cluster https://{{hostname}}:2380
	  

When working you could take etcdctl for a spin, I left http on port 4001 open on localhost so we don't have to use certificates when we are already on the server. But that is easily removed if you want to. to se what is saved at root type:

/path/to/etcdctl ls /
	  

If everything is working you have a working installation of etcd, and can start using it. You probably want to create a systemd-file for running it on startup.


Network (flannel)

Flanneld is a network overlay that keeps in order which subnet is used by docker on each host, so we define that flannel has a /16-net (255 /24-net) and that every docker-node should have a /24-net. It can be used without docker, for instance if you want docker-containers to connect over differend machines on a privete docker-ip-range In this guide I will use 18.16.0.0/16 as the main range, let's tell etcd about it!

flannel-etcd-conf.json

{
    "Network": "18.16.0.0/16",
    "SubnetLen": 24,
    "Backend": {
        "Type": "vxlan",
        "VNI": 1
     }
}
	  

and run the command:

/path/to/etcdctl set /flannel/network/config < flannel-etcd-conf.json
	  

We can now run flannel with

/path/to/flanneld \
  --ip-masq=false \
  --etcd-endpoints=https://{{etcd_master}}:2379 \
  --etcd-cafile=etcd-ca.pem \
  --etcd-certfile=etcd-{{hostname}}-client.pem \
  --etcd-keyfile=etcd-{{hostname}}-client-key.pem \
--etcd-prefix=/flannel/network

source /etc/sysconfig/docker-network # or where docker sets/gets network-configs on your os

/path/to/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker

#start docker your normal way
systemctl start docker
	  

Kubernetes

Kubernetes is a collection of programs connecting over a rest-api. Broadly speaking all components talk directly to the apiserver to get instructions, the apiserver stores state in etcd. Kubelet talks to docker, and docker uses flannel for nettwork.