Running your first Kata containers on the Kubernetes cluster

When I first meet the container technology and starting to use it a lot with Docker and Kubernetes later, I wondered about how to make it more secure and also how can I combine the old, good know, virtualization (KVM/QEMU) and modern containers technologies. Why? Because even independently from containers popularity they still only a cgroups and namespaces in shared host Linux kernel. When you use it for your own projects it’s OK, but if you share some Kubernetes cluster for the external customers, being as some kind of cloud provider for example? It can lead to big troubles when some of your customers miss with security, and their docker containers or Kubernetes pods will be hacked. As all these clients only neighbors in the “hostel” of Linux kernel.

And the second thing that I wanted from the containers and virtualization combination, the availability to run a real VMs under the modern containers orchestrators, like Kubernetes for example. That may sound strange, but if we use Kubernetes as the main cluster technology inside our infrastructure, it reasonable to try to use it for different kinds of tasks and not only for the containers.

Today I wanna focus on the containers security problem and show one of the ways of solving it by using cool Kata containers project, as the new runtime engine for Pods in the working Kubernetes cluster.

About Kata Containers

Kata Containers is an open source community working to build a secure container runtime with lightweight virtual machines that feel and perform like containers, but provide stronger workload isolation using hardware virtualization technology as a second layer of defense.

As you can see the Kata containers use the real light VM for any new Kubernetes Pod or Docker container, so it provides more strong separating between containers as they start using its own Linux kernels instead namespaces in the main host system kernel.

Sounds good and more secure as for me, so let’s try and test how it works.

Kata containers a bit young project and there still a lot of undocumented things in it, that why I decided to create an article about using it with the existing Kubernetes cluster.

First, there are links to get acquainted with this project:

Kata Containers Home | Kata Containers

kata-containers/documentation

OK, in this example we’ll run Kata containers on existing and working Kubernetes on-premise cluster, created with the help of Kubeadm utility.

Things that we need:

• Configured and working Kubernetes cluster v 1.14 or v 1.15
• One or more worker nodes in the cluster must be able to run Qemu as the virtualization platform, so these nodes must be bare-metal and have a real CPU with virtualization support. You can check this by running this command on your Kubernetes worker nodes:

kubernetes-node# cat /proc/cpuinfo | grep vmx

The Kata containers can only run with CRI-O or containerd, but not with dockershim that is the main CRI API for many Kubernetes clusters by default. If you didn’t specify any other CRI API when you created the cluster, you will get a dockershim on all your Kubernetes nodes.

Try to run:

kubernetes-node# docker ps

If you see any output, that means you are using dockershim CRI runtime for this host.

First, let’s change the CRI runtime on nodes that will be used for running Kata containers in the future. I’ll use the containerd for this example.

Very often the containerd are installed with Docker CE as an additional package, on my Debian hosts it was already installed:

kubernetes-node# dpkg -l | grep container
ii  containerd.io   1.2.5-1 amd64 An open and reliable container runtime
ii  docker-ce    5:18.09.4~3-0~debian-stretch   amd64 Docker: the open-source application container engine
ii  docker-ce-cli   5:18.09.4~3-0~debian-stretch   amd64 Docker CLI: the open-source application container engine

If nope, you can install it manually as a package or from source code.

OK, first of all, if the node that we’ll migrate to containerd have any running production pods, first drain this node or migrate all pods to other nodes. Then login to this node and remove cri plugin from the list ofdisabled_plugins in the containerd configuration file /etc/containerd/config.toml.

before 
disabled_plugins = ["cri"]
after 
disabled_plugins = [""]

Next, we need to prepare this node for the containerd:

kubernetes-node# modprobe overlay
kubernetes-node# modprobe br_netfilter
kubernetes-node# cat > /etc/sysctl.d/99-kubernetes-cri.conf <<EOF
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
kubernetes-node# sysctl --system

Now we need to configure Kubelet on this node for using containerd:

kubernetes-node# cat << EOF | sudo tee  /etc/systemd/system/kubelet.service.d/0-containerd.conf
[Service]                                                 
Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock"
EOF

On some systems, you also may need to add default config for kubelet:

kubernetes-node# vi /etc/default/kubelet
KUBELET_EXTRA_ARGS= --container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock

After that done, restart containerd and kubelet services on this node:

kubernetes-node# systemctl daemon-reload
kubernetes-node# systemctl restart containerd
kubernetes-node# systemctl restart kubelet

Now wait a bit till kubelet will change the runtime engine, and then try to run docker ps command again:

kubernetes-node#  docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

You must see the empty output, that means Kuibernetes start using containerd instead of docker as a container runtime.

Also, you can check this by running kubectl get nodes -o wide command:

kubectl# kubectl get nodes -o wide
NAME              STATUS .....  CONTAINER-RUNTIME
kubernetes-node   Ready  . .... containerd://1.2.5

For listing containers on this node, you now must use ctr command:

kubernetes-node# ctr --namespace k8s.io containers ls

Good, now we have the right configured kubelet and containerd and we can move on and install Kata containers inside our cluster. There’s are prepared deployments and some documentation about how to do it in the kata-deploy section of the repository. Clone it and use the readme file for additional information.

Run next commands for deploy Kata containers support into your Kubernetes cluster:

kubectl# kubectl apply -f https://raw.githubusercontent.com/kata-containers/packaging/master/kata-deploy/kata-rbac.yaml
kubectl# kubectl apply -f https://raw.githubusercontent.com/kata-containers/packaging/master/kata-deploy/kata-deploy.yaml
#For the Kubernetes 1.14 and 1.15 run:
kubectl# kubectl apply -f https://raw.githubusercontent.com/kata-containers/packaging/master/kata-deploy/k8s-1.14/kata-qemu-runtimeClass.yaml

After Kata containers installation the nodes that available for running it must get specific labels (katacontainers.io/kata-runtime=true):

kubectl# kubectl get nodes --show-labels | grep kata
kubernetes-node         Ready   <none>   5d23h   v1.15.0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,katacontainers.io/kata-runtime=true,kubernetes.io/arch=amd64,kubernetes.io/hostname=kubernetes-node,kubernetes.io/os=linux

If you get these labels on previously configured nodes, that means you can run Kata containers now. For testing it we can use example deployment from the kata-deploy repo:

kubectl# kubectl apply -f https://raw.githubusercontent.com/kata-containers/packaging/master/kata-deploy/examples/test-deploy-kata-qemu.yaml

Congrats, now you have a first running container on the base of real VM with Qemu virtualization.

Good luck.