Containerd basic Commands and Usage

In Kubernetes 1.24 and newer versions, opting for containerd as the runtime is advisable. Let’s explore the differences in directory structure between containerd and Docker.

What is containerd

Containerd is a container runtime that manages the complete container lifecycle, including image transfer, storage, execution, and network attachments. It supports the Open Container Initiative (OCI) runtime specification and is designed to be embedded into larger systems. Containerd is used by Kubernetes and other container orchestration tools for managing containers. It is open-source and part of the Cloud Native Computing Foundation (CNCF).

Basic Containerd filesystem locations

• When containerd pulls an image, it is stored in /var/lib/containerd/io.containerd.content.v1.content/blob/sha256.

• When containerd is the runtime, the standard output (stdout) log file is stored in /var/log/containers.
• For containerd as the runtime, all running files (merger layer) are stored in /run/containerd/io.containerd.runtime.v2.task/k8s.io/.
• In containerd as the runtime, both lower and upper layers exist in /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/.
• In containerd as the runtime, empty mounts of the container are stored in /var/lib/kubelet/pods/****/volumes/kubernetes.io~empty-dir/.

Containerd Installation

There are two installation methods. One involves installing Docker, as Docker itself includes containerd installation, requiring only minimal configuration adjustments. The other method is to directly download and use the binary package provided by the official GitHub, and we opt for the latter.

Download the containerd tar package and extract it to /usr/local.

tar Cxzvf /usr/local containerd-1.6.2-linux-amd64.tar.gz

Configure containerd:

containerd config default > /etc/containerd/config.toml

# Configure containerd to use systemd as cgroup
systemd_cgroup = true

# Adjust directories as needed
# 'root' stores persistent data including Snapshots, Content, Metadata, and various plugin data. Each plugin has its own directory, and containerd itself does not store any data. All its functionalities come from loaded plugins.
root = "/var/lib/containerd"

# 'state' stores temporary runtime data, including sockets, PID, mount points, runtime status, and plugin data that doesn't need persistence.
state = "/run/containerd"

If you prefer using systemd to manage containerd, create the /usr/lib/systemd/system/containerd.service file:

Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target

[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd

Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
LimitNPROC=655350
LimitCORE=655350
LimitNOFILE=655350
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target

After creating the service file, reload and enable auto-start at boot:

systemctl daemon-reload
systemctl enable --now containerd
systemctl restart containerd

Install runc

To install runc, which is a dependency for containerd, we need to download the latest version from the official GitHub.

Download the runc tar package:

# Execute this command to install runc.
install -m 755 runc.amd64 /usr/local/sbin/runc

This command sets the permissions and installs runc to /usr/local/sbin.

Install CNI

containerd lacks built-in network-related functionalities and requires the integration of a CNI (Container Network Interface) plugin. Otherwise, attempting to create a container will result in a CNI error.

To address this, download the CNI plugin tar package

$ mkdir -p /opt/cni/bin
$ tar Cxzvf /opt/cni/bin cni-plugins-linux-amd64-v1.2.0.tgz
./
./macvlan
./static
./vlan
./portmap
./host-local
./vrf
./bridge
./tuning
./firewall
./host-device
./sbr
./loopback
./dhcp
./ptp
./ipvlan
./bandwidth

# restart container
systemctl start containerd
systemctl status containerd

Management Tools

Here’s the information about management tools:

• ctr: This is a command line client provided as part of the containerd project. It’s primarily used for debugging purposes. You can check its help information using ctr --help.
• nerdctl: This tool is more user-friendly and compatible with Docker commands. It’s suitable for standalone nodes and personal use, providing an experience similar to Docker with minimal learning costs. You can replace Docker commands with nerdctl commands. More information can be found on the nerdctl official GitHub page.
• crictl: This command line client utilizes the Kubernetes CRI-compatible container runtime interface to manage containers and Pods primarily in Kubernetes environments. It’s primarily used for debugging purposes. You can find more information on the crictl official GitHub page.

Although these tools have their own characteristics, they may not be particularly easy to use. For example, when using nerdctl, mapping a host port to a container may not be immediately reflected in netstat -tnlp output on the host, despite the mapping being successful.

Here’s a comparison overview of the tools:

Comparing tools and commands, crictl is more recommended in Kubernetes scenarios. Here’s a comparison of commands:

To install a single-node Kubernetes cluster and test it, follow these steps:

1. First, configure the yum source:

cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

1. Install kubelet, kubectl, and kubeadm:

yum install -y kubelet-1.24.11 kubeadm-1.24.11 kubectl-1.24.11 --disableexcludes=kubernet

3. Initialize a single-node Kubernetes cluster:

kubeadm init --apiserver-advertise-address=192.168.22.226 --image-repository registry

To install a single-node Kubernetes cluster and test it, follow these steps:

1. First, configure the yum source:

cat > /etc/yum.repos.d/kubernetes.repo << EOF
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
 http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

2. Install kubelet, kubectl, and kubeadm:

yum install -y kubelet-1.24.11 kubeadm-1.24.11 kubectl-1.24.11 --disableexcludes=kubernetes

3. Initialize a single-node Kubernetes cluster:

kubeadm init --apiserver-advertise-address=192.168.22.226 --image-repository registry.aliyuncs.com/google_containers --control-plane-endpoint=192.168.22.226 --kubernetes-version v1.24.11 --service-cidr=10.1.0.0/16 --pod-network-cidr=10.244.0.0/16 --v=5

4. View all pods using nerdctl:

$ nerdctl -n k8s.io ps

CONTAINER ID    IMAGE                                                                       COMMAND                   CREATED           STATUS    PORTS    NAMES
01228b6dd482    registry.aliyuncs.com/google_containers/pause:3.6                           "/pause"                  13 minutes ago    Up                 k8s://kube-system/coredns-74586cf9b6-p42vf
2bdd9d045757    registry.aliyuncs.com/google_containers/pause:3.6                           "/pause"                  13 minutes ago    Up                 k8s://kube-system/kube-scheduler-dev01
37cbd0677e1a    registry.aliyuncs.com/google_containers/pause:3.6                           "/pause"                  13 minutes ago    Up                 k8s://kube-system/etcd-dev01
40ca90ad1c4f    registry.aliyuncs.com/google_containers/pause:3.6                           "/pause"                  13 minutes ago    Up                 k8s://kube-system/kube-controller-manager-dev01
451f34d736f1    registry.aliyuncs.com/google_containers/coredns:v1.8.6                      "/coredns -conf /etc…"    13 minutes ago    Up                 k8s://kube-system/coredns-74586cf9b6-jwt87/coredns
5152d7631cb2    registry.aliyuncs.com/google_containers/coredns:v1.8.6                      "/coredns -conf /etc…"    13 minutes ago    Up                 k8s://kube-system/coredns-74586cf9b6-p42vf/coredns
6040f26266b9    registry.aliyuncs.com/google_containers/kube-controller-manager:v1.24.11    "kube-controller-man…"    13 minutes ago    Up                 k8s://kube-system/kube-controller-manager-dev01/kube-controller-manager
73f778808fab    registry.aliyuncs.com/google_containers/pause:3.6                           "/pause"                  13 minutes ago    Up                 k8s://kube-system/kube-apiserver-dev01
9fa31ae29788    registry.aliyuncs.com/google_containers/etcd:3.5.6-0                        "etcd --advertise-cl…"    13 minutes ago    Up                 k8s://kube-system/etcd-dev01/etcd
a06a11848bfd    registry.aliyuncs.com/google_containers/pause:3.6                           "/pause"                  13 minutes ago    Up                 k8s://kube-system/coredns-74586cf9b6-jwt87
b30c34841895    registry.aliyuncs.com/google_containers/kube-apiserver:v1.24.11             "kube-apiserver --ad…"    13 minutes ago    Up                 k8s://kube-system/kube-apiserver-dev01/kube-apiserver
b82098336ac6    registry.aliyuncs.com/google_containers/kube-proxy:v1.24.11                 "/usr/local/bin/kube…"    13 minutes ago    Up                 k8s://kube-system/kube-proxy-j264w/kube-proxy
ba0cfb9ec381    registry.aliyuncs.com/google_containers/pause:3.6                           "/pause"                  13 minutes ago    Up                 k8s://kube-system/kube-proxy-j264w
df4c61aabb97    registry.aliyuncs.com/google_containers/kube-scheduler:v1.24.11             "kube-scheduler --au…"    13 minutes ago    Up                 k8s://kube-system/kube-scheduler-dev01/kube-scheduler

6. Alternatively, view all pods using crictl:

$ crictl ps

CONTAINER           IMAGE               CREATED             STATE               NAME                      ATTEMPT             POD ID              POD
5152d7631cb25       a4ca41631cc7a       11 minutes ago      Running             coredns                   0                   01228b6dd4828       coredns-74586cf9b6-p42vf
451f34d736f13       a4ca41631cc7a       11 minutes ago      Running             coredns                   0                   a06a11848bfd8       coredns-74586cf9b6-jwt87
b82098336ac6c       d6ccf30c8566e       11 minutes ago      Running             kube-proxy                0                   ba0cfb9ec381a       kube-proxy-j264w
b30c348418959       0849bf5f3ef4e       11 minutes ago      Running             kube-apiserver            0                   73f778808fab6       kube-apiserver-dev01
9fa31ae29788a       fce326961ae2d       11 minutes ago      Running             etcd                      0                   37cbd0677e1a1       etcd-dev01
df4c61aabb976       b1ef1dbb09ca5       11 minutes ago      Running             kube-scheduler            0                   2bdd9d045757a       kube-scheduler-dev01
6040f26266b93       4b0f3887b66e7       11 minutes ago      Running             kube-controller-manager   0                   40ca90ad1c4f8       kube-controller-manager-dev01

Directory Verification

To create a deployment and verify the storage path of containerd data on the host, follow these steps:

• Create the deployment YAML:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "2"
  labels:
    app: sample
  name: sample
  namespace: default
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: sample
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: sample
    spec:
      containers:
      - image: nginx:alpine
        imagePullPolicy: IfNotPresent
        name: nginx
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/log/app
          name: tmp-vol
      volumes:
      - emptyDir: {}
        name: tmp-vol
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      terminationGracePeriodSeconds: 30

• Viewing the pod UID:

uid:baf1e1a6-cf0e-474c-912d-e32a0d424c25

• Entering the pod container and creating a file:

$ cd /var/log/app/
$ echo hello >> app.log

• Entering the directory corresponding to the UID in the kubelet pods directory of the host machine:

$ cd /var/lib/kubelet/pods/baf1e1a6-cf0e-474c-912d-e32a0d424c25/volumes/kubernetes.io~empty-dir/tmp-vol/
$ cat app.log 
hello

• Creating a file within the /var/log/nginx directory of the container using touch test.txt.

Next, using crictl ps to check the container ID of the pod and entering the corresponding directory:

/run/containerd/io.containerd.runtime.v2.task/k8s.io/20220e35cc63208ac772bffed478d34a5bb54b0ffbb9a447d169f5b7415b3063/rootfs

This is equivalent to entering the / directory of the container, then entering /var/log/nginx. You can verify that test.txt also appears on the host machine, confirming the content we created above.

Conclusion

This time, I’ve simply documented some basic usage of containerd. As this article focuses on basic installation and exploration, our understanding of containerd remains at this foundational level. To uncover more of its wonderful capabilities, we must continue using and learning about it in the future.