Exploring Kubernetes : Basic Usage of ETCD

Introduction

In daily maintenance of kubernetes clusters, you will more or less need to deal with etcd. One of the annoying points about etcd is that it needs to specify a certificate, and the commands are also easy to forget, so I have sorted out the commands that are often used daily to prevent them from happening in the future. If you want to use it, you have to search online.

Common commands

Before using etcdctl, let’s first set up the basic environment to avoid needing to execute a long list of commands.

Replace ETCDCTL_ENDPOINTS with the IP address of your etcd node in your environment.

export ETCDCTL_API=3
export ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt
export ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt
export ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key
export ETCDCTL_ENDPOINTS=https://172.20.7.165:2379,https://172.20.7.166:2379,https://172.20.7.230:2379

View status

Check cluster node status

etcdctl endpoint status --write-out=table

Check cluster node health

etcdctl endpoint health

Delete member

List cluster node members

etcdctl member list

Remove cluster node members

etcdctl member remove xxxx

ETCD Backup

etcdctl snapshot save /path/to/backup.db

ETCD Recover

Execute this command once for each node, ensuring to modify the node name and IP accordingly.

ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd-master-212-snapshot.db \
  --data-dir=/var/lib/etcd \
  --name=etcd-master-212 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/peer.crt \
  --key=/etc/kubernetes/pki/etcd/peer.key \
  --initial-cluster-token=etcd-cluster-0 \
  --initial-cluster=etcd-master-212=https://10.20.176.212:2380,etcd-master-213=https://10.20.176.213:2380,etcd-master-214=https://10.20.176.214:2380 \
  --initial-advertise-peer-urls=https://10.20.176.212:2380

View Data

# --keys-only means only looking at the key, and the corresponding --print-value-only means only looking at the value

etcdctl get --prefix / --keys-only|head -10

View all pods

etcdctl get --prefix /registry/pods --keys-only|head -10

View pods in a namespace

etcdctl get --prefix /registry/pods/kube-system --keys-only|head -10

To retrieve the specific value of a key, note that resource values like pods are stored in protobuf format. As a result, using the etcdctl command directly for querying may display garbled characters. To decode proto format content, we can utilize the ‘etcdhelper‘ tool provided by OpenShift.

Download and compile the tool.

etcdhelper -cacert /etc/kubernetes/pki/etcd/ca.crt -cert /etc/kubernetes/pki/etcd/server.crt -key /etc/kubernetes/pki/etcd/server.key get /registry/pods/kube-system/etcd-172.20.7.165

/v1, Kind=Pod
{
  "kind": "Pod",
  "apiVersion": "v1",
  "metadata": {
    "name": "etcd-172.20.7.165",
    "namespace": "kube-system",
    "uid": "7d415149-d379-4c6f-880b-8b762a2a40a6",
    "creationTimestamp": "2024-01-05T09:02:48Z",
    "labels": {
      "component": "etcd",
      "tier": "control-plane"
    },
    "annotations": {
      "kubeadm.kubernetes.io/etcd.advertise-client-urls": "https://172.20.7.165:2379",
      "kubernetes.io/config.hash": "eb5fa33c2cfc8c18aa2aca4f2295eeb6",
      "kubernetes.io/config.mirror": "eb5fa33c2cfc8c18aa2aca4f2295eeb6",
      "kubernetes.io/config.seen": "2024-01-05T17:02:48.934117687+08:00",
      "kubernetes.io/config.source": "file",
      "seccomp.security.alpha.kubernetes.io/pod": "runtime/default"
    },
    "ownerReferences": [
      {
        "apiVersion": "v1",
        "kind": "Node",
        "name": "172.20.7.165",
        "uid": "de21e10c-c54e-4d5d-8e07-194db93c993d",
        "controller": true
      }
    ],
    ......

Conclusion

In this way we can view the resource information stored in etcd by kubernetes.