Introduction
Kubernetes is a powerful set of tools for managing automatically scalable, highly available distributed cloud-native applications, but many people make some common mistakes.
In this article, we’ll explore some of the most common pitfalls when using Kubernetes and provide tips on how to avoid them.
Do not set resource requests
This is undeniably one of the most critical issues. CPU requests are frequently either unset or set at very low levels (to accommodate a large number of pods per node), leading to node overload. When demand surges, the node’s CPU becomes fully utilized, and our workload receives only the allocated resources, causing CPU throttling. This results in heightened application latency, timeouts, and other related issues.
Here are some example scenarios; proceed with caution:
BestEffort
resources: {}
Very low CPU performance
resources:
requests:
cpu: "1m"
Conversely, even when the node’s CPU isn’t fully engaged, setting a CPU limit too low can needlessly hinder pod operation, resulting in heightened latency.
There’s an ongoing debate about CPU throttling and its potential to exacerbate issues rather than resolve them, particularly concerning CPU CFS quotas in the Linux kernel and the effects of CPU settings. Turning off CFS quotas is a topic of discussion.
Therefore, it’s advisable not to overcommit memory and to employ guaranteed QoS (Quality of Service) by setting memory requests equal to the limit, as illustrated in the example below.
Burstable (more likely to be OOMkilled often)
resources:
requests:
memory: "128Mi"
cpu: "500m"
limits:
memory: "256Mi"
cpu: 2
Guaranteed
resources:
requests:
memory: "128Mi"
cpu: 2
limits:
memory: "128Mi"
cpu: 2
So what can help you when setting up your resources?
You can use metrics-server to view the current CPU and memory usage of a pod (and the containers within it). Chances are you’re already running it. Just run the following command:
kubectl top pods
kubectl top pods --containers
kubectl top nodes
However, these only show current usage. While you can get a rough idea of the numbers, ultimately you want to see these usage metrics in a timely manner (eg to help answer questions like: What was the CPU usage during peak hours, yesterday morning, etc.). For this you can use Prometheus, DataDog and many other tools. They simply ingest metrics from a metrics server, store them , and then query and chart them.
VerticalPodAutoscaler can help you automate this manual process – view CPU/memory usage in real time and reset new requests and limits based on these conditions .
Omit health checks
When deploying a service to Kubernetes, health checks play an important role in maintaining the service. In a Kubernetes environment, health check utilization is very low. Health checks allow you to keep tabs on the health of your pods and their containers.
Kubernetes has three main tools for health checking:
- Configuring Liveness Check allows Kubernetes to check whether the application is alive . The Kubelet agent running on each node uses liveness probes to ensure that containers are running as expected.
- Readiness Checks run throughout the container’s life cycle . Kubernetes uses this probe to understand when a container is ready to accept traffic.
- Startup Probe determines when a container application starts successfully . If the startup check fails, the pod will be restarted.
Use latest tag
The latest tag is generally undescriptive and difficult to use. Regarding the use of docker images in production environments: the latest label, the Kubernetes documentation makes it very clear:
When deploying containers in production, you should avoid using the :latest tag because it makes it difficult to track which version of the image is running and makes it difficult to roll back .
I feel like we’ve seen less of this lately because a lot of us have been through this so many times that we stopped using it and :latesteveryone started to have a fixed version.
Containers with excessive permissions
An over-authorized container refers to a container that has been granted too many permissions, such as accessing resources that ordinary containers cannot access. This is a common mistake developers make when using Kubernetes, and it poses a security risk.
For example, running a Docker daemon inside a Docker container is an example of a privileged container, which is not necessarily secure. To avoid this, it is recommended to avoid giving containers the CAP_SYS_ADMIN capability , as it accounts for more than 25% of all kernel vulnerabilities.
Additionally, it is important to avoid giving full permissions to the container and giving permissions to the container’s host file system. This means that the container can be used to replace the malicious binary, thereby compromising the entire host.
To prevent containers from being overprivileged, permission settings must be carefully configured and never run processes with higher permissions than required. Also, it is important to use monitoring and logging to detect and resolve issues.
Lack of monitoring and logging
A lack of monitoring and logging in a Kubernetes environment can harm its security and overall performance. Insufficient logging and monitoring can create challenges for incident investigation and response efforts, making it difficult to effectively detect and resolve issues.
A common pitfall is not being able to find points of failure in the Kubernetes platform and applications due to a lack of relevant logs or metrics .
To solve this problem, you must set up appropriate monitoring and logging tools such as Prometheus, Grafana, Fluentd, and Jaeger to collect, analyze, and visualize metrics, logs, and traces to gain insights into the performance and health of your Kubernetes environment.
By implementing strong monitoring and logging practices, enterprises can effectively correlate information, gain deeper insights, and overcome challenges associated with the dynamics and complexity of Kubernetes environments.
Default namespace for objects
Using a default namespace for all objects in Kubernetes creates organizational and management challenges. The default namespace is where services and applications are created by default and is also the active namespace unless explicitly specified.
Reliance entirely on the default namespace results in a lack of isolation and organization of different components or teams within the cluster. This can lead to difficulties with resource management, access control, and visibility. To avoid this, it is recommended to create custom namespaces for different projects, teams, or applications to enable better organization, resource allocation, and access control within the Kubernetes cluster .
By leveraging multiple namespaces, users can effectively divide and manage resources, improving the overall operating efficiency and security of the Kubernetes environment .
Missing Security configuration
Security should always be kept in mind when deploying applications. So, what are the most important things to consider when it comes to security? For example, using endpoints accessible outside the cluster, not protecting confidentiality, not considering how to safely run permissioned containers, etc.
Kubernetes security is an integral part of any Kubernetes deployment. Security challenges include:
- Authorization : Authentication and authorization are critical to controlling access to resources in a Kubernetes cluster.
- Networking : Kubernetes networking involves managing overlay networks and service endpoints to ensure that traffic between containers is routed securely within the cluster.
- Storage : Security of storage in a cluster includes ensuring that data cannot be accessed by unauthorized users or processes and that data is kept secure.
The Kubernetes API server has a REST interface that provides access to all stored information. This means that users can access any information stored in the API simply by sending an HTTP request to the API. To prevent unauthenticated users from accessing this data, you need to configure authentication for the API server using a supported method such as username/password or token-based authentication.
This is not only about the security of the cluster itself, but also about the security of secrets and configurations on the cluster. To protect your cluster from vulnerabilities, you need to configure a set of security controls on your cluster.
Securing a Kubernetes cluster using role-based access control (RBAC) is one such powerful security control: Role-based access control can be used to secure a Kubernetes cluster by restricting access to resources based on the roles assigned to the user . These roles can be configured as Administrator or Operator.
The Administrator role has full access rights, while the Operator role has limited rights to resources within the cluster. This way we can control and manage anyone who accesses the cluster.
Missing Pod Disruption Budget
You run production workloads on kubernetes. From time to time, your nodes and clusters need to be upgraded or decommissioned. PodDisruptionBudget (pdb) is a service assurance API between cluster administrators and cluster users.
Be sure to create pdbs to avoid unnecessary service interruptions due to node exhaustion.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: db-pdb
spec:
minAvailable: 2
selector:
matchLabels:
app: database
This way, as a cluster user, you can tell the cluster administrator that no matter what you do, you want at least two replicas to be available at all times.
Pod’s self-anti-paralysis function
For example, when running 3 copies of a pod of a deployment, if the node goes down, so will all the copies. Why?
This is because you cannot rely on the Kubernetes scheduler to enforce anti-affinite for your pods. You must define them clearly.
......
labels:
app: db
......
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: "app"
operator: In
values:
- db
topologyKey: "kubernetes.io/hostname"
That’s it. This will ensure that the pods are scheduled on different nodes (checked only at scheduling time, not execution time, therefore requiredDuringSchedulingIgnoredDuringExecution）.
We are talking about podAntiAffinity on different node names and not different availability zones, ie topologyKey："kubernetes.io/hostname. If you really need proper HA, learn more about this topic.
Load balancing per HTTP service
You may have more HTTP services in your cluster that you want to expose to the outside world.
If you make a kubernetes service type：LoadBalancerpublic, its controller (vendor specific) will provision and regulate external LoadBalancers, and these resources can become expensive (external static IPv4 addresses, per-second pricing…) because You need to create a lot of these resources.
In this case, it might make more sense to share an external load balancer that you can expose your service as type：NodePort. Alternatively, deploy something like nginx-ingress-controller (or traefik or Istio) as a single NodePort endpoint exposed to an external load balancer and route traffic across the cluster based on the kubernetes ingress resource.
Other (micro)services within the cluster that talk to each other can do so via the ClusterIP service and DNS service discovery out of the box .
NOTE: Do not use their public DNS/IP as this may impact their latency and cloud costs.
Unaware cluster automatic expansion
When adding and removing nodes from a cluster, simple metrics such as the CPU utilization of those nodes should not be considered. When scheduling pods, you need to make decisions based on a number of scheduling constraints such as pod and node affinities, taints and tolerances, resource requests, QoS, and more. If the external automatic scheduler does not understand these constraints, it can cause trouble.
Imagine that there is a new pod that needs to be scheduled, but all available CPUs are requested and the pod is stuck in the pending state. The external autoscaler sees the current CPU average used (not requested) and therefore does not scale (does not add another node). The pod cannot be scheduled.
Scaling in (removing a node from the cluster) is always more difficult. Suppose you have a stateful pod (with a persistent volume attached), since persistent volumes are typically resources that belong to a specific availability zone and are not replicated within the zone, your custom autoscaler removes the node with this pod, The scheduler cannot schedule it on other nodes because it is restricted to the only availability zone with persistent disks. The pod is stuck in pending status again.
Summarize
In summary, Kubernetes is a powerful tool for managing containerized applications, but it also comes with its own set of challenges. To avoid common mistakes and pitfalls, it’s important to pay close attention to your interactions with Kubernetes and understand the differences in how it interacts with deployed services.
Don’t expect everything to run automatically, invest some time in making your applications cloud native. By avoiding these mistakes, you can deploy Kubernetes efficiently and improve the stability, performance, and security of your Kubernetes environment.
Author
