15 Kubernetes Scheduling Scenario Practical Guide

Kubernetes scheduling is a key component in ensuring that Pods in the cluster run on the appropriate nodes. By flexibly configuring scheduling policies, resource utilization, load balancing, and high availability can be improved.

In this article, we’ll delve into some practical Kubernetes scheduling scenarios and provide corresponding configuration examples and best practices.

1. Basic scenario – NodeSelector

Scenario description: We have some nodes marked with SSD hard drives, and we want to schedule Pods that require high-performance storage to these nodes.

Pod configuration:

apiVersion: v1
kind: Pod
metadata:
  name: high-performance-pod
spec:
  containers:
  - name: my-container
    image: my-image
  nodeSelector:
    disktype: ssd

2. Advanced Scenario-Node Affinity

Scenario description: We want to schedule tasks that require GPUs to nodes with GPU labels.

Pod configuration:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
spec:
  containers:
  - name: my-container
    image: my-image
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: gpu
            operator: In
            values:
            - "true"

3. Resource allocation – Pod priority and pre-selection scheduling

Scenario description: To ensure that critical tasks have higher priority, we can define a PriorityClass and apply it to the Pod.

PriorityClass configuration:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
globalDefault: false
description: "High priority class"

Pod configuration:

apiVersion: v1
kind: Pod
metadata:
  name: high-priority-pod
spec:
  containers:
  - name: my-container
    image: my-image
  priorityClassName: high-priority

4. Prevent Pods from running on the same node – Pod Anti-Affinity

Scenario description: Through Pod Anti-Affinity, we can ensure that Pods in the same group will not be scheduled to the same node to improve high availability.

Pod configuration:

apiVersion: v1
kind: Pod
metadata:
  name: anti-affinity-pod
spec:
  containers:
  - name: my-container
    image: my-image
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - web
        topologyKey: kubernetes.io/hostname

5. Multi-copy topology domain distribution – Pod Topology Spread

Scenario description: Ensure that multiple Pods of the same application are distributed in different topological domains to improve availability.

Deployment configuration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: "app"
                operator: In
                values:
                - "web"
            topologyKey: "kubernetes.io/hostname"

6. Node Taints and Pod Tolerations

Scenario description: Through the Taints of nodes, we can mark nodes, and only Pods with corresponding Tolerations can be scheduled to these nodes.

Node configuration:

apiVersion: v1
kind: Node
metadata:
  name: node1
spec:
  taints:
  - key: special
    value: unique
    effect: NoSchedule

Pod configuration:

apiVersion: v1
kind: Pod
metadata:
  name: toleration-pod
spec:
  containers:
  - name: my-container
    image: my-image
  tolerations:
  - key: "special"
    operator: "Equal"
    value: "unique"
    effect: "NoSchedule"

7. Custom Scheduler – Custom scheduler

Scenario description: Customize the scheduler to achieve specific scheduling requirements, such as based on business rules or special hardware conditions.

Custom scheduler example:

1. Create a custom scheduler plugin

// my_scheduler.go
package main

import (
	"k8s.io/kubernetes/pkg/scheduler"
	"k8s.io/kubernetes/pkg/scheduler/framework"
	"k8s.io/kubernetes/pkg/scheduler/framework/plugins/defaultbinder"
	"k8s.io/kubernetes/pkg/scheduler/framework/plugins/defaultpreemption"
	"k8s.io/kubernetes/pkg/scheduler/framework/plugins/names"
)

const (
	// YourSchedulerName is the name of your custom scheduler
	YourSchedulerName = "my-scheduler"
)

// New initializes a new scheduler with your custom plugins
func New() *scheduler.Config {
	return &scheduler.Config{
		Client:              scheduler.NewHTTPClient(),
		SchedulerName:       YourSchedulerName,
		PercentageOfNodesToScore: 0.25,
		Profiles: []scheduler.Profile{
			{
				Name: YourSchedulerName,
				Plugins: []scheduler.Plugin{
					defaultpreemption.Name: defaultpreemption.New,
					defaultbinder.Name:     defaultbinder.New,
					names.NewNodeResourcesFit(),
					names.NewNodePorts(),
					names.NewNodeAffinity(YourSchedulerName),
					names.NewNodeAffinityPriority(YourSchedulerName),
				},
			},
		},
	}
}

func main() {
	// Use the New() function to create a new scheduler with your custom plugins
	config := New()
	command := app.NewSchedulerCommand(
		// Use the WithConfig function to set your custom scheduler configuration
		app.WithConfig(config),
	)
	f := command.Flags()
	f.AddGoFlagSet(flag.CommandLine)

	if err := command.Execute(); err != nil {
		os.Exit(1)
	}
}

2. Compile and run a custom scheduler

go build my_scheduler.go
./my_scheduler

8. Pod Priority and Preemption – Pod priority and preemption

Scenario description: By setting the priority and preemption policy of the Pod, ensure that key tasks are prioritized.

Pod configuration:

apiVersion: v1
kind: Pod
metadata:
  name: priority-pod
spec:
  containers:
  - name: my-container
    image: my-image
  priorityClassName: high-priority

9. Resource Limits and Requests- Resource Limits and Requests

Scenario description: By setting resource limits and requests for Pods, the scheduler can better optimize resource utilization.

Pod configuration:

apiVersion: v1
kind: Pod
metadata:
  name: resource-pod
spec:
  containers:
  - name: my-container
    image: my-image
    resources:
      limits:
        cpu: "2"
        memory: "1Gi"
      requests:
        cpu: "1"
        memory: "500Mi"

10. Affinity and Anti-Affinity Rules – Affinity and Anti-Affinity Rules

Scenario description: Use affinity and anti-affinity rules to ensure that Pods are on specific nodes or avoid being scheduled to the same node as other Pods.

Pod configuration:

apiVersion: v1
kind: Pod
metadata:
  name: affinity-pod
spec:
  containers:
  - name: my-container
    image: my-image
  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: security
            operator: In
            values:
            - "high"
        topologyKey: kubernetes.io/hostname

11. Pod Disruption Budget – Pod disruption budget

Scenario description: Use the Pod interruption budget to limit the number of Pods allowed to be interrupted during maintenance to ensure system stability.

PodDisruptionBudget configuration:

apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: web-pdb
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: web

12. Horizontal Pod Autoscaler – Horizontal expander

Scenario description: Use horizontal expanders to automatically adjust the number of Pods based on CPU usage or other metrics to meet application needs.

HorizontalPodAutoscaler configuration:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: web-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web-deployment
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

13. Pod Overhead – Pod overhead

Scenario description: By setting the Pod overhead, tell the scheduler to consider the additional resources required by the Pod to avoid scheduling too many Pods on the node.

Pod configuration:

apiVersion: v1
kind: Pod
metadata:
  name: overhead-pod
spec:
  containers:
  - name: my-container
    image: my-image
    resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "500m"
    overhead:
      podFixed: 100Mi
      ephemeral-storage: 1Gi

14. Node Local DNS Cache – Node local DNS cache

Scenario description: Enable local DNS caching on the node to improve DNS query performance.

kubelet configuration:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
clusterDomain: cluster.local
featureGates:
  CoreDNSLocalCache: true

15. Pod Priority Class – Pod priority class

Scenario description: Use Pod priority classes to divide Pods into different priorities to ensure that key tasks are scheduled first.

PriorityClass configuration:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
globalDefault: false
description: "High priority class"
preemptionPolicy: PreemptLowerPriority

Check our last updated Kubernetes Exam Guides (CKAD , CKA , CKS) :

Conclusion

These scenarios cover Kubernetes scheduling practical cases from basic to advanced. According to your needs, you can choose the appropriate scenario for configuration to optimize the resource utilization and performance of the cluster. 

In actual applications, adjust the configuration according to specific needs to ensure that the scheduler’s strategy meets business and performance requirements.

Trivy is a powerful and easy-to-use tool for scanning Docker images, helping you maintain a secure Kubernetes environment.

Check last Kubernetes Exams (CKAD , CKA , CKS) Coupon Page to get discounts on certification registration.