Assign CPU Resources to Containers and Pods
A Container is guaranteed to have as much CPU as it requests, but is not allowed to use more CPU than its limit.
Before you begin
- Each node in your cluster must have at least 1 CPU.
- Have the
metrics-serverservice running in your cluster
To see whether metrics-server (or another provider of the resource metrics API, metrics.k8s.io) is running, type the following command:
kubectl get apiservices | grep metrics
v1beta1.metrics.k8s.io 1d
Create a namespace
Create a namespace so that the resources you create in this exercise are isolated from the rest of your cluster.
kubectl create namespace cpu-example
namespace "cpu-example" created
Specify a CPU request and a CPU limit
To specify a CPU request for a Container, include the resources:requests field in the Container resource manifest. To specify a CPU limit, include resources:limits.
You create a Pod that has one Container. The Container has a request of 0.5 CPU and a limit of 1 CPU. Here is the configuration file for the Pod
cat << EOF > cpu-request-limit.yaml
apiVersion: v1
kind: Pod
metadata:
name: cpu-demo
namespace: cpu-example
spec:
containers:
- name: cpu-demo-ctr
image: vish/stress
resources:
limits:
cpu: "1"
requests:
cpu: "0.5"
args:
- -cpus
- "2"
EOF
The args section of the configuration file provides arguments for the Container when it starts. The -cpus "2" argument tells the Container to attempt to use 2 CPUs.
Create the Pod:
kubectl create -f cpu-request-limit.yaml --namespace=cpu-example
pod "cpu-demo" created
Verify that the Pod Container is running:
kubectl get pod cpu-demo --namespace=cpu-example
NAME READY STATUS RESTARTS AGE
cpu-demo 0/1 Pending 0 10s
kubectl get pod cpu-demo --namespace=cpu-example
NAME READY STATUS RESTARTS AGE
cpu-demo 1/1 Running 0 3m
View detailed information about the Pod:
kubectl get pod cpu-demo --output=yaml --namespace=cpu-example
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: 2019-01-04T08:15:57Z
name: cpu-demo
namespace: cpu-example
resourceVersion: "218903"
selfLink: /api/v1/namespaces/cpu-example/pods/cpu-demo
uid: f6bcdf89-0ff8-11e9-8810-42010a8000ba
spec:
containers:
- args:
- -cpus
- "2"
image: vish/stress
imagePullPolicy: Always
name: cpu-demo-ctr
resources:
limits:
cpu: "1"
requests:
cpu: 500m
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-d55zh
readOnly: true
dnsPolicy: ClusterFirst
nodeName: gke-admatic-cluster-default-pool-e5aae271-spw8
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: default-token-d55zh
secret:
defaultMode: 420
secretName: default-token-d55zh
status:
conditions:
- lastProbeTime: null
lastTransitionTime: 2019-01-04T08:18:00Z
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: 2019-01-04T08:18:04Z
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: 2019-01-04T08:18:00Z
status: "True"
type: PodScheduled
containerStatuses:
- containerID: docker://6f8735513b1eac2f7a906200f46709628866e36621a3207b7b1a4fea6bc03575
image: vish/stress:latest
imageID: docker-pullable://vish/stress@sha256:b6456a3df6db5e063e1783153627947484a3db387be99e49708c70a9a15e7177
lastState: {}
name: cpu-demo-ctr
ready: true
restartCount: 0
state:
running:
startedAt: 2019-01-04T08:18:04Z
hostIP: 10.128.0.5
phase: Running
podIP: 10.4.3.3
qosClass: Burstable
startTime: 2019-01-04T08:18:00Z
The output shows that the one Container in the Pod has a CPU request of 500 milliCPU and a CPU limit of 1 CPU.
...
resources:
limits:
cpu: "1"
requests:
cpu: 500m
...
Use kubectl top to fetch the metrics for the pod:
kubectl top pod cpu-demo --namespace=cpu-example
NAME CPU(cores) MEMORY(bytes)
cpu-demo 974m 0Mi
The output shows that the Pod is using 974 milliCPU, which is just a bit less than the limit of 1 CPU specified in the Pod configuration file.
Recall that by setting -cpu "2", you configured the Container to attempt to use 2 CPUs, but the Container is only being allowed to use about 1 CPU. The Container CPU use is being throttled, because the Container is attempting to use more CPU resources than its limit.
kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
gke-admatic-cluster-default-pool-e5aae271-0b3v 38m 4% 777Mi 29%
gke-admatic-cluster-default-pool-e5aae271-9tk6 37m 3% 733Mi 27%
gke-admatic-cluster-default-pool-e5aae271-spw8 986m 104% 482Mi 18%
gke-admatic-cluster-default-pool-e5aae271-vt5z 48m 5% 815Mi 30%
Note: Another possible explanation for the CPU throttling is that the Node might not have enough CPU resources available. Recall that the prerequisites for this exercise require each of your Nodes to have at least 1 CPU. If your Container runs on a Node that has only 1 CPU, the Container cannot use more than 1 CPU regardless of the CPU limit specified for the Container.
CPU units
The CPU resource is measured in CPU units. One CPU, in Kubernetes, is equivalent to:
- 1 AWS vCPU
- 1 GCP Core
- 1 Azure vCore
- 1 Hyperthread on a bare-metal Intel processor with Hyperthreading
Fractional values are allowed. A Container that requests 0.5 CPU is guaranteed half as much CPU as a Container that requests 1 CPU. You can use the suffix m to mean milli. For example 100m CPU, 100 milliCPU, and 0.1 CPU are all the same. Precision finer than 1m is not allowed.
CPU is always requested as an absolute quantity, never as a relative quantity; 0.1 is the same amount of CPU on a single-core, dual-core, or 48-core machine.
Delete your Pod:
kubectl delete pod cpu-demo --namespace=cpu-example
pod "cpu-demo" deleted
Specify a CPU request that is too big for your Nodes
CPU requests and limits are associated with Containers, but it is useful to think of a Pod as having a CPU request and limit. The CPU request for a Pod is the sum of the CPU requests for all the Containers in the Pod. Likewise, the CPU limit for a Pod is the sum of the CPU limits for all the Containers in the Pod.
Pod scheduling is based on requests. A Pod is scheduled to run on a Node only if the Node has enough CPU resources available to satisfy the Pod CPU request.
You create a Pod that has a CPU request so big that it exceeds the capacity of any Node in your cluster. Here is the configuration file for a Pod that has one Container. The Container requests 2 CPU, which exceeds the capacity of any Node in our cluster.
cat << EOF > cpu-request-limit-2.yaml
apiVersion: v1
kind: Pod
metadata:
name: cpu-demo-2
namespace: cpu-example
spec:
containers:
- name: cpu-demo-ctr-2
image: vish/stress
resources:
limits:
cpu: "4"
requests:
cpu: "2"
args:
- -cpus
- "2"
EOF
Create the Pod:
kubectl create -f cpu-request-limit-2.yaml --namespace=cpu-example
pod "cpu-demo-2" created
View the Pod status:
kubectl get pod cpu-demo-2 --namespace=cpu-example
NAME READY STATUS RESTARTS AGE
cpu-demo-2 0/1 Pending 0 14s
The output shows that the Pod status is Pending. That is, the Pod has not been scheduled to run on any Node, and it will remain in the Pending state indefinitely:
View detailed information about the Pod, including events:
kubectl describe pod cpu-demo-2 --namespace=cpu-example
Name: cpu-demo-2
Namespace: cpu-example
Node: <none>
Labels: <none>
Annotations: <none>
Status: Pending
IP:
Containers:
cpu-demo-ctr-2:
Image: vish/stress
Port: <none>
Host Port: <none>
Args:
-cpus
2
Limits:
cpu: 4
Requests:
cpu: 2
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-d55zh (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
default-token-d55zh:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-d55zh
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 17s (x7 over 48s) default-scheduler 0/4 nodes are available: 4 Insufficient cpu.
Normal NotTriggerScaleUp 8s (x4 over 44s) cluster-autoscaler pod didn't trigger scale-up (it wouldn't fit if a new node is added)
The output shows that the Container cannot be scheduled because of insufficient CPU resources on the Nodes:
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 17s (x7 over 48s) default-scheduler 0/4 nodes are available: 4 Insufficient cpu.
...
Delete your Pod:
kubectl delete pod cpu-demo-2 --namespace=cpu-example
pod "cpu-demo-2" deleted
If you do not specify a CPU limit
If you do not specify a CPU limit for a Container, then one of these situations applies:
- The Container has no upper bound on the CPU resources it can use. The Container could use all of the CPU resources available on the Node where it is running.
- The Container is running in a namespace that has a default CPU limit, and the Container is automatically assigned the default limit. Cluster administrators can use a
LimitRangeto specify a default value for the CPU limit.
Motivation for CPU requests and limits
By configuring the CPU requests and limits of the Containers that run in your cluster, you can make efficient use of the CPU resources available on your cluster Nodes. By keeping a Pod CPU request low, you give the Pod a good chance of being scheduled. By having a CPU limit that is greater than the CPU request, you accomplish two things:
- The Pod can have bursts of activity where it makes use of CPU resources that happen to be available.
- The amount of CPU resources a Pod can use during a burst is limited to some reasonable amount.
Clean up
Delete your namespace:
kubectl delete namespace cpu-example
namespace "cpu-example" deleted