Autoscaling clusters
With the cluster-autoscaler add-on, we can scale the worker pools in your Red Hat OpenShift on IBM Cloud classic or VPC cluster automatically to increase or decrease the number of worker nodes in the worker pool based on the sizing needs of our scheduled workloads. The cluster-autoscaler add-on is based on the Kubernetes Cluster-Autoscaler project.
Want to autoscale the pods instead? Check out Scaling apps.
Understand scale-up and scale-down
The cluster autoscaler periodically scans the cluster to adjust the number of worker nodes within the worker pools that it manages in response to the workload resource requests and any custom settings that you configure, such as scanning intervals. Every minute, the cluster autoscaler checks for the following situations.- Pending pods to scale up: A pod is considered pending when insufficient compute resources exist to schedule the pod on a worker node. When the cluster autoscaler detects pending pods, the autoscaler scales up your worker nodes evenly across zones to meet the workload resource requests.
- Underutilized worker nodes to scale down: By default, worker nodes that run with less than 50% of the total compute resources that are requested for 10 minutes or more and that can reschedule their workloads onto other worker nodes are considered underutilized. If the cluster autoscaler detects underutilized worker nodes, it scales down your worker nodes one at a time so that we have only the compute resources that we need. If we want, we can customize the default scale-down utilization threshold of 50% for 10 minutes.
Scanning and scaling up and down happens at regular intervals over time, and depending on the number of worker nodes might take a longer period of time to complete, such as 30 minutes.
The cluster autoscaler adjusts the number of worker nodes by considering the resource requests that you define for the deployments, not actual worker node usage. If the pods and deployments don't request appropriate amounts of resources, we must adjust their configuration files. The cluster autoscaler can't adjust them for you. Also, keep in mind that worker nodes use some of their compute resources for basic cluster functionality, default and custom add-ons, and resource reserves.
What does scaling up and down look like?
In general, the cluster autoscaler calculates the number of worker nodes that the cluster needs to run its workload. Scaling the cluster up or down depends on many factors, including
the following.
- The minimum and maximum worker node size per zone that you set.
- Your pending pod resource requests and certain metadata that you associate with the workload, such as anti-affinity, labels to place pods only on certain flavors, or pod disruption budgets.
- The worker pools that the cluster autoscaler manages, potentially across zones in a multizone cluster.
- The custom Helm chart values that are set, such as skipping worker nodes for deletion if they use local storage.
For more information, see the Kubernetes Cluster Autoscaler FAQs for How does scale-up work? and How does scale-down work?.
Can I change how scale-up and scale-down work?
We can customize settings or use other Kubernetes resources to affect how scaling up and down work.
- Scale-up: Customize the cluster autoscaler configmap values such as scanInterval, expander, skipNodes, or maxNodeProvisionTime. Review ways to overprovision worker nodes so that we can scale up worker nodes before a worker pool runs out of resources. We can also set up Kubernetes pod budget disruptions and pod priority cutoffs to affect how scaling up works.
- Scale-down: Customize the cluster autoscaler configmap values such as scaleDownUnneededTime, scaleDownDelayAfterAdd, scaleDownDelayAfterDelete, or scaleDownUtilizationThreshold.
Can I increase the minimum size per zone to trigger a scale up my cluster to that size?
No, setting a minSize does not automatically trigger a scale-up. The minSize is a threshold so that the cluster
autoscaler does not scale below a certain number of worker nodes per zone. If the cluster does not yet have that number per zone, the cluster autoscaler does not scale up until we have workload resource requests that require more resources.
For example, if we have a worker pool with one worker node per three zones (three total worker nodes) and set the minSize to 4 per zone, the cluster autoscaler does not immediately provision an additional three worker
nodes per zone (12 worker nodes total). Instead, the scale-up is triggered by resource requests. If you create a workload that requests the resources of 15 worker nodes, the cluster autoscaler scales up the worker pool to meet this request.
Now, the minSize means that the cluster autoscaler does not scale down below four worker nodes per zone even if you remove the workload that requests the amount.
How is this behavior different from worker pools that are not managed by the cluster autoscaler?
When you create a worker pool, you specify how many worker
nodes per zone it has. The worker pool maintains that number of worker nodes until you resize or rebalance it. The worker pool does not add or remove worker nodes for you. If we have more pods than can be scheduled, the pods remain in pending state until you resize the worker pool.
When you enable the cluster autoscaler for a worker pool, worker nodes are scaled up or down in response to the pod spec settings and resource requests. You don't need to resize or rebalance the worker pool manually.
Can I see an example of how the cluster autoscaler scales up and down?
Consider the following image for an example of scaling the cluster up and down.
Figure: Autoscaling a cluster up and down.
- The cluster has four worker nodes in two worker pools that are spread across two zones. Each pool has one worker node per zone, but Worker Pool A has a flavor of u2c.2x4 and Worker Pool B has a flavor of b2c.4x16. Your total compute resources are roughly 10 cores (2 cores x 2 worker nodes for Worker Pool A, and 4 cores x 2 worker nodes for Worker Pool B). Your cluster currently runs a workload that requests 6 of these 10 cores. Additional computing resources are taken up on each worker node by the reserved resources that are required to run the cluster, worker nodes, and any add-ons such as the cluster autoscaler.
- The cluster autoscaler is configured to manage both worker pools with the following minimum and maximum size-per-zone:
- Worker Pool A: minSize=1, maxSize=5.
- Worker Pool B: minSize=1, maxSize=2.
- You schedule deployments that require 14 additional pod replicas of an app that requests one core of CPU per replica. One pod replica can be deployed on the current resources, but the other 13 are pending.
- The cluster autoscaler scales up your worker nodes within these constraints to support the additional 13 pod replicas resource requests.
- Worker Pool A: Seven worker nodes are added in a round-robin method as evenly as possible across the zones. The worker nodes increase the cluster compute capacity by roughly 14 cores (2 cores x 7 worker nodes).
- Worker Pool B: Two worker nodes are added evenly across the zones, reaching the maxSize of two worker nodes per zone. The worker nodes increase cluster capacity by roughly 8 cores (4 cores x 2 worker node).
- The 20 pods with one-core requests are distributed as follows across the worker nodes. Because worker nodes have resource reserves as well as pods that run to cover default cluster features, the pods for the workload cannot use all the available
compute resources of a worker node. For example, although the b2c.4x16 worker nodes have four cores, only three pods that request a minimum of one core each can be scheduled onto the worker nodes.
Worker Pool Zone Type # worker nodes # pods A dal10 u2c.2x4 Four nodes Three pods A dal12 u2c.2x4 Five nodes Five pods B dal10 b2c.4x16 Two nodes Six pods B dal12 b2c.4x16 Two nodes Six pods - You no longer need the additional workload, so you delete the deployment. After a short period of time, the cluster autoscaler detects that the cluster no longer needs all its compute resources and scales down the worker nodes one at a time.
- Your worker pools are scaled down. The cluster autoscaler scans at regular intervals to check for pending pod resource requests and underutilized worker nodes to scale your worker pools up or down.
Following scalable deployment practices
Make the most out of the cluster autoscaler by using the following strategies for the worker node and workload deployment strategies. For more information, see the Kubernetes Cluster Autoscaler FAQs.Try out the cluster autoscaler with a few test workloads to get a good feel for how scale-up and scale-down work, what custom values we might want to configure, and any other aspects that we might want, like overprovisioning worker nodes or limiting apps. Then, clean up your test environment and plan to include these custom values and additional settings with a fresh installation of the cluster autoscaler.
Can I autoscale multiple worker pools at once?
Yes, after you install the cluster autoscaler, we can choose which worker pools within the cluster to autoscale in the configmap. We can run only one autoscaler per cluster. Create and enable autoscaling on worker pools other than the default worker pool, because the default worker pool has system components that can prevent automatically scaling down.How can I make sure that the cluster autoscaler responds to what resources my app needs?
The cluster autoscaler scales the cluster in response to the workload resource requests. As such, specify resource requests for all the deployments because the resource requests are what the cluster autoscaler uses to calculate how many worker nodes are needed to run the workload. Keep in mind that autoscaling is based on the compute usage that the workload configurations request, and does not consider other factors such as machine costs.Can I scale down a worker pool to zero (0) nodes?
No, we cannot set the cluster autoscaler minSize to 0. Additionally, unless you disable all public application load balancers (ALBs) in each zone of the cluster, we must change the minSize to 2 worker nodes per zone so that the ALB pods can be spread for high availability. Additionally, we can taint your worker pool to achieve a scale to down a minimum of 1.If your worker pool has zero (0) worker nodes, the worker pool cannot be scaled. Disable cluster autoscaling for the worker pool, manually resize the worker pool to at least one, and re-enable cluster autoscaling.
Can I optimize my deployments for autoscaling?
Yes, we can add several Kubernetes features to the deployment to adjust how the cluster autoscaler considers your resource requests for scaling.- Taint your worker pool to allow only the deployments or pods with the matching toleration to be deployed to your worker pool.
- Add a label to your worker pool other than the default worker pool. This label is used in the deployment configuration to specify nodeAffinity or nodeSelector which limits the workloads that can be deployed on the worker nodes in the labeled worker pool.
- Use pod disruption budgets to prevent abrupt rescheduling or deletions of our pods.
- If you're using pod priority, we can edit the priority cutoff to change what types of priority trigger scaling up. By default, the priority cutoff is zero (0).
Can I use taints and tolerations with autoscaled worker pools?
Yes, but make sure to apply taints at the worker pool level so that all existing and future worker nodes get the same taint. Then, we must include a matching toleration in the workload configuration so that these workloads are scheduled onto your autoscaled worker pool with the matching taint. Keep in mind that if we deploy a workload that is not tolerated by the tainted worker pool, the worker nodes are not considered for scale-up and more worker nodes might be ordered even if the cluster has sufficient capacity. However, the tainted worker pool is still identified as underutilized if they have less than the threshold (by default 50%) of their resources utilized and thus are considered for scale-down.Why are my autoscaled worker pools unbalanced?
During a scale-up, the cluster autoscaler balances nodes across zones, with a permitted difference of plus or minus one (+/- 1) worker node. Your pending workloads might not request enough capacity to make each zone balanced. In this case, if we want to manually balance the worker pools, update the cluster autoscaler configmap to remove the unbalanced worker pool. Then run the ibmcloud oc worker-pool rebalance command, and add the worker pool back to the cluster autoscaler configmap.Why can't I resize or rebalance my worker pool?
When the cluster autoscaler is enabled for a worker pool, we cannot resize or rebalance your worker pools. You must edit the configmap to change the worker pool minimum or maximum sizes, or disable cluster autoscaling for that worker pool. Don't use the ibmcloud oc worker rm command to remove individual worker nodes from your worker pool, which can unbalance the worker pool.Further, if you do not disable the worker pools before you disable the cluster-autoscaler add-on, the worker pools cannot be resized manually. Reinstall the cluster autoscaler, edit the configmap to disable the worker pool, and try again.
Preparing the cluster for autoscaling
Before you install the IBM Cloud cluster autoscaler add-on, we can set up the cluster to prepare the cluster for autoscaling.- Install the required CLI and plug-ins:
- IBM Cloud CLI (ibmcloud)
- IBM Cloud Kubernetes Service plug-in (ibmcloud oc alias for OpenShift clusters)
- IBM Cloud Container Registry plug-in (ibmcloud cr)
- Kubernetes (kubectl)
- Create a standard cluster.
- Access the OpenShift cluster.
- Confirm that your IBM Cloud Identity and Access Management credentials are stored in the cluster. The cluster autoscaler uses this secret to authenticate credentials. If the secret is missing, create it by resetting credentials.
oc get secrets -n kube-system | grep storage-secret-store
- Plan to autoscale a worker pool other than the default worker pool, because the default worker pool has system components that can prevent automatically scaling down. Include a label for the worker pool so that you
can set node affinity for the workloads that we want to deploy to the worker pool that has autoscaling
enabled. For example, your label might be app: nginx. Choose from the following options:
- Create a VPC or classic worker pool other than the default worker pool with the label that we want to use with the workloads to run on the autoscaled worker pool.
- Add the label to an existing worker pool other than the default worker pool.
-
Confirm that your worker pool has the necessary labels for autoscaling. In the output, you see the required ibm-cloud.kubernetes.io/worker-pool-id label and the label that you previously created for node affinity. If you do not see these labels, add a new worker pool, and then add your label for node affinity.
ibmcloud oc worker-pool get --cluster <cluster_name_or_ID> --worker-pool <worker_pool_name_or_ID> | grep Labels
Example output of a worker pool with the label:
Labels: ibm-cloud.kubernetes.io/worker-pool-id=a1aa111111b22b22cc3c3cc444444d44-4d555e5
-
Taint the worker pools that we want to autoscale so that the worker pool does not accept workloads except the ones that we want to run on the autoscaled worker pool. We can learn more about taints and tolerations in the community Kubernetes documentation. As an example, we might set a taint of use=autoscale:NoExecute. In this example, the NoExecute toleration evicts and pods that do not have the matching the toleration.
-
Install the cluster autoscaler add-on.
We might also install the cluster autoscaler Helm chart, but the Helm chart is deprecated and becomes unsupported tentatively 15 September 2020. We cannot install the add-on and the Helm chart in the same cluster at the same time.
Installing the cluster autoscaler add-on in the cluster
-
If you previously installed the cluster autoscaler Helm chart, make a backup of our configmap, disable autoscaling, and remove the Helm chart.
The cluster autoscaler add-on configmap contains different parameters than the cluster autoscaler Helm chart configmap. We can make a backup of our Helm chart configmap to use as reference, but do not apply the backup directly to your cluster with the add-on. Instead, we can update the cluster autoscaler add-on confimap with the values from your Helm chart backup.
-
Install the cluster autoscaler add-on to the cluster.
In the console
- From the Red Hat OpenShift on IBM Cloud cluster dashboard, select the cluster where we want to enable autoscaling.
- On the Overview page, click Add-ons.
- On the Add-ons page, locate the Cluster Autoscaler add-on and click Install
In the CLI
-
Install the cluster-autoscaler add-on.
ibmcloud oc cluster addon enable cluster-autoscaler —-cluster <cluster_name>
Example output
Enabling add-on `cluster-autoscaler` for cluster <cluster_name>... The add-on might take several minutes to deploy and become ready for use. OK
-
Verify that the add-on is installed and Ready.
ibmcloud oc cluster addon ls --cluster <cluster_name>
Example output
Name Version Health State Health Status cluster-autoscaler 1.0.1 normal Addon Ready
-
By default, no worker pools are enabled for autoscaling. To enable autoscaling on the worker pools, update the cluster autoscaler configmap to enable scaling for the worker pools.
Installing the cluster autoscaler Helm chart in the cluster
The cluster autoscaler Helm chart is deprecated. For the latest version of the cluster autoscaler, install the add-on
Install the IBM Cloud cluster autoscaler plug-in with a Helm chart to autoscale worker pools in the cluster.- Prepare the cluster for autoscaling.
- Follow the instructions to install the Helm version 3 client on the local machine.
- Add and update the Helm repo where the cluster autoscaler Helm chart is.
helm repo add iks-charts https://icr.io/helm/iks-charts
helm repo update
-
Decide if we want to customize the cluster autoscaler settings, such as the worker pools that are autoscaled, or the amount of time that the cluster autoscaler waits before scaling worker nodes up or down. We can customize your settings by using the --set flag in the helm install command. Depending on the settings that we want to customize, we might need to prepare multiple --set flags before you can install the Helm chart. For example, we might want to autoscale your default worker pool by preparing the following --set flag. Note: If your default shell is zsh, start a bash session before running the following command.
--set workerpools[0].<pool_name>.max=<number_of_workers>,workerpools[0].<pool_name>.min=<number_of_workers>,workerpools[0].<pool_name>.enabled=(true|false)
Understand the --set workerpools options:
- workerpools[0]: The first worker pool to enable or disable for autoscaling. You must include three parameters for each worker pool for the command to succeed: the maximum number of worker nodes (max), the minimum number of worker nodes (min), and whether we want to enable (true) or disable (false) autoscaling for this worker pool. To include multiple worker pools, include a comma-separated list and increase the number in brackets, such as: workerpools[0].default...,workerpools[1].pool1...,workerpools[2].pool2....
- <pool_name>: The name or ID of the worker pool that we want to enable or disable for autoscaling. To list available worker pools, run ibmcloud oc worker-pool ls --cluster <cluster_name_or_ID>.
- max=<number_of_workers>: Specify the maximum number of worker nodes per zone that the cluster autoscaler can scale up to. The value must be equal to or greater than the value that you set for the min=<number_of_workers> size.
- min=<number_of_workers>: Specify the minimum number of worker nodes per zone that the cluster autoscaler can scale down to. The value must be 2 or greater so that your ALB pods can be spread
for high availability. If you disabled all public ALBs in each zone of our standard cluster, we can set the value to 1.
Keep in mind that setting a min size does not automatically trigger a scale-up. The min size is a threshold so that the cluster autoscaler does not scale below this minimum number of worker nodes per zone. If the cluster does not have this number of worker nodes per zone yet, the cluster autoscaler does not scale up until we have workload resource requests that require more resources.
- enabled=(true|false): Set the value to true to enable the cluster autoscaler to scale your worker pool. Set the value to false to stop the cluster autoscaler from scaling the worker pool. Later, if we want to remove the cluster autoscaler, we must first disable each worker pool in the configmap.
-
Install the cluster autoscaler Helm chart in the kube-system namespace of the cluster. In the example command, the autoscale worker pool is enabled for autoscaling with the Helm chart installation. The worker pool details are added to the cluster autoscaler config map.
Helm 3 install commandHelm 2 install commandScroll for moreScroll for more
Scroll for moreScroll for moreCommand helm install ibm-iks-cluster-autoscaler iks-charts/ibm-iks-cluster-autoscaler --namespace kube-system --set workerpools[0].default.max=5,workerpools[0].autoscale.min=2,workerpools[0].default.enabled=true
Command helm install iks-charts/ibm-iks-cluster-autoscaler --namespace kube-system --name ibm-iks-cluster-autoscaler --set workerpools[0].autoscale.max=5,workerpools[0].default.min=2,workerpools[0].default.enabled=true
NAME: ibm-iks-cluster-autoscaler LAST DEPLOYED: Fri Jan 17 12:20:30 2020 NAMESPACE: kube-system STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: Thank you for installing: ibm-iks-cluster-autoscaler. Your release is named: ibm-iks-cluster-autoscaler For more information about using the cluster autoscaler, refer to the chart README.md file.
-
Verify that the installation is successful.
- Check that the cluster autoscaler pod is in a Running state.
oc get pods --namespace=kube-system | grep ibm-iks-cluster-autoscaler
Example output:ibm-iks-cluster-autoscaler-8497bfc968-dbn7w 1/1 Running 0 9m
- Check that the cluster autoscaler service is created.
oc get service --namespace=kube-system | grep ibm-iks-cluster-autoscaler
Example output:ibm-iks-cluster-autoscaler ClusterIP 172.21.xxx.xx <none> 8085/TCP 9m
-
Optional: If you enabled autoscaling for a worker pool during the Helm chart installation, verify that the config map is correct by checking that the workerPoolsConfig.json field is updated and that the workerPoolsConfigStatus field shows a SUCCESS message.
oc get cm iks-ca-configmap -n kube-system -o yaml
Example output where the default worker pool is enabled for autoscaling:
apiVersion: v1 data: workerPoolsConfig.json: | [{"name": "autoscale", "minSize": 1, "maxSize": 2, "enabled": true }] kind: ConfigMap metadata: annotations: workerPoolsConfigStatus: '{"1:2:default":"SUCCESS"}' creationTimestamp: "2019-08-23T14:26:54Z" name: iks-ca-configmap namespace: kube-system resourceVersion: "12757878" selfLink: /api/v1/namespaces/kube-system/configmaps/iks-ca-configmap uid: bd661f95-35ef-433d-97e0-5d1ac092eafb
- Check that the cluster autoscaler pod is in a Running state.
-
Repeat these steps for every cluster where we want to provision the cluster autoscaler.
-
Optional: If you did not set any worker pools for autoscaling with the installation, we can Update the cluster autoscaler configuration.
Updating the cluster autoscaler configmap to enable scaling
Update the cluster autoscaler configmap to enable automatically scaling worker nodes in your worker pools based on the minimum and maximum values that you set.After you edit the configmap to enable a worker pool, the cluster autoscaler scales the cluster in response to the workload requests. As such, we cannot resize or rebalance your worker pools. Scanning and scaling up and down happens at regular intervals over time, and depending on the number of worker nodes might take a longer period of time to complete, such as 30 minutes. Later, if we want to remove the cluster autoscaler, we must first disable each worker pool in the configmap.
Before beginning:
- Install the cluster autoscaler add-on or the Helm chart.
- Access the OpenShift cluster.
To update the cluster autoscaler configmap and values:
-
Edit the cluster autoscaler configmap YAML file.
oc edit cm iks-ca-configmap -n kube-system -o yaml
Example output
apiVersion: v1 data: workerPoolsConfig.json: | [ {"name": "<worker_pool>","minSize": 1,"maxSize": 2,"enabled":false} ] kind: ConfigMap -
Edit the configmap with the parameters to define how the cluster autoscaler scales the cluster worker pool. Note: Unless you disabled all public application load balancers (ALBs) in each zone of our standard cluster, we must change the minSize to 2 per zone so that the ALB pods can be spread for high availability.
Parameter with default value Description "name": "default" Replace "default" with the name or ID of the worker pool that we want to scale. To list worker pools, run ibmcloud oc worker-pool ls --cluster <cluster_name_or_ID>.
To manage more than one worker pool, copy the JSON line to a comma-separated line, such as follows.[ {"name": "default","minSize": 1,"maxSize": 2,"enabled":false}, {"name": "Pool2","minSize": 2,"maxSize": 5,"enabled":true} ]
The cluster autoscaler can scale only worker pools that have the ibm-cloud.kubernetes.io/worker-pool-id label. To check whether your worker pool has the required label, run ibmcloud oc worker-pool get --cluster <cluster_name_or_ID> --worker-pool <worker_pool_name_or_ID> | grep Labels. If your worker pool does not have the required label, add a new worker pool and use this worker pool with the cluster autoscaler.
"minSize": 1 Specify the minimum number of worker nodes per zone that the cluster autoscaler can scale down the worker pool to. The value must be 2 or greater so that your ALB pods can be spread for high availability. If you disabled all public ALBs in each zone of our standard cluster, we can set the value to 1. Set a minSize does not automatically trigger a scale-up. The minSize is a threshold so that the cluster autoscaler does not scale below a certain number of worker nodes per zone. If the cluster does not yet have that number per zone, the cluster autoscaler does not scale up until we have workload resource requests that require more resources. For example, if we have a worker pool with one worker node per three zones (three total worker nodes) and set the minSize to 4 per zone, the cluster autoscaler does not immediately provision an additional three worker nodes per zone (12 worker nodes total). Instead, the scale-up is triggered by resource requests. If you create a workload that requests the resources of 15 worker nodes, the cluster autoscaler scales up the worker pool to meet this request. Now, the minSize means that the cluster autoscaler does not scale down below four worker nodes per zone even if you remove the workload that requests the amount. For more information, see the Kubernetes docs.
"maxSize": 2 Specify the maximum number of worker nodes per zone that the cluster autoscaler can scale up the worker pool to. The value must be equal to or greater than the value that you set for the minSize. "enabled": false Set the value to true for the cluster autoscaler to manage scaling for the worker pool. Set the value to false to stop the cluster autoscaler from scaling the worker pool.
Later, if we want to remove the cluster autoscaler, we must first disable each worker pool in the configmap. -
Save the configuration file.
- Get the cluster autoscaler pod.
oc get pods -n kube-system
-
Review the Events section of the cluster autoscaler pod for a ConfigUpdated event to verify that the configmap is successfully updated. The event message for the configmap is in the following format: minSize:maxSize:PoolName:<SUCCESS|FAILED>:error message.
oc describe pod -n kube-system <cluster_autoscaler_pod>
Example output:
Name: ibm-iks-cluster-autoscaler-857c4d9d54-gwvc6 Namespace: kube-system ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ConfigUpdated 3m ibm-iks-cluster-autoscaler-857c4d9d54-gwvc6 {"1:3:default":"SUCCESS:"}
If you enable a worker pool for autoscaling and then later add a zone to this worker pool, restart the cluster autoscaler pod so that it picks up this change: oc delete pod -n kube-system <cluster_autoscaler_pod>.
Customizing the cluster autoscaler configuration values
Customize the cluster autoscaler settings such as the amount of time it waits before scaling worker nodes up or down.Before beginning:
Customizing the cluster autoscaler add-on configmap
When you modify a configmap parameter other than the worker pool minSize, maxSize, or if you enable or disable a worker pool, the cluster autoscaler pods are restarted.
-
Review the default parameters for the cluster autoscaler configmap.
-
Download the cluster autoscaler add-on configmap and review the parameters.
oc get cm iks-ca-configmap -n kube-system -o yaml > configmap.yaml
-
Open the configmap.yaml file and update the settings that we want to change.
-
Reapply the cluster autoscaler add-on configmap.
oc apply -f configmap.yaml
-
Verify that the pods are restarted successfully.
oc get pods -n kube-system | grep autoscaler
Customizing the cluster autoscaler Helm chart values
The cluster autoscaler Helm chart is deprecated. For the latest version of the cluster autoscaler, install the add-on
-
To change any of the cluster autoscaler configuration values, update the config map or the Helm chart with the new values. Include the --recreate-pods flag so that any existing cluster autoscaler pods are re-created to pick up the custom setting changes. The following example command changes the scan interval to 2m and enables autoscaling for the autoscale worker pool, with a maximum of 5 and minimum of 3 worker nodes per zone.
helm upgrade --set scanInterval=2m --set workerpools[0].default.max=5,workerpools[0].autoscale.min=3,workerpools[0].default.enabled=true ibm-iks-cluster-autoscaler iks-charts/ibm-iks-cluster-autoscaler -i --recreate-pods --namespace kube-system
To reset the Helm chart to the default values:
helm upgrade --reset-values ibm-iks-cluster-autoscaler iks-charts/ibm-iks-cluster-autoscaler --recreate-pods
-
Verify your changes.
- Review the Helm chart values again.
helm get values ibm-iks-cluster-autoscaler -a
- Review the Helm chart values again.
-
Verify that the installation is successful.
- Check that the cluster autoscaler pod is in a Running state.
oc get pods --namespace=kube-system | grep ibm-iks-cluster-autoscaler
Example output:ibm-iks-cluster-autoscaler-8497bfc968-dbn7w 1/1 Running 0 9m
- Check that the cluster autoscaler service is created.
oc get service --namespace=kube-system | grep ibm-iks-cluster-autoscaler
Example output:ibm-iks-cluster-autoscaler ClusterIP 172.21.xxx.xx <none> 8085/TCP 9m
- Check that the cluster autoscaler pod is in a Running state.
-
Verify that the config map is correct by checking that the workerPoolsConfig.json field is updated and that the workerPoolsConfigStatus field shows a SUCCESS message.
oc get cm iks-ca-configmap -n kube-system -o yaml
Example output where the autoscale worker pool is enabled for autoscaling:
apiVersion: v1 data: workerPoolsConfig.json: | [{"name": "autoscale", "minSize": 3, "maxSize": 5, "enabled": true }] kind: ConfigMap metadata: annotations: workerPoolsConfigStatus: '{"1:2:default":"SUCCESS"}' creationTimestamp: "2019-08-23T14:26:54Z" name: iks-ca-configmap namespace: kube-system resourceVersion: "12757878" selfLink: /api/v1/namespaces/kube-system/configmaps/iks-ca-configmap uid: bd661f95-35ef-433d-97e0-5d1ac092eafb
Deploying apps to your autoscaled worker pools
To limit a pod deployment to a specific worker pool that is managed by the cluster autoscaler, use a combination of labels and nodeSelector or nodeAffinity to deploy apps only to the autoscaled worker pools. With nodeAffinity, we have more control over how the scheduling behavior works to match pods to worker nodes. Then, use taints and tolerations so that only these apps can run on your autoscaled worker pools.For more information, see the following Kubernetes docs:
Before beginning:
To limit pods to run on certain autoscaled worker pools:
- Make sure that you labeled and tainted your autoscaled worker pool as described in Preparing the cluster for autoscaling.
-
In the pod spec template, match the nodeSelector or nodeAffinity to the label that you used in your worker pool.
Example of nodeSelector:
... spec: containers: - name: nginx image: nginx imagePullPolicy: IfNotPresent nodeSelector: app: nginx ...Example of nodeAffinity:
spec: containers: - name: nginx image: nginx imagePullPolicy: IfNotPresent affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: use operator: In values: - autoscale -
In the pod spec template, match the toleration to the taint you set on the worker pool.
Example NoExecute toleration:
tolerations: - key: use operator: "Exists" effect: "NoExecute" -
Deploy the pod. Because of the matching label, the pod is scheduled onto a worker node that is in the labeled worker pool. Because of the matching toleration, the pod can run on the tainted worker pool.
oc apply -f pod.yaml
Scaling up worker nodes before the worker pool has insufficient resources
As described in the Understand how the cluster autoscaler works topic and the Kubernetes Cluster Autoscaler FAQs, the cluster autoscaler scales up your worker pools in response to your requested resources of the workload against the available recourses of the worker pool. However, we might want the cluster autoscaler to scale up worker nodes before the worker pool runs out of resources. In this case, the workload does not need to wait as long for worker nodes to be provisioned because the worker pool is already scaled up to meet the resource requests.The cluster autoscaler does not support early scaling (overprovisioning) of worker pools. However, we can configure other Kubernetes resources to work with the cluster autoscaler to achieve early scaling.
- Pause pods
- We can create a deployment that deploys pause containers
in
pods with specific resource requests, and assign the deployment a low pod priority. When these resources are needed by higher priority workloads, the pause pod is preempted and becomes a pending pod. This event triggers the cluster autoscaler
to scale up.
For more information about setting up a pause pod deployment, see the Kubernetes FAQ.
We can use this example overprovisioning configuration file to create the priority class, service account, and deployments.
If you use this method, make sure that you understand how pod priority works and how to set pod priority for the deployments. For example, if the pause pod does not have enough resources for a higher priority pod, the pod is not preempted. The higher priority workload remains in pending, so the cluster autoscaler is triggered to scale up. However, in this case, the scale-up action is not early because the workload that we want to run cannot be scheduled because of insufficient resources. Pause pod must have the matching nodeAffinity or nodeSelector as well as the matching tolerations that you set for the worker pool.
- Horizontal pod autoscaling (HPA)
- Because horizontal pod autoscaling is based on the average CPU usage of the pods, the CPU usage limit that you set is reached before the worker pool runs out of resources. More pods are requested, which then triggers the cluster autoscaler
to scale up the worker pool.
For more information about setting up HPA, see the Kubernetes docs.
Upgrading the cluster autoscaler add-on
This topic applies only to the cluster autoscaler add-on.
If you upgrade the cluster to a version that is not supported by the cluster autoscaler add-on, our apps might experience downtime and the cluster might not scale.
The cluster autoscaler has two types of updates.
- Patch updates: Patch updates are delivered automatically by IBM and do not contain any feature updates or changes in the supported add-on and cluster versions.
- Release updates: Release updates contain new features for the cluster autoscaler or changes in the supported add-on or cluster versions. You must manually apply release updates to the cluster autoscaler add-on.
-
Check the version of the cluster autoscaler add-on that is deployed in the cluster. If an update is available, review the release notes for the latest add-on version.
ibmcloud oc cluster addon ls --cluster <cluster_name>
-
Update the cluster autoscaler add-on.
ibmcloud oc cluster addon update cluster-autoscaler --version <version> --cluster <cluster_name>
Example output
During the update, potentially disruptive changes might occur. For more information, see 'http://ibm.biz/addon-update'. Continue? [y/N]> y Updating add-on cluster-autoscaler to 1.0.1 for cluster ca-zone-bal1.17... OK
-
Verify the add-on is successfully updated and Ready.
ibmcloud oc cluster addon ls --cluster <cluster_name>
Upgrading a cluster autoscaler Helm chart release
This topic applies only to the cluster autoscaler Helm chart.
The cluster autoscaler Helm chart is deprecated. For the latest version of the cluster autoscaler, install the add-on. We can not have both the cluster autoscaler Helm chart and the cluster autoscaler add-on installed at the same time.
We can upgrade the existing cluster autoscaler release to the latest version of the Helm chart. To check your current release version, run helm list -n <namespace> | grep cluster-autoscaler. Compare your version to the latest available release by reviewing the Chart Version in the IBM Cloud Helm Catalog.Prerequisites
Before beginning to upgrade the cluster autoscaler release, complete the following steps.This topic applies only to the cluster autoscaler Helm chart.
- Access the OpenShift cluster.
- To review the changelog of chart versions, download the source code tar file and open the RELEASENOTES.MD file.
Upgrading the cluster autoscaler release version
This topic applies only to the cluster autoscaler Helm chart.
To upgrade the cluster autoscaler release, we can update the Helm chart repo and re-create the cluster autoscaler pods. Use the same version of Helm that you used to install the initial Helm chart and release. For example, if you installed the release with Helm version 2, these upgrade steps might not work if you now have Helm version 3. Instead, see Upgrading a release from Helm version 2 to version 3.
Before beginning, see the Prerequisites.
-
Update the Helm repo to retrieve the latest version of all Helm charts in this repo.
helm repo update
-
Optional: Download the latest Helm chart to the local machine. Then, extract the package and review the release.md file to find the latest release information.
helm pull iks-charts/ibm-iks-cluster-autoscaler
-
Find the name of the cluster autoscaler release that you installed in the cluster.
helm list -n <namespace> | grep cluster-autoscaler
Example output:
myhelmchart 1 Mon Jan 7 14:47:44 2019 DEPLOYED ibm-iks-cluster-autoscaler-1.0.1 kube-system
-
Upgrade the cluster autoscaler release to the latest version.
helm upgrade --force --recreate-pods <helm_chart_name> iks-charts/ibm-iks-cluster-autoscaler
-
Verify that the cluster autoscaler configmap workerPoolsConfig.json section is set to "enabled": true for the worker pools that we want to scale.
oc describe cm iks-ca-configmap -n kube-system
Example output
Name: iks-ca-configmap Namespace: kube-system Labels: <none> Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","data":{"workerPoolsConfig.json":"[\n {\"name\": \"docs\",\"minSize\": 1,\"maxSize\": 3,\"enabled\":true}\n]\n"},"kind":"ConfigMap",... Data ==== workerPoolsConfig.json: ---- [ {"name": "<worker_pool","minSize": 1,"maxSize": 3,"enabled":true} ] Events: <none>
Upgrading a release from Helm v2 to v3
This topic applies only to the cluster autoscaler Helm chart.
When you upgrade a release of the cluster autoscaler, we must use the same version of Helm that you used to install the initial Helm chart and release. The cluster autoscaler Helm chart supports both Helm version 2.15 and 3.0. If you installed the Helm chart and release with Helm v2 and then try to upgrade from a Helm v3 client, we might experience errors. Instead, uninstall the release and reinstall the latest release with Helm v3. For more information, see Migrating from Helm v2 to v3.Additionally, if we have release version 1.0.2 or earlier, we must uninstall that release before you install the latest release.
Before beginning, see the Prerequisites.
- Migrate to or install Helm version 3.
- Get the cluster autoscaler configmap.
oc get cm iks-ca-configmap -n kube-system -o yaml > iks-ca-configmap.yaml
- Remove all worker pools from the configmap by setting the "enabled" value to false.
oc edit cm iks-ca-configmap -n kube-system
- If you applied custom settings to the Helm chart, note your custom settings.
helm get values ibm-iks-cluster-autoscaler -a
- Uninstall your current Helm chart.
helm uninstall ibm-iks-cluster-autoscaler -n <namespace>
- Update the Helm chart repo to get the latest cluster autoscaler Helm chart version.
helm repo update
- Install the latest cluster autoscaler Helm chart. Apply any custom settings that you previously used with the --set flag, such as scanInterval=2m.
helm install ibm-iks-cluster-autoscaler iks-charts/ibm-iks-cluster-autoscaler --namespace kube-system [--set <custom_settings>]
- Apply the cluster autoscaler configmap that you previously retrieved to enable autoscaling for the worker pools.
oc apply -f iks-ca-configmap.yaml
- Get the cluster autoscaler pod.
oc get pods -n kube-system
-
Review the Events section of the cluster autoscaler pod and look for a ConfigUpdated event to verify that the configmap is successfully updated. The event message for the configmap is in the following format: minSize:maxSize:PoolName:<SUCCESS|FAILED>:error message.
oc describe pod -n kube-system <cluster_autoscaler_pod>
Example output:
Name: ibm-iks-cluster-autoscaler-857c4d9d54-gwvc6 Namespace: kube-system ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ConfigUpdated 3m ibm-iks-cluster-autoscaler-857c4d9d54-gwvc6 {"1:3:default":"SUCCESS:"}
Remove the cluster autoscaler
If you do not want to automatically scale your worker pools, we can uninstall the cluster autoscaler Helm chart. After the removal, we must resize or rebalance your worker pools manually.Before beginning: Access the OpenShift cluster.
-
Optional: To refer to your autoscaling settings later, make a backup of our configmap.
oc get cm iks-ca-configmap -o yaml > backup_configmap.yaml
-
In the cluster autoscaler configmap, remove the worker pool by setting the "enabled" value to false.
oc edit cm iks-ca-configmap -n kube-system
Example output
apiVersion: v1 data: workerPoolsConfig.json: | [ {"name": "<worker_pool>","minSize": 1,"maxSize": 2,"enabled":false} ] kind: ConfigMap metadata: creationTimestamp: 2018-11-29T18:43:46Z name: iks-ca-configmap namespace: kube-system resourceVersion: "2227854" selfLink: /api/v1/namespaces/kube-system/configmaps/iks-ca-configmap uid: b45d047b-f406-11e8-b7f0-82ddffc6e65e
If you already deleted the Helm chart and see a message such as iks-ca-configmap not found, redeploy the cluster autoscaler Helm chart to the cluster and try to remove it again.
-
Disable the add-on or uninstall the Helm chart.
Add-on
-
Disable the add-on.
ibmcloud oc addon disable cluster-autoscaler --cluster <cluster_name>
-
Verify the add-on is disabled.
ibmcloud oc cluster addon ls --cluster <cluster_name>
Helm chart
- List the existing Helm charts and note the name of the cluster autoscaler.
helm list --all-namespaces
- Remove the existing Helm chart from the cluster.
helm uninstall ibm-iks-cluster-autoscaler -n <namespace>
-
Cluster autoscaler add-on parameter reference
This table refers to the cluster autoscaler add-on parameters. For Helm chart values, see Cluster autoscaler Helm chart parameter reference
| Parameter | Description | Default value |
|---|---|---|
| expander | Specify how the cluster autoscaler determines which worker pool to scale if we have multiple worker pools. Possible values are:
|
random |
| ignoreDaemonSetsUtilization | When set to true, the cluster autoscaler ignores DaemonSet pods when it calculates resource utilization for scale-down. | false |
| imagePullPolicy | Specify when to pull the Docker image. Possible values are:
|
Always |
| livenessProbeFailureThreshold | Specify the number of times that the kubelet retries a liveness probe after the pod starts and the first liveness probe fails. After the failure threshold is reached, the container is restarted and the pod is marked Unready for a readiness probe, if applicable. | 3 |
| livenessProbePeriodSeconds | Specify the interval in seconds that the kubelet performs a liveness probe. | 10 |
| livenessProbeTimeoutSeconds | Specify the time in seconds after which the liveness probe times out. | 10 |
| maxBulkSoftTaintCount | Set the maximum number of worker nodes that can be tainted or untainted with PreferNoSchedule at the same time. To disable this feature, set to 0. | 0 |
| maxBulkSoftTaintTime | Set the maximum amount of time that worker nodes can be tainted or untainted with PreferNoSchedule at the same time. | 10m |
| maxFailingTime | Set the maximum time in minutes that the cluster autoscaler pod runs without a successfully completed action before the pod is automatically restarted. | 15m |
| maxInactivity | Set the maximum time in minutes that the cluster autoscaler pod runs without any recorded activity before the pod is automatically restarted. | 10m |
| maxNodeProvisionTime | Set the maximum amount of time in minutes that a worker node can take to begin provisioning before the cluster autoscaler cancels the scale-up request. | 120m |
| maxRetryGap | Set the maximum time in seconds to retry after failing to connect to the service API. Use this parameter and the retryAttempts parameter to adjust the retry window for the cluster autoscaler. | 60 |
| resourcesLimitsCPU | Set the maximum amount of worker node CPU that the ibm-iks-cluster-autoscaler pod can consume. | 600m |
| resourcesLimitsMemory | Set the maximum amount of worker node memory that the ibm-iks-cluster-autoscaler pod can consume. | 600Mi |
| resourcesRequestsCPU | Set the minimum amount of worker node CPU that the ibm-iks-cluster-autoscaler pod starts with. | 200m |
| resourcesRequestsMemory | Set the minimum amount of worker node memory that the ibm-iks-cluster-autoscaler pod starts with. | 200Mi |
| retryAttempts | Set the maximum number of attempts to retry after failing to connect to the service API. Use this parameter and the maxRetryGap parameter to adjust the retry window for the cluster autoscaler. | 64 |
| scaleDownDelayAfterAdd and scaleDownDelayAfterDelete | Set the amount of time in minutes that the cluster autoscaler waits to start scaling actions again after scaling up (add) or scaling down (delete). | 10m |
| scaleDownUnneededTime | Set the amount of time in minutes that a worker node must be unnecessary before it can be scaled down. | 10m |
| scaleDownUtilizationThreshold | Set the worker node utilization threshold. If the worker node utilization goes below the threshold, the worker node is considered to be scaled down. Worker node utilization is calculated as the sum of the CPU and memory resources that are requested by all pods that run on the worker node, divided by the worker node resource capacity. | 0.5 |
| scanInterval | Set how often in minutes that the cluster autoscaler scans for workload usage that triggers scaling up or down. | 1m |
| skipNodesWithLocalStorage | When set to true, worker nodes that have pods that are saving data to local storage are not scaled down. | true |
| skipNodesWithSystemPods | When set to true, worker nodes that have kube-system pods are not scaled down. Do not set the value to false because scaling down kube-system pods might have unexpected results. | true |
| workerPoolsConfig.json | The worker pools that we want to autoscale, including their minimum and maximum number of worker nodes per zone:
If you enable a worker pool for autoscaling and then later add a zone to this worker pool, restart the cluster autoscaler pod so that it picks up this change: oc delete pod -n kube-system <cluster_autoscaler_pod>. By default, the default worker pool is not enabled, with a max value of 2 and a min value of 1. |
Disabled |
Cluster autoscaler Helm chart parameter reference
This table refers to the cluster autoscaler Helm chart parameters. For add-on values, see Cluster autoscaler add-on parameter reference
Scroll for moreScroll for more| Parameter | Description | Default value |
|---|---|---|
| api_route parameter | Set the Red Hat OpenShift on IBM Cloud API endpoint for the region that the cluster is in. | No default; uses the targeted region that the cluster is in. |
| resources.limits.cpu parameter | Set the maximum amount of worker node CPU that the ibm-iks-cluster-autoscaler pod can consume. | 300m |
| resources.limits.memory parameter | Set the maximum amount of worker node memory that the ibm-iks-cluster-autoscaler pod can consume. | 300Mi |
| resources.requests.cpu parameter | Set the minimum amount of worker node CPU that the ibm-iks-cluster-autoscaler pod starts with. | 100m |
| resources.requests.memory parameter | Set the minimum amount of worker node memory that the ibm-iks-cluster-autoscaler pod starts with. | 100Mi |
| maxNodeProvisionTime parameter | Set the maximum amount of time in minutes that a worker node can take to begin provisioning before the cluster autoscaler cancels the scale-up request. | 120m |
| scaleDownUnneededTime parameter | Set the amount of time in minutes that a worker node must be unnecessary before it can be scaled down. | 10m |
| scaleDownDelayAfterAdd, scaleDownDelayAfterDelete parameters | Set the amount of time in minutes that the cluster autoscaler waits to start scaling actions again after scaling up (add) or scaling down (delete). | 10m |
| scaleDownUtilizationThreshold parameter | Set the worker node utilization threshold. If the worker node utilization goes below the threshold, the worker node is considered to be scaled down. Worker node utilization is calculated as the sum of the CPU and memory resources that are requested by all pods that run on the worker node, divided by the worker node resource capacity. | 0.5 |
| scanInterval parameter | Set how often in minutes that the cluster autoscaler scans for workload usage that triggers scaling up or down. | 1m |
| expander parameter | Specify how the cluster autoscaler determines which worker pool to scale if we have multiple worker pools. Possible values are:
|
random |
| ignoreDaemonSetsUtilization | When set to true, the cluster autoscaler ignores DaemonSet pods when it calculates resource utilization for scale-down. | false |
| maxBulkSoftTaintCount | Set the maximum number of worker nodes that can be tainted or untainted with PreferNoSchedule at the same time. To disable this feature, set to 0. | 0 |
| maxBulkSoftTaintTime | Set the maximum amount of time that worker nodes can be tainted or untainted with PreferNoSchedule at the same time. | 10m |
| skipNodes.withLocalStorage parameter | When set to true, worker nodes that have pods that are saving data to local storage are not scaled down. | true |
| skipNodes.withSystemPods parameter | When set to true, worker nodes that have kube-system pods are not scaled down. Do not set the value to false because scaling down kube-system pods might have unexpected results. | true |
| image.repository parameter | Specify the cluster autoscaler Docker image to use. To get a list of cluster autoscaler images, target the global IBM Cloud Container Registry API by running ibmcloud cr region-set global. Then, list the available cluster autoscaler images by running ibmcloud cr images --include-ibm | grep cluster-autoscaler. To review the supported Kubernetes versions for each cluster autoscaler image version, download the source code tar file and open the RELEASENOTES.MD file. | icr.io/iks-charts/ibm-iks-cluster-autoscaler |
| image.pullPolicy parameter | Specify when to pull the Docker image. Possible values are:
|
IfNotPreset |
| livenessProbe.periodSeconds | Specify the interval in seconds that the kubelet performs a liveness probe. | 10 |
| livenessProbe.failureThreshold | Specify the number of times that the kubelet retries a liveness probe after the pod starts and the first liveness probe fails. After the failure threshold is reached, the container is restarted and the pod is marked Unready for a readiness probe, if applicable. | 3 |
| livenessProbe.successThreshold | Specify the minimum, consecutive number of times that the liveness probe must be successful after a previous failure before the pod is marked Ready. If you set a liveness probe, we must set this value to at least 1. | 1 |
| livenessProbe.timeoutSeconds | Specify the time in seconds after which the liveness probe times out. | 10 |
| max-inactivity | Set the maximum time in minutes that the cluster autoscaler pod runs without any recorded activity before the pod is automatically restarted. | 10m |
| max-failing-time | Set the maximum time in minutes that the cluster autoscaler pod runs without a successfully completed action before the pod is automatically restarted. | 15m |
| customImageVersion | To override the default installation version, specify the version of the cluster autoscaler Helm chart that we want to install. | N/A |
| maxRetryGap | Set the maximum time in seconds to retry after failing to connect to the service API. Use this parameter and the retryAttempts parameter to adjust the retry window for the cluster autoscaler. | 60 |
| retryAttempts | Set the maximum number of attempts to retry after failing to connect to the service API. Use this parameter and the maxRetryGap parameter to adjust the retry window for the cluster autoscaler. | 32 |
| workerpools parameter | The worker pools that we want to autoscale, including their minimum and maximum number of worker nodes per zone. These settings are mirrored in the cluster autoscaler config map. To set the worker pool, format the
option as follows: --set workerpools[0].<pool_name>.max=<number_of_workers>,workerpools[0].<pool_name>.min=<number_of_workers>,workerpools[0].<pool_name>.enabled=(true|false) Understanding the --set workerpools options:
If you enable a worker pool for autoscaling and then later add a zone to this worker pool, restart the cluster autoscaler pod so that it picks up this change: oc delete pod -n kube-system <cluster_autoscaler_pod>. By default, the default worker pool is not enabled, with a max value of 2 and a min value of 1. |
Disabled |