What is a Cluster Autoscaler?

The cluster autoscaler on AWS scales worker nodes within any specified autoscaling group. In this article, we will learn how to set up a cluster autoscaler to spin up new nodes whenever any pod goes into pending state. For more details read here.

This article assumes that you have set up your Kubernetes cluster running in AWS using Kops.

Setup Cluster Autoscaler

First of all edit your instance group to include the below labels and permissions:

$ kubectl edit ig <IG NAME> --state s3://<YOUR S3 BUCKET NAME>

spec:
  cloudLabels:
    k8s.io/cluster-autoscaler/enabled: ""
    k8s.io/cluster-autoscaler/node-template/label: ""
    maxSize: 3
    minSize: 1

Now edit the cluster to give appropriate IAM Roles for autoscaling:

$ kubectl edit cluster --state s3://<YOUR S3 BUCKET NAME>


spec:
  additionalPolicies:
    node: |
      [
        {
          "Effect": "Allow",
          "Action": [
            "autoscaling:DescribeAutoScalingGroups",
            "autoscaling:DescribeAutoScalingInstances",
            "autoscaling:SetDesiredCapacity",
            "autoscaling:DescribeLaunchConfigurations",
            "autoscaling:DescribeTags",
            "autoscaling:TerminateInstanceInAutoScalingGroup"
          ],
          "Resource": ["*"]
        }
      ]

Update the cluster after doing the above changes:

$ kubectl update cluster --state s3://<YOUR S3 BUCKET NAME> --yes

After adding the above configurations in your Cluster spec file. Let’s install the Cluster autoscaler. But, be sure to ensure that metrics-server is configured in your cluster. Because the metrics server is responsible for checking the health of the pods after every 1 minute. And this is used by Horizontal Pod Autoscaler to scale up or down the number of pods.

Run the below commands to configure Cluster Autoscaler and keep in mind to change the necessary field as per your cluster setup:

CLOUD_PROVIDER=aws
IMAGE=k8s.gcr.io/cluster-autoscaler:v1.12.1 # Change the version of cluster autoscaler based on your Kubernetes Version
MIN_NODES=1
MAX_NODES=5
AWS_REGION=ap-south-1
# For AWS GROUP_NAME should be the name of ASG as seen on AWS console
GROUP_NAME="nodes.k8s.example.com"
SSL_CERT_PATH="/etc/ssl/certs/ca-certificates.crt" # (/etc/ssl/certs for gce, /etc/ssl/certs/ca-bundle.crt for RHEL7.X)

addon=cluster-autoscaler.yml
wget -O ${addon} https://raw.githubusercontent.com/kubernetes/kops/master/addons/cluster-autoscaler/v1.8.0.yaml

sed -i -e "s@{{CLOUD_PROVIDER}}@${CLOUD_PROVIDER}@g" "${addon}"
sed -i -e "s@{{IMAGE}}@${IMAGE}@g" "${addon}"
sed -i -e "s@{{MIN_NODES}}@${MIN_NODES}@g" "${addon}"
sed -i -e "s@{{MAX_NODES}}@${MAX_NODES}@g" "${addon}"
sed -i -e "s@{{GROUP_NAME}}@${GROUP_NAME}@g" "${addon}"
sed -i -e "s@{{AWS_REGION}}@${AWS_REGION}@g" "${addon}"
sed -i -e "s@{{SSL_CERT_PATH}}@${SSL_CERT_PATH}@g" "${addon}"

kubectl apply -f ${addon}

Now, Cluster Autoscaler pod will be up and running. It may happen that until it comes up it may go in CrashLoopBackOff state for some time as it might be doing some probes in the background.

Once cluster autoscaler pod is up and in running state, test this setup by running a pod requesting CPU more than what is available on the currently running nodes. The pod will first go in pending state and cluster autoscale will spin up a new node and that pod will get scheduled on the newly provisioned node.
To test scale down delete that pod and wait for 10-15 mins for Cluster autoscaler to deprovision one of the node which is underutilized.

Feel free to ask any questions in the comments section if you face any issues while setting this up. Hope this article is helpful.

4 thoughts to “Configure Cluster Autoscaler in Kubernetes

Leave a comment

Your email address will not be published. Required fields are marked *