Skip to main content

Recycle-all-nodes

When a launch template is updated, this will cause all of the nodes to recycle. Reasons to update the launch configuration are most likely to revolve around editing the User data script, which is ran when a node is booted. Reasons to edit User data include:

  • Changes to docker auth credentials
  • Changes to eks-bootstrap-env.sh
  • Changes to kubelet environment variables

Recycling process

AWS will handle the process of killing the nodes with the old launch configurations and launching new nodes with the latest launch configuration. AWS will initially spin up some extra nodes (for around a 60 node cluster it will spin up 8 extra) to provide spaces for pods as old nodes are cordoned, drained and deleted.

  • watch kubectl get nodes --sort-by=.metadata.creationTimestamp

The above command will output all of the nodes like this:

NAME                                           STATUS   ROLES    AGE     VERSION
ip-172-20-124-118.eu-west-2.compute.internal   Ready,SchedulingDisabled      <none>   47h     v1.22.15-eks-fb459a0
ip-172-20-101-81.eu-west-2.compute.internal    Ready,SchedulingDisabled      <none>   47h     v1.22.15-eks-fb459a0
ip-172-20-119-182.eu-west-2.compute.internal   Ready    <none>   47h     v1.22.15-eks-fb459a0
ip-172-20-106-20.eu-west-2.compute.internal    Ready    <none>   47h     v1.22.15-eks-fb459a0
ip-172-20-127-1.eu-west-2.compute.internal     Ready    <none>   47h     v1.22.15-eks-fb459a0

Where nodes have the Status “Ready,SchedulingDisabled” this indicates the nodes which have the old update templates, these are the ones that are cordoned off, whereas those which are “Ready” are the new nodes with the new update template.

When all nodes have been recycled they will all have a status of “Ready”.

This process can take 3 to 4 hours on a cluster of ~60 nodes, depending on how quickly you resolve the gotchas below.

Gotchas

When AWS is draining the old nodes, even after choosing the “force update” option, it will still respect the “Pod Disruption Budget” and will not evict pods if it will break the PDB.

If a “Pod Disruption Budget” is poorly configured, kubernetes can’t evict a specific pod. Without manual intervention, this will indefinitely stall the update and the nodes will not continue to recycle. This will eventually cause the update to stop, wait and then exit, leaving the remaining nodes with the old update template and others with the new update template.

AWS EKS in some circumstances has even reverted the update and started to drain the new nodes and replace them with the old update template again. So it’s important to monitor how the update is going and act when nodes get stuck.

To resolve the issue:

  1. Copy below script and save it as delete-pods-in-namespace.sh.

    #!/bin/bash

delete_pods() {
  NAMESPACE=$(echo "$1" | sed -E 's/\/api\/v1\/namespaces\/(.*)\/pods\/.*/\1/')
  POD=$(echo "$1" | sed -E 's/.*\/pods\/(.*)\/eviction\?timeout=.*/\1/')

  echo $NAMESPACE

  echo $POD

  kubectl delete pod -n $NAMESPACE $POD
}

export -f delete_pods

TIME_NOW_EPOCH=$(date +%s)

START_TIME=$(($TIME_NOW_EPOCH - 180))

CLUSTER_LOG_GROUP=$1

QUERY_ID=$(aws logs start-query \
  --start-time $START_TIME \
  --end-time $TIME_NOW_EPOCH \
  --log-group-name $CLUSTER_LOG_GROUP \
  --query-string 'fields @timestamp, @message | filter @logStream like "kube-apiserver-audit" | filter ispresent(requestURI) | filter objectRef.subresource = "eviction" | filter responseObject.status = "Failure" | display @logStream, requestURI, responseObject.message | stats count(*) as retry by requestURI, requestObject.message' \
  | jq -r '.queryId' )

sleep 2

RESULTS=$(aws logs get-query-results --query-id $QUERY_ID)

echo -n $RESULTS | jq '.results[]' | grep '/api/v1' | awk '{ print $2 }' | xargs -I {} bash -c 'delete_pods {}'

exit 0

  2. Run chmod +x delete-pods-in-namespace.sh.

  3. Evict the offending pod by running the below script:

    watch -n 300 ./delete-pods-in-namespace.sh '/aws/eks/<cluster-name>/cluster' > deleted_pods.log

    The <cluster-name> is the short name of the cluster e.g. cp-2901-1531

  4. Run tail -f deleted_pods.log in another terminal.

If you want to find the offending pod manually, follow these steps:

  1. Use Cloudwatch Logs > Logs Insights to identify the offending pod
  2. Select the relevant cluster from the log group drop down

  3. Paste the following query (this will identify which pods have failed to be deleted and how many times deletion has been retried) into the box:

    fields @timestamp, @message | filter @logStream like "kube-apiserver-audit" | filter ispresent(requestURI) | filter objectRef.subresource = "eviction" | filter responseObject.status = "Failure" | display @logStream, requestURI, responseObject.message | stats count(*) as retry by requestURI, requestObject.message

  4. If there are results they will have a pattern like this: /api/v1/namespaces/$NAMESPACE/pods/$POD_NAME-$POD_ID/eviction?timeout=19s

  5. You may also go to the CloudWatch Dashboard directly to identify the offending pod.

  6. You can then run the following command to manually delete the pod kubectl delete pod -n $NAMESPACE $POD_NAME-$POD_ID

Nodes should continue to recycle and after a few moments there should be one less node with the status “Ready,SchedulingDisabled”

This page was last reviewed on 10 April 2024. It needs to be reviewed again on 10 October 2024 by the page owner #cloud-platform .
This page was set to be reviewed before 10 October 2024 by the page owner #cloud-platform. This might mean the content is out of date.