Skip to main content

Upgrade EKS cluster

This page covers the steps required to follow when carrying out an EKS version upgrade.

Pre-requisites

Before you begin, there are a few pre-requisites:

  • Your GPG key must be added to the infrastructure repo so that you can run git-crypt unlock.

  • You have the AWS CLI profile moj-cp with suitable credentials.

  • You have terraform, docker and the Concourse fly CLI installed.

  • Review the changelog of the Kubernetes release and the EKS release you are planning to upgrade to.

  • Review the official EKS upgrading a cluster document for any extra steps that are a part of a specific EKS release.

  • Run kubent against cluster to find deprecated APIs.

Creating Cluster Upgrade GitHub Issues

In order to help maintain a clear process for steps to follow in carrying out a cluster upgrade, we have an issue template which is used by this script to create individual issues for each step of the upgrade process. As you work through these issues, its helpful to review and update the template where necessary to ensure it remains relevant for future upgrades.

steps:

  1. Go to the Kubernetes Upgrade Issues Action in the cloud-platform repo

  2. Click on “Run workflow”

  3. Enter the required inputs:

  • Target Kubernetes version (e.g. 1.34)
  1. Click “Run workflow” to create the issues.

Upgrade Steps

Compatibility Check

The following areas need to be looked into to determine if there’s any additional preparation work to do:

  • Kubernetes API Deprecations/Removals
  • EKS addons
  • Cluster Components

Deprecated/Removed Kubernetes APIs

For Kubernetes API deprecations or removals you can use kubent and pluto to scan the cluster and find if there are any resources impacted in upcoming releases.

From the AWS console you can also see “Upgrade Insights” which has a break down of API deprecations and removals. You can drill down into specific versions and see the resources effected. In particular, the User Agent field here can be useful for tracking down API calling services.

Sometimes the User Agent ID isn’t clear enough to immediately identify where the resource is effected, if this is the case it’s worth cross checking components or helm chart versions. Additionally, you can head over to CloudWatch > Log groups > /aws/eks/[cluster-name] and view the kube-apiserver-audit logs, and filter by the userAgent field, which can help determine the source of the API calls.

Users will need to be notified if their resources are affected by API deprecations or removals.

EKS addons

For guidance on EKS addon upgrades, refer to this runbook.

Cluster Components

The generated GitHub issues cover component upgrade steps.

Preparing for upgrade

Communication is an important part of the upgrade procedure, make sure to update #ask-cloud-platform and #cloud-platform-update when commencing the upgrade. Create a thread in #cloud-platform to keep the team updated on the current status of the upgrade.

Pause the following pipelines:

  • bootstrap
  • infrastructure-live
  • infrastructure-live-2
  • infrastructure-manager

Notify the Cloud Platform team in #cloud-platform that the upgrade is commencing and that the pipelines have been paused.

Update cluster.tf in cloud-platform-infrastructure with the version of Kubernetes you are upgrading to.

Run a tf plan against the cluster your upgrading to check to see if everything is expected, the only changes should be to resources relating to the the version upgrade.

IMPORTANT: Do NOT run tf apply as this will, at best, certainly time out and fail, and leaves us in a position where we have no control over node group upgrades. EKS Upgrades steps are manually carried out through the AWS Console.

Monitoring the upgrade

Before you start the upgrade it is useful to have a few monitoring resources up and running so you can catch any issues quickly.

K9s is a useful tool to have open in a few terminal windows, the following views are helpful:

  • nodes - see nodes recycling and coming up with new version
  • events - check to see if there are any errors
  • pods - you can use vim style searching to see pods in Error, Pending, ContainerCreating states.

You may refer to Monitoring with K9s section for more details. This OpenSearch dashboard is used to monitor the IP assignment for pods when they are rescheduled. If there is a spike in errors then there could be a starvation of IP addresses while scheduling pods.

Starting the upgrade

As with preparing for the upgrade communication is really important, keep the thread in #cloud-platform up to date as much as possible.

Increasing coredns pods

Consider scaling up the coredns replicas by a couple of pods to help ensure consistent DNS resolution during the upgrade process.

kubectl scale deployment coredns --replicas=x -n kube-system

Scale down Descheduler deployment

kubectl scale deployment descheduler --replicas=0 -n kube-system

NOTE: These are temporary measures, double check the deployments for the current replicasets, as you will need this for when you scale back after the completion of the upgrade.

Upgrading the control pane

Log in to the AWS console and select the EKS cluster we’re going to upgrade.

In the top right corner there should be a button called Upgrade now, click that and ensure the correct Kubernetes version is selected then press Update.

Control pane updates usually take ~10 minutes to complete.

Upgrading cluster node groups

As with control plane, for an EKS upgrade we handover the node group upgrade to AWS, using their graceful rolling update strategy.

NOTE:This does not apply for node group recycles, for which we use the cordon-and-drain Concourse pipeline. See this runbook for guidance on how to recycle node groups.

Setup eviction monitoring pipeline

Since we are using the rolling update strategy for node groups, we need to monitor for any stuck pod evictions that may occur during the upgrade process (for example due to bad PDB configurations).

We have a Concourse pipeline to monitor for pod evictions, and cleanly drain pods from nodes that are being upgraded. If you fail to run this pipeline during a node group upgrade, the upgrade will fail out and rollback. This is bad. See the associated Concourse README for instructions on setting up the pipeline.

Upgrading the cluster node groups

From the cluster control panel select Compute tab.

Select Upgrade now next to the target node group (ie monitoring, thanos, containment, default depending on the cluster).

For update strategy select “Rolling update”

Click Update

NOTE: We generally start with monitoring node group first, and then each additional node group, leaving default node group to last.

Repeat the above steps for each node group in the cluster.

Once the upgrade has completed notify the Slack channels.

Finishing the upgrade

Create a new pull request in the cloud-platform-infrastructure repo with the updated version strings.

Unpause the following pipelines in this order and check to make sure no changes are present:

  1. infrastructure-live-2
  2. infrastructure-manager
  3. infrastructure-live

If there are no changes for terraform shown in each pipeline then the PR can be merged in.

Unpause the bootstrap pipeline. If there is any output in the eviction monitoring pipeline you’re interested in reviewing, copy this output before unpausing bootstrap, as the next cycle will delete the eviction monitoring pipeline.

Scale down the coredns pods.

kubectl scale deployment coredns --replicas=x -n kube-system

Scale up Descheduler deployment

kubectl scale deployment descheduler --replicas=1 -n kube-system

Finishing touches

The kubectl version in the cloud-platform-cli and cloud-platform-tools-image needs updating to match the current Kubernetes version.

Documentation used as part of the upgrade should be reviewed and refined if needed.

This page was last reviewed on 23 July 2026. It needs to be reviewed again on 23 January 2027 by the page owner #cloud-platform-notify .