Skip to main content

Rotate User AWS Credentials

Most of the terraform modules supported by Cloud Platform uses IRSA and hence no long term credentials are created. If any other resources refer to credentials created by terraform sucn as IAM user, Cognito users, etc, then use this runbook if those AWS credentials have been exposed.

In this case, ‘AWS credentials’ refers to credentials created by terraform, for resources in the user’s namespace. Use this runbook if a user’s AWS credentials have been exposed.

We will make terraform destroy the compromised credentials, and then recreate them. For this example, we will use the case where an IAM user access key and secret in a namespace was exposed.

Please amend the code snippets below to match your actual circumstance.

1. Prepare terraform to manipulate the user’s AWS resources

You need a working copy of the environments github repo.

cd cloud-platform-environments
git checkout main
git pull

Launch the tools-shell. This will have all the binary with correct versions needs for performing terraform operations

make tools-shell

Set pingdom environment variables(Optional)

If the changes involve applying “pingdom_check”, set the environment variables for pingdom. The values are stored as secrets in manager cluster - concourse-main namespace.

Target the live cluster

aws eks --region eu-west-2 update-kubeconfig --name live
export TF_VAR_vpc_name="live-1"
export TF_VAR_eks_cluster_name="live"
export TF_VAR_github_owner="ministryofjustice"
export TF_VAR_kubernetes_cluster=""
export KUBE_CONFIG_PATH=${HOME}/.kube/config
export KUBECONFIG=${HOME}/.kube/config

Set the namespace name

I’m using offender-events-dev for this example. Use whatever is relevant for your case.

export NAMESPACE=offender-events-dev

cd namespaces/${NAMESPACE}/resources

Terraform Init

terraform init \
  -backend-config="bucket=cloud-platform-terraform-state" \
  -backend-config="key=cloud-platform-environments/${NAMESPACE}/terraform.tfstate" \
  -backend-config="region=eu-west-1" \

Note: Bucket key above is referencing to “live-1”, as state is stored in “” for namespaces in “live” cluster.

Terraform Plan/Apply

terraform plan

The terraform plan should output No changes. Infrastructure is up-to-date.

2. Identify the compromised terraform object

$ terraform show | less

Look for the compromised access key. In this case, the access key was AKIAXXXXXXXXXXXXXXXX, and it occurs in two places in the output.

  id = my-namespace/iam-credentials
  data.access_key_id = AKIAXXXXXXXXXXXXXXXX
  data.secret_access_key = ...
  secret = ...

The first occurence is when terraform stores the credentials in a kubernetes secret.

The second occurence is the one we want, where terraform creates the credentials (the aws_iam_access_key.key resource, for the IAM User).

3. Destroy the compromised key

$ terraform destroy --target=aws_iam_access_key.key

If this looks like it’s going to do the right thing, enter ‘yes’ to confirm.

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  - aws_iam_access_key.key

Plan: 0 to add, 0 to change, 1 to destroy.

4. Let terraform create a new key

$ terraform plan

This should report that it will create a new aws_iam_access_key.key resource and modify the corresponding kubernetes_secret resource.

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create
  ~ update in-place

Terraform will perform the following actions:

  ~ kubernetes_secret.iam-credentials
      data.access_key_id:     "AKIAXXXXXXXXXXXXXXXX" => ""
      data.secret_access_key: "..." => ""

  + aws_iam_access_key.key
      id:                     <computed>
      encrypted_secret:       <computed>
      key_fingerprint:        <computed>
      secret:                 <computed>
      ses_smtp_password:      <computed>
      status:                 <computed>
      user:                   "iam-user-0000000000000000"

Plan: 1 to add, 1 to change, 0 to destroy.

If all is well:

$ terraform apply

If this looks like it’s going to do the right thing, enter ‘yes’ to confirm.

At this point, a new set of AWS credentials should have been created for the existing IAM user, and the kubernetes secret should contain the new access key and secret.

Note: It is possible that applications might experience downtime if, for example, a pod which was launched with the old password drops the connection to AWS and tries to open a new one (which will fail, because the password is no longer valid). To make pods pick up the new password, perform a manual rollout on every relevant deployment: bash kubectl rollout restart "deployment/{deployment}" -namespace="{namespace}" This will rotate all pods according to the rollout strategy used in deployments in the namespace, which will pick up the new iam keys from the kubernetes secret.

This page was last reviewed on 2 February 2024. It needs to be reviewed again on 2 August 2024 by the page owner #cloud-platform .
This page was set to be reviewed before 2 August 2024 by the page owner #cloud-platform. This might mean the content is out of date.