Skip to main content

Rotate User AWS Credentials

Use this runbook if a user’s AWS credentials have been exposed. In this case, ‘AWS credentials’ refers to credentials created by terraform, for resources in the user’s namespace.

We will make terraform destroy the compromised credentials, and then recreate them. For this example, we will use the case where an AWS credential for an ECR in a namespace was exposed.

Please amend the code snippets below to match your actual circumstance.

1. Prepare terraform to manipulate the user’s AWS resources

You need a working copy of the environments github repo.

cd cloud-platform-environments
git checkout main
git pull

Launch the tools-shell. This will have all the binary with correct versions needs for performing terraform operations

make tools-shell

Set pingdom environment variables(Optional)

If the changes involve applying “pingdom_check”, set the environment variables for pingdom. The values are stored as secrets in manager cluster - concourse-main namespace.


Target the live-1 cluster

kops export kubecfg
export TF_VAR_cluster_name="live-1"
export TF_VAR_cluster_state_bucket=cloud-platform-terraform-state
export TF_VAR_cluster_state_key="cloud-platform/live-1/terraform.tfstate"
#needed by tf k8s provider
export KUBE_CONFIG_PATH=${HOME}/.kube/config

Set the namespace name

I’m using offender-events-dev for this example. Use whatever is relevant for your case.

export NAMESPACE=offender-events-dev

cd namespaces/${NAMESPACE}/resources

Terraform Init

terraform init \
  -backend-config="bucket=cloud-platform-terraform-state" \
  -backend-config="key=cloud-platform-environments/${NAMESPACE}/terraform.tfstate" \
  -backend-config="region=eu-west-1" \

Terraform Plan/Apply

terraform plan

The terraform plan should output No changes. Infrastructure is up-to-date.

2. Identify the compromised terraform object

$ terraform show | less

Look for the compromised access key. In this case, the access key was AKIAXXXXXXXXXXXXXXXX, and it occurs in two places in the output.

  id = my-namespace/ecr-repo-my-repo
  data.% = 3
  data.access_key_id = AKIAXXXXXXXXXXXXXXXX
  data.repo_url = ...
  data.secret_access_key = ...
  secret = ...
  ses_smtp_password = ...

The first occurence is when terraform stores the credentials in a kubernetes secret.

The second occurence is the one we want, where terraform creates the credentials (the aws_iam_access_key.key resource, for the ECR repo).

3. Destroy the compromised key

$ terraform destroy --target=module.ecr-repo-my-repo.aws_iam_access_key.key

If this looks like it’s going to do the right thing, enter ‘yes’ to confirm.

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  - module.ecr-repo-my-repo.aws_iam_access_key.key

Plan: 0 to add, 0 to change, 1 to destroy.

4. Let terraform create a new key

$ terraform plan

This should report that it will create a new aws_iam_access_key.key resource and modify the corresponding kubernetes_secret resource.

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create
  ~ update in-place

Terraform will perform the following actions:

  ~ kubernetes_secret.ecr-repo-my-repo
      data.%:                 "" => <computed>
      data.access_key_id:     "AKIAXXXXXXXXXXXXXXXX" => ""
      data.repo_url:          "..." => ""
      data.secret_access_key: "..." => ""

  + module.ecr-repo-my-repo.aws_iam_access_key.key
      id:                     <computed>
      encrypted_secret:       <computed>
      key_fingerprint:        <computed>
      secret:                 <computed>
      ses_smtp_password:      <computed>
      status:                 <computed>
      user:                   "ecr-user-0000000000000000"

Plan: 1 to add, 1 to change, 0 to destroy.

If all is well:

$ terraform apply

If this looks like it’s going to do the right thing, enter ‘yes’ to confirm.

At this point, a new set of AWS credentials should have been created for the existing IAM user, and the kubernetes secret should contain the new access key and secret.

This page was last reviewed on 19 April 2021. It needs to be reviewed again on 19 July 2021 by the page owner #cloud-platform .
This page was set to be reviewed before 19 July 2021 by the page owner #cloud-platform. This might mean the content is out of date.