Skip to main content

Rotate User AWS Credentials

Use this runbook if a user’s AWS credentials have been exposed. In this case, ‘AWS credentials’ refers to credentials created by terraform, for resources in the user’s namespace.

We will make terraform destroy the compromised credentials, and then recreate them. For this example, we will use the case where an AWS credential for an ECR in a namespace was exposed.

Please amend the code snippets below to match your actual circumstance.

1. Prepare terraform to manipulate the user’s AWS resources

You need a working copy of the environments github repo.

cd cloud-platform-environments
git checkout main
git pull

Launch the tools-shell. This will have all the binary with correct versions needs for performing terraform operations

make tools-shell

Set pingdom environment variables(Optional)

If the changes involve applying “pingdom_check”, set the environment variables for pingdom. The values are stored as secrets in manager cluster - concourse-main namespace.

export PINGDOM_USER="XXXXXXXXXXX"
export PINGDOM_PASSWORD="XXXXXXXXXXXX"
export PINGDOM_API_KEY="XXXXXXXXXXXXX"

Target the live cluster

aws eks --region eu-west-2 update-kubeconfig --name live
# TF_VAR_cluster_name is referencing VPC name
export TF_VAR_cluster_name="live" 
export TF_VAR_cluster_state_bucket=cloud-platform-terraform-state
export TF_VAR_cluster_state_key="cloud-platform/live/terraform.tfstate"
#needed by tf k8s provider
export KUBE_CONFIG_PATH=${HOME}/.kube/config

Set the namespace name

I’m using offender-events-dev for this example. Use whatever is relevant for your case.

export NAMESPACE=offender-events-dev

cd namespaces/live.cloud-platform.service.justice.gov.uk/${NAMESPACE}/resources

Terraform Init

terraform init \
  -backend-config="bucket=cloud-platform-terraform-state" \
  -backend-config="key=cloud-platform-environments/live.cloud-platform.service.justice.gov.uk/${NAMESPACE}/terraform.tfstate" \
  -backend-config="region=eu-west-1" \
  -backend-config="dynamodb_table=cloud-platform-environments-terraform-lock"

Note: Bucket key above is referencing to “live”, as state is stored in “live.cloud-platform.service.justice.gov.uk” for namespaces in “live” cluster.

Terraform Plan/Apply

terraform plan

The terraform plan should output No changes. Infrastructure is up-to-date.

2. Identify the compromised terraform object

$ terraform show | less

Look for the compromised access key. In this case, the access key was AKIAXXXXXXXXXXXXXXXX, and it occurs in two places in the output.

...
kubernetes_secret.ecr-repo-my-repo:
  id = my-namespace/ecr-repo-my-repo
  data.% = 3
  data.access_key_id = AKIAXXXXXXXXXXXXXXXX
  data.repo_url = ...
  data.secret_access_key = ...
...
module.ecr-repo-my-repo.aws_iam_access_key.key:
  id = AKIAXXXXXXXXXXXXXXXX
  secret = ...
  ses_smtp_password = ...
...

The first occurence is when terraform stores the credentials in a kubernetes secret.

The second occurence is the one we want, where terraform creates the credentials (the aws_iam_access_key.key resource, for the ECR repo).

3. Destroy the compromised key

$ terraform destroy --target=module.ecr-repo-my-repo.aws_iam_access_key.key

If this looks like it’s going to do the right thing, enter ‘yes’ to confirm.

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  - module.ecr-repo-my-repo.aws_iam_access_key.key


Plan: 0 to add, 0 to change, 1 to destroy.

4. Let terraform create a new key

$ terraform plan

This should report that it will create a new aws_iam_access_key.key resource and modify the corresponding kubernetes_secret resource.

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create
  ~ update in-place

Terraform will perform the following actions:

  ~ kubernetes_secret.ecr-repo-my-repo
      data.%:                 "" => <computed>
      data.access_key_id:     "AKIAXXXXXXXXXXXXXXXX" => ""
      data.repo_url:          "..." => ""
      data.secret_access_key: "..." => ""

  + module.ecr-repo-my-repo.aws_iam_access_key.key
      id:                     <computed>
      encrypted_secret:       <computed>
      key_fingerprint:        <computed>
      secret:                 <computed>
      ses_smtp_password:      <computed>
      status:                 <computed>
      user:                   "ecr-user-0000000000000000"


Plan: 1 to add, 1 to change, 0 to destroy.

If all is well:

$ terraform apply

If this looks like it’s going to do the right thing, enter ‘yes’ to confirm.

At this point, a new set of AWS credentials should have been created for the existing IAM user, and the kubernetes secret should contain the new access key and secret.

Rotate RDS Credentials

If a user’s RDS credentials(database_username and database_password) have been exposed. Follow “Rotate User AWS Credentials” guidance above until step 2, in step 3 instead of “Destroy the compromised key”, we should taint the password.

$  terraform taint module.rds.random_password.password

Running taint will show the below message.

Resource instance module.rds.random_password.password has been marked as tainted.

Let terraform create a new password

$ terraform plan

This should report that it will create a new rds.random_password.password resource and modify the corresponding kubernetes_secret resource.

Terraform will perform the following actions:

  # kubernetes_secret.rds will be updated in-place
  ~ resource "kubernetes_secret" "rds" {
      ~ data      = (sensitive value)
        id        = "rds/postgres"
    }

  # module.rds.aws_db_instance.rds will be updated in-place
  ~ resource "aws_db_instance" "rds" {
        id                                    = "cloud-platform-xxxxxx"
        name                                  = "dbxxxxxxx"
      ~ password                              = (sensitive value)
        tags                                  = {
            "application"            = "app"
            "business-unit"          = "business-unit"
            "environment-name"       = "production"
            "infrastructure-support" = "team@digital.justice.gov.uk"
            "is-production"          = "true"
            "namespace"              = "namespace-name"
            "owner"                  = "namespace-woner"
        }

      + timeouts {
          + create = "2h"
          + delete = "2h"
          + update = "2h"
        }
    }

  # module.rds.random_password.password is tainted, so must be replaced
-/+ resource "random_password" "password" {
      ~ id          = "none" -> (known after apply)
      ~ result      = (sensitive value)
        # (9 unchanged attributes hidden)
    }

Plan: 1 to add, 2 to change, 1 to destroy.

If all is well:

$ terraform apply

If this looks like it’s going to do the right thing, enter ‘yes’ to confirm.

At this point, a new RDS password should have been created, and the kubernetes secret should contain the db password.

Note: It is possible that applications might experience downtime if, for example, a pod which was launched with the old password drops a DB connection and tries to open a new one (which will fail, because the password is no longer valid). To make pods pick up the new password, perform a manual rollout on every relevant deployment: bash kubectl rollout restart "deployment/{deployment}" -namespace="{namespace}" This will rotate all pods according to the rollout strategy used in deployments in the namespace, which will pick up the new DB password from the kubernetes secret.

This page was last reviewed on 17 June 2022. It needs to be reviewed again on 17 September 2022 by the page owner #cloud-platform .
This page was set to be reviewed before 17 September 2022 by the page owner #cloud-platform. This might mean the content is out of date.