Rotate User AWS Credentials
Use this runbook if a user’s AWS credentials have been exposed. In this case, ‘AWS credentials’ refers to credentials created by terraform, for resources in the user’s namespace.
We will make terraform destroy the compromised credentials, and then recreate them. For this example, we will use the case where an AWS credential for an ECR in a namespace was exposed.
Please amend the code snippets below to match your actual circumstance.
1. Prepare terraform to manipulate the user’s AWS resources
You need a working copy of the environments github repo.
cd cloud-platform-environments
git checkout main
git pull
Launch the tools-shell. This will have all the binary with correct versions needs for performing terraform operations
make tools-shell
Set pingdom environment variables(Optional)
If the changes involve applying “pingdom_check”, set the environment variables for pingdom.
The values are stored as secrets in manager cluster - concourse-main namespace.
export PINGDOM_USER="XXXXXXXXXXX"
export PINGDOM_PASSWORD="XXXXXXXXXXXX"
export PINGDOM_API_KEY="XXXXXXXXXXXXX"
Target the live cluster
aws eks --region eu-west-2 update-kubeconfig --name live
Set cluster related environment variables
# TF_VAR_cluster_name is referencing VPC name
export TF_VAR_cluster_name="live"
export TF_VAR_cluster_state_bucket=cloud-platform-terraform-state
export TF_VAR_cluster_state_key="cloud-platform/live/terraform.tfstate"
#needed by tf k8s provider
export KUBE_CONFIG_PATH=${HOME}/.kube/config
Set the namespace name
I’m using offender-events-dev for this example. Use whatever is relevant for your case.
export NAMESPACE=offender-events-dev
cd namespaces/live.cloud-platform.service.justice.gov.uk/${NAMESPACE}/resources
Terraform Init
terraform init \
-backend-config="bucket=cloud-platform-terraform-state" \
-backend-config="key=cloud-platform-environments/live.cloud-platform.service.justice.gov.uk/${NAMESPACE}/terraform.tfstate" \
-backend-config="region=eu-west-1" \
-backend-config="dynamodb_table=cloud-platform-environments-terraform-lock"
Note: Bucket key above is referencing to “live”, as state is stored in “live.cloud-platform.service.justice.gov.uk” for namespaces in “live” cluster.
Terraform Plan/Apply
terraform plan
The terraform plan should output No changes. Infrastructure is up-to-date.
2. Identify the compromised terraform object
$ terraform show | less
Look for the compromised access key. In this case, the access key was AKIAXXXXXXXXXXXXXXXX, and it occurs in two places in the output.
...
kubernetes_secret.ecr-repo-my-repo:
id = my-namespace/ecr-repo-my-repo
data.% = 3
data.access_key_id = AKIAXXXXXXXXXXXXXXXX
data.repo_url = ...
data.secret_access_key = ...
...
module.ecr-repo-my-repo.aws_iam_access_key.key:
id = AKIAXXXXXXXXXXXXXXXX
secret = ...
ses_smtp_password = ...
...
The first occurence is when terraform stores the credentials in a kubernetes secret.
The second occurence is the one we want, where terraform creates the credentials (the aws_iam_access_key.key resource, for the ECR repo).
3. Destroy the compromised key
$ terraform destroy --target=module.ecr-repo-my-repo.aws_iam_access_key.key
If this looks like it’s going to do the right thing, enter ‘yes’ to confirm.
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
- destroy
Terraform will perform the following actions:
- module.ecr-repo-my-repo.aws_iam_access_key.key
Plan: 0 to add, 0 to change, 1 to destroy.
4. Let terraform create a new key
$ terraform plan
This should report that it will create a new aws_iam_access_key.key resource and modify the corresponding kubernetes_secret resource.
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
~ update in-place
Terraform will perform the following actions:
~ kubernetes_secret.ecr-repo-my-repo
data.%: "" => <computed>
data.access_key_id: "AKIAXXXXXXXXXXXXXXXX" => ""
data.repo_url: "..." => ""
data.secret_access_key: "..." => ""
+ module.ecr-repo-my-repo.aws_iam_access_key.key
id: <computed>
encrypted_secret: <computed>
key_fingerprint: <computed>
secret: <computed>
ses_smtp_password: <computed>
status: <computed>
user: "ecr-user-0000000000000000"
Plan: 1 to add, 1 to change, 0 to destroy.
If all is well:
$ terraform apply
If this looks like it’s going to do the right thing, enter ‘yes’ to confirm.
At this point, a new set of AWS credentials should have been created for the existing IAM user, and the kubernetes secret should contain the new access key and secret.
Rotate RDS Credentials
If a user’s RDS credentials(database_username and database_password) have been exposed. Follow “Rotate User AWS Credentials” guidance above until step 2, in step 3 instead of “Destroy the compromised key”, we should taint the password.
$ terraform taint module.rds.random_password.password
Running taint will show the below message.
Resource instance module.rds.random_password.password has been marked as tainted.
Let terraform create a new password
$ terraform plan
This should report that it will create a new rds.random_password.password resource and modify the corresponding kubernetes_secret resource.
Terraform will perform the following actions:
# kubernetes_secret.rds will be updated in-place
~ resource "kubernetes_secret" "rds" {
~ data = (sensitive value)
id = "rds/postgres"
}
# module.rds.aws_db_instance.rds will be updated in-place
~ resource "aws_db_instance" "rds" {
id = "cloud-platform-xxxxxx"
name = "dbxxxxxxx"
~ password = (sensitive value)
tags = {
"application" = "app"
"business-unit" = "business-unit"
"environment-name" = "production"
"infrastructure-support" = "team@digital.justice.gov.uk"
"is-production" = "true"
"namespace" = "namespace-name"
"owner" = "namespace-woner"
}
+ timeouts {
+ create = "2h"
+ delete = "2h"
+ update = "2h"
}
}
# module.rds.random_password.password is tainted, so must be replaced
-/+ resource "random_password" "password" {
~ id = "none" -> (known after apply)
~ result = (sensitive value)
# (9 unchanged attributes hidden)
}
Plan: 1 to add, 2 to change, 1 to destroy.
If all is well:
$ terraform apply
If this looks like it’s going to do the right thing, enter ‘yes’ to confirm.
At this point, a new RDS password should have been created, and the kubernetes secret should contain the db password.
Note: It is possible that applications might experience downtime if, for example, a pod which was launched with the old password drops a DB connection and tries to open a new one (which will fail, because the password is no longer valid). To make pods pick up the new password, perform a manual rollout on every relevant deployment:
bash
kubectl rollout restart "deployment/{deployment}" -namespace="{namespace}"
This will rotate all pods according to the rollout strategy used in deployments in the namespace, which will pick up the new DB password from the kubernetes secret.