Rotate User AWS Credentials
Most of the terraform modules supported by Cloud Platform uses IRSA and hence no long term credentials are created. If any other resources refer to credentials created by terraform sucn as IAM user, Cognito users, etc, then use this runbook if those AWS credentials have been exposed.
In this case, ‘AWS credentials’ refers to credentials created by terraform, for resources in the user’s namespace. Use this runbook if a user’s AWS credentials have been exposed.
We will make terraform destroy the compromised credentials, and then recreate them. For this example, we will use the case where an IAM user access key and secret in a namespace was exposed.
Please amend the code snippets below to match your actual circumstance.
1. Prepare terraform to manipulate the user’s AWS resources
You need a working copy of the environments github repo.
cd cloud-platform-environments
git checkout main
git pull
Launch the tools-shell. This will have all the binary with correct versions needs for performing terraform operations
make tools-shell
Set pingdom environment variables(Optional)
If the changes involve applying “pingdom_check”, set the environment variables for pingdom.
The values are stored as secrets in manager cluster - concourse-main namespace.
Target the live cluster
aws eks --region eu-west-2 update-kubeconfig --name live
Set cluster related environment variables
export TF_VAR_vpc_name="live-1"
export TF_VAR_eks_cluster_name="live"
export TF_VAR_github_owner="ministryofjustice"
export TF_VAR_github_token="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
export TF_VAR_kubernetes_cluster="DF366E49809688A3B16EEC29707D8C09.gr7.eu-west-2.eks.amazonaws.com"
export PINGDOM_API_TOKEN='XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
export KUBE_CONFIG_PATH=${HOME}/.kube/config
export KUBECONFIG=${HOME}/.kube/config
Set the namespace name
I’m using offender-events-dev for this example. Use whatever is relevant for your case.
export NAMESPACE=offender-events-dev
cd namespaces/live.cloud-platform.service.justice.gov.uk/${NAMESPACE}/resources
Terraform Init
terraform init \
-backend-config="bucket=cloud-platform-terraform-state" \
-backend-config="key=cloud-platform-environments/live-1.cloud-platform.service.justice.gov.uk/${NAMESPACE}/terraform.tfstate" \
-backend-config="region=eu-west-1" \
-backend-config="dynamodb_table=cloud-platform-environments-terraform-lock"
Note: Bucket key above is referencing to “live-1”, as state is stored in “live-1.cloud-platform.service.justice.gov.uk” for namespaces in “live” cluster.
Terraform Plan/Apply
terraform plan
The terraform plan should output No changes. Infrastructure is up-to-date.
2. Identify the compromised terraform object
$ terraform show | less
Look for the compromised access key. In this case, the access key was AKIAXXXXXXXXXXXXXXXX, and it occurs in two places in the output.
...
kubernetes_secret.iam-credentials:
id = my-namespace/iam-credentials
data.access_key_id = AKIAXXXXXXXXXXXXXXXX
data.secret_access_key = ...
...
aws_iam_access_key.user:
id = AKIAXXXXXXXXXXXXXXXX
secret = ...
...
The first occurence is when terraform stores the credentials in a kubernetes secret.
The second occurence is the one we want, where terraform creates the credentials (the aws_iam_access_key.key resource, for the IAM User).
3. Destroy the compromised key
$ terraform destroy --target=aws_iam_access_key.key
If this looks like it’s going to do the right thing, enter ‘yes’ to confirm.
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
- destroy
Terraform will perform the following actions:
- aws_iam_access_key.key
Plan: 0 to add, 0 to change, 1 to destroy.
4. Let terraform create a new key
$ terraform plan
This should report that it will create a new aws_iam_access_key.key resource and modify the corresponding kubernetes_secret resource.
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
~ update in-place
Terraform will perform the following actions:
~ kubernetes_secret.iam-credentials
data.access_key_id: "AKIAXXXXXXXXXXXXXXXX" => ""
data.secret_access_key: "..." => ""
+ aws_iam_access_key.key
id: <computed>
encrypted_secret: <computed>
key_fingerprint: <computed>
secret: <computed>
ses_smtp_password: <computed>
status: <computed>
user: "iam-user-0000000000000000"
Plan: 1 to add, 1 to change, 0 to destroy.
If all is well:
$ terraform apply
If this looks like it’s going to do the right thing, enter ‘yes’ to confirm.
At this point, a new set of AWS credentials should have been created for the existing IAM user, and the kubernetes secret should contain the new access key and secret.
Note: It is possible that applications might experience downtime if, for example, a pod which was launched with the old password drops the connection to AWS and tries to open a new one (which will fail, because the password is no longer valid).
To make pods pick up the new password, perform a manual rollout on every relevant deployment:
bash
kubectl rollout restart "deployment/{deployment}" -namespace="{namespace}"
This will rotate all pods according to the rollout strategy used in deployments in the namespace, which will pick up the new iam keys from the kubernetes secret.