Skip to main content

Credentials rotation for auth0 apps

Cloud-Platform team uses auth0 in almost every component requiring authentication (clusters, Prometheus, Kibana, Grafana, etc). To rotate credentials we should we must follow steps below but it is important to keep in mind that users will require to re-authenticate (generate new kubeconfig file)

Preparation

  • Ensure users are aware of the change. It is important to point out that they will have to reauthenticate against the cluster after the change (following user guide).
  • You must have all the tools and variables (such as Terraform, TF’s auth0 provider, environment variables, kops, etc) ready to apply changes against our infrastructure repo.
  • It will be handy to have auth0 management console already opened in a browser tab, so you can verify and monitor the changes.

1) Taint resources (terraform)

Tainting auth0 resource will force new credentials generation. Go to cloud-platform-infrastructure/terraform/cloud-platform directory and run

$ terraform workspace select live-1
$ terraform taint module.auth0.auth0_client.components

Make sure with terraform plan that auth0 resource is going to be recreated as well as kops template file (it’s going to be needed in the next step).

Apply changes if you are happy with them:

$ terraform apply

2) Rolling update of cluster (kops)

From the previous step a new cloud-platform-infrastructure/kops/live-1.yaml file should have been generated. To apply this file we already have a runbook to follow.

NOTE: The change will require rolling replacement of the master nodes

3) Apply changes within components (terraform)

Execute terraform plan inside cloud-platform-infrastructure/terraform/cloud-platform-components directory and to ensure changes match resources below, if they do, apply them:

$ terraform apply -target=module.prometheus.kubernetes_secret.grafana_secret -target=module.prometheus.helm_release.prometheus_proxy -target=module.prometheus.helm_release.alertmanager_proxy  -target=helm_release.kuberos

Unfortunately, grafana pod will not pick up the secret change, so it needs to be recyled (just delete it and it will automatically get redeployed):

$ GrafanaPodName=$(kubectl -n monitoring get pods -o name | grep grafana)
$ kubectl -n monitoring delete pod $GrafanaPodName

4) Update environment repo

We already have a ticket to get rid of this step. Meanwhile we get it done, we should update oidc-components-secret.yaml file with the new auth0 credentials.

5) Verifiying changes

In order to verify that the changes were successfully applied, follow the checklist below (order doesn’t matter):

6) Update pipelines

Our pipelines read auth0 credentials from a K8S secret inside the manager cluster. This secret is updated through concourse’s TF module variable called tf_provider_auth0_client_secret and tf_provider_auth0_client_id in cloud-platform-infrastructure/blob/main/terraform/cloud-platform-eks/components/terraform.tfvars

You only need to update the source code file and have your changed merged to master. The bootstrap pipeline will apply your change.

This page was last reviewed on 20 April 2021. It needs to be reviewed again on 20 July 2021 by the page owner #cloud-platform .
This page was set to be reviewed before 20 July 2021 by the page owner #cloud-platform. This might mean the content is out of date.