Credentials rotation for auth0 apps
Cloud-Platform team uses auth0 in almost every component requiring authentication (clusters, Prometheus, Kibana, Grafana, etc). To rotate credentials we should we must follow steps below but it is important to keep in mind that users will require to re-authenticate (generate new kubeconfig file)
- Ensure users are aware of the change. It is important to point out that they will have to reauthenticate against the cluster after the change (following user guide).
- You must have all the tools and variables (such as Terraform, TF’s auth0 provider, environment variables, kops, etc) ready to apply changes against our infrastructure repo.
- It will be handy to have auth0 management console already opened in a browser tab, so you can verify and monitor the changes.
1) Taint resources (terraform)
Tainting auth0 resource will force new credentials generation. Go to
cloud-platform-infrastructure/terraform/cloud-platform directory and run
$ terraform workspace select live-1 $ terraform taint module.auth0.auth0_client.components
Make sure with
terraform plan that auth0 resource is going to be recreated as well as kops template file (it’s going to be needed in the next step).
Apply changes if you are happy with them:
$ terraform apply
2) Rolling update of cluster (kops)
NOTE: The change will require rolling replacement of the master nodes
3) Apply changes within components (terraform)
terraform plan inside
cloud-platform-infrastructure/terraform/cloud-platform-components directory and to ensure changes match resources below, if they do, apply them:
$ terraform apply -target=module.prometheus.kubernetes_secret.grafana_secret -target=module.prometheus.helm_release.prometheus_proxy -target=module.prometheus.helm_release.alertmanager_proxy -target=helm_release.kuberos
Unfortunately, grafana pod will not pick up the secret change, so it needs to be recyled (just delete it and it will automatically get redeployed):
$ GrafanaPodName=$(kubectl -n monitoring get pods -o name | grep grafana) $ kubectl -n monitoring delete pod $GrafanaPodName
4) Update environment repo
5) Verifiying changes
In order to verify that the changes were successfully applied, follow the checklist below (order doesn’t matter):
- You can authenticate to the cluster (follow user guide)
- Ensure you can authenticate to Grafana
- Ensure you can authenticate to AlertManager
- Ensure you can authenticate to Kibana
- Ensure you can authenticate to Prometheus
6) Update pipelines
Our pipelines read auth0 credentials from a K8S secret inside the manager cluster. This secret is updated through concourse’s TF module variable called
tf_provider_auth0_client_id in cloud-platform-infrastructure/blob/main/terraform/cloud-platform-eks/components/terraform.tfvars
You only need to update the source code file and have your changed merged to master. The bootstrap pipeline will apply your change.