Change Process in Cloud Platform
The Cloud Platform team implements and encourages an Infrastructure as Code (IaC) methodology to manage the platform. It means every single change is done through code (if you spot something different, raise it with us please).
cloud-platform-infrastructure repository is our main repository where infrastructure gets created and all components fit together. From this repository everything gets linked.
Mostly all components are Terraform modules, and they are managed in their own git repositories; you can list all modules here.
Making Changes to cloud-platform-infrastructure
- Create a PR which includes the change you want to make. Make sure the plan pipeline is happy with the change. It is good practice to link the PR to the issue that describes the change.
- Chase the team to get your PR approved, and if it involves downtime, ensure you have communications in place.
- For certain changes, pausing the infrastructure apply pipelines and running them individually is favourable. If you are pausing bootstrap and infrastructure pipelines, be sure to communicate this in the team slack channel.
- Merge and wait until the apply pipeline applies your change.
Making Changes to Terraform modules
Updating a module is very simple and consists of the following steps:
- Create a PR and ask a team member for approval.
- Merge PR
- Create a release following semantic version spec.
- In order to keep our definitions/instances up to date, update all existing instances in environments repo and cloud-platform-infrastructure repo. If this step affects team’s environments it will need planning and communication.
Steps 2 and 3 can be done at any time and they don’t affect the running infrastructure, since all Terraform modules calls have a pinned release version. No Terraform module should point to master branches or latest releases automatically.
Finally, create the PR which updates the definition where your module is called and get ready to apply it following the guidelines above. For clarity, reference the module release in the PR description.
Making Changes to Helm Charts
Cloud Platform’s Helm Charts are hosted in cloud-platform-helm-charts repository using Github Pages. Changes in Helm Charts should include:
- Bump chart version everytime you make a change
- Regenerate the
index.ymlfollowing this instructions - PR against gh-pages branch
Making changes to environments (Service Teams)
The Cloud Platform source of truth for team’s environments (Kubernetes namespaces) is cloud-platform-environments. Changes under this repository follow a similar process:
- Create a PR and verify the planned changes in the plan pipeline. If you are happy with them, ask a team member for approval.
- Merge PR
Within the environments repository, we have an end to end process fully automated with pipelines. Once the change is merged, it is going to be automatically applied by the apply pipeline (apply-live and apply-namespace-changes-live and if the change is destroying a namespace, it will be detected by detect-deleted-namespace pipeline.
Communications
If any changes involve downtime, we must inform teams about it via Slack (#cloud-platform-update). It is important to include which services are going to be down and for how long.
Sometimes teams misinterpret downtimes of Cloud Platform services to involve downtime of their services, so it is very important to be precise about the change and its potential impact.