Skip to main content
Cloud Platform Runbooks
Menu
Feedback / Report a problem
Documentation
GitHub
Table of contents
Search (via Google)
Search
Cloud Platform Runbooks
Cloud Platform Team Alliance
How We Work
Getting Help
Sprints and ceremonies
Firebreak
Story
Story points
The Board / Tickets
Running out of tickets mid sprint
Adding tickets to the backlog
Making changes to code
Reviewing/Merging PRs
Support Squad
The 🔨 Hammer of Justice
Secondary Support
Incident Process
1. Confirm that the event constitutes an incident
2. Declare the incident
Examples
3. Assign roles
3.1 Incident Lead
3.2 Scribe
3.3 Communications Lead
Transferring roles
4. Fix the problem
5. End the incident
6. Post-incident procedure
Incident Log
Incident on 2024-09-20 - EKS Subnet Route Table Associations destroyed
Incident on 2025-01-13 - Auth0 Terraform provider credentials exposed
Incident on 2025-03-18 - AWS EKS Upgrade to 1.30 Desheduler kill pods on Cordon Nodes
Incident on 2024-07-25 Elasticsearch no longer receiving logs
Incident on 2024-04-15 - Prometheus restarted during WAL reload several times which resulted in missing metrics
Incident on 2023-11-01 - Prometheus restarted several times which resulted in missing metrics
Incident on 2023-09-18 - Lack of Disk space on nodes
Incident on 2023-08-04 - Dropped logging in kibana
Incident on 2023-07-25 - Prometheus on live cluster DOWN
Incident on 2023-07-21 - VPC CNI not allocating IP addresses
Incident on 2023-02-02 - CJS Dashboard Performance
Incident on 2023-06-06 - User services down
Incident on 2023-01-11 - Cluster image pull failure due to DockerHub password rotation
Incident on 2023-01-05 - CircleCI Security Incident
Incident on 2022-11-15 - Prometheus eks-live DOWN
Incident on 2022-07-11 - Slow performance for 25% of ingress traffic
Incident on 2022-03-10 - All ingress resources using *.apps.live.cloud-platform urls showing certificate issue
Incident on 2022-01-22 - some DNS records got deleted at the weekend
Incident on 2021-11-05 - ModSec ingress controller is erroring
Incident on 2021-09-30 - SSL Certificate Issue in browsers
Incident on 2021-09-04 - Pingdom check Prometheus Cloud-Platform - Healthcheck is DOWN
Incident on 2021-07-12 - All ingress resources using *apps.live-1 domain names stop working
Incident on 2021-06-09 - All users are unable to create new ingress rules, following bad ModSec Ingress-controller upgrade
Incident on 2021-05-10 - Apply Pipeline downtime due to accidental destroy of Manager cluster
Incident on 2020-10-06 - Intermittent “micro-downtimes” on various services using dedicated ingress controllers
Incident on 2020-09-28 - Termination of nodes updating kops Instance Group
Incident on 2020-09-21 - Some cloud-platform components destroyed
Incident on 2020-09-07 - All users are unable to create new ingress rules
Incident on 2020-08-25 - Connectivity issues with eu-west-2a
Incident on 2020-08-14 - Ingress-controllers crashlooping
Incident on 2020-08-07 - Master node provisioning failure
Incident on 2020-08-04
Incident on 2020-04-15 Nginx/TLS
Incident on 2020-02-25
Incident on 2020-02-18
Incident on 2020-02-12
Change Process in Cloud Platform
Making Changes to cloud-platform-infrastructure
Making Changes to Terraform modules
Making Changes to Helm Charts
Making changes to environments (Service Teams)
Communications
Add Concourse to a test cluster
Pre-requisites
Process
Upgrade EKS cluster
Pre-requisites
Creating Cluster Upgrade GitHub Issues
Upgrade Steps
Compatibility Check
 Preparing for upgrade
Monitoring the upgrade
Starting the upgrade
Finishing the upgrade
Finishing touches
Upgrade AMI Version
Recycle all nodes
Upgrade EKS addons
Listing available EKS upgrades
eksctl Install
Preparing for upgrade
Starting the upgrade
Finish the upgrade
Upgrade EKS Terraform Module
Pre-requisites
Upgrade Steps
Upgrade with no breaking changes
Upgrade with breaking changes
Making changes to EKS node groups, instances types, or launch templates
Process for recycling all nodes in a cluster
Notes:
Useful commands:
Monitoring nodes
Upgrade Terraform Version
Introduction
Recommendations
Caveats
How to perform the upgrade - divide and conquer
Before the upgrade
Environments state files
Infrastructure state files
Upgrade cluster components
Planning
Testing the upgrade in a test cluster
Setup environment
Create test cluster
Run a shell in the tools image
Authenticate to the test cluster
Run the integration tests
Testing the upgrade
Things to observe when testing the upgrade
Performing the upgrade
Delete a cluster
Delete the cluster with Concourse delete-cluster pipeline
Delete an EKS cluster manually
Upgrade user components
Making changes to your module
Semantic Versioning and release
Upgrading a user component in environments
Container Images used by Cluster Components
How to update this runbook
Check current components images versions
Latest version for k8s 1.30
Latest version available
Urgency
calico-apiserver
calico-system
cert-manager
concourse
external-secrets-operator
gatekeeper-system
ingress-controllers
kube-system
kuberos
logging
monitoring
overprovision
tigera-operator
trivy-system
velero
Updating Prisoner Content Hub WAF
Moving components modules into core
Notes on the pipeline
 Process
Disaster Recovery
What is .terraform.lock.hcl?
Working with .terraform.lock.hcl files
Rules
Commiting changes to the lock file
Add nodes to the AWS EKS cluster
Requirements
Cluster configuration:
Credentials rotation for auth0 apps
Preparation
1) Taint resources (terraform)
2) Apply changes within components (terraform)
3) Verifying changes
4) Update Manager cluster within components (terraform)
Monitor EKS Cluster
Monitoring with K9s
Installation
Launching K9s
Monitoring Nodes
Monitoring Pods
Monitoring Events
Further reading
Monitoring with Stern
Basic Usage
Further reading
Git-crypt
Adding new user to the keyring
Rotating the git-crypt key
Add a custom domain
Add a new Alertmanager receiver and a slack webhook
Pre-requisites
Creating a new receiver set
Information Alerts
Create a Pingdom integration id
How to create integration id (webhook)
Cloud Platform Disaster Recovery Scenarios
Losing a Namespace
Impact
Possible Cause
Restore process
Losing a Kubernetes Component or Object
Impact
Possible Cause
Restore process
Losing the whole cluster
Impact
Possible Cause
How this plan is tested:
Assumptions
Restore process
Restoring a Lost Cluster to a New One with a Different Name
Deleted terraform state
Impact
Possible Cause
Restore process
Recovering more complex scenarios
Resolving a PartiallyFailed backup alert
Create and access bastion node.
Create bastion node
Access bastion node
Rotate User AWS Credentials
Set pingdom environment variables(Optional)
Target the live cluster
Set cluster-related environment variables
Set the namespace name
Terraform Init
Terraform Plan/Apply
2. Identify the compromised terraform object
3. Destroy the compromised key
4. Let terraform create a new key
AWS Compromised Credentials
Steps for a leaked credentials
Getting new credentials
Audit the compromised credentials
AWS Console Access
Steps to create/delete Cloud Platform team users
Activating MFA for new users
Modifying Cloud Platform users permissions
Troubleshooting for modifying Cloud Platform users permissions
Make an S3 bucket public
Delete Prometheus Metrics
1. Identify the metrics you want to delete
2. Enable the admin interface in Prometheus
3. Launch a port-forward pod
4. Forward local traffic to Prometheus
5. Use curl (in another terminal) to hit the API endpoint
6. Use this script to delete multiple metrics
7. Clean up
Manually Plan/Apply Namespace Resources in live cluster
Start in the appropriate branch of the environments repo
Set pingdom environment variables(Optional)
Target the live cluster
Set some environment variables
Set the namespace name
Terraform Init
Terraform Plan/Apply
Manually Delete Namespace Resources
Prerequisites
Environment Variables
Deleting a namespace
Locating PR number
RDS Snapshots
Prerequisites
Considerations
Restoring live services from an RDS DB snapshot
1. Get the Database Instance Identifier
2. List Available Snapshots
5. Add SKIP_FILE to Pause Pipeline
4. Create a Test Database with Sample Data (For Testing)
5. Verify Storage Configuration
6. Modify the RDS Module for Restoration
7. Rename the Current Database Instance Before Merging
8. Handle Final Snapshot Conflicts
Expected Timeline and Status Changes
Verification After Restoration
Version Downgrade Limitations and Process
Important: Version Downgrades Cannot Be Done In-Place
Example of a Failed Downgrade Scenario
Correct Process for Version Downgrades
Known Issues and Solutions
Final Snapshot Naming Conflicts
Engine Version Conflicts
Read Replica Issues
Metrics and Observations
Manage Published Grafana Dashboard Snapshots
Steps to remove Snapshot
Remove data from Elasticsearch
Stop the breach first
Things to know
Get yourself access
Build your query
Delete by query
Removing specific logs filtered by phrase
Deleting documents stored in warm storage
Open search best practices
Some general info about shards
FreeStorageSpaceTooLow alerts
Connecting to the Elastic search api
Connecting to the OpenSearch api
Verify you’re connected to the api
Modsec logging architecture
Debugging
OpenSearch PodSecurity Violations Alert
OpenSearch Alert/Monitor
Checking logs for PSA violations in OpenSearch
Fixing PSA Violations
Slack Alert
Delete terraform state lock
Command-line method
AWS Console method
Terraform command
Terraform state lock - Error refreshing state
How to Investigate Divergence Errors
Reproduce the plan
How to Investigate External-Dns Errors
Troubleshooting
Invalid Change Batch
Rate Limited / Throttled
How to Investigate PrometheusOperatorReconcile Errors
Troubleshooting
Analyze VPC Flow Logs
Recycle-node
Recycle-all-nodes
Recycling process
High level method
Revoke auth0 kubeconfig access token
1. Revoke existing tokens generated from github
2. Recreate ​​auth0_client.kubernetes
Creating a live-like cluster
Pre-requisites
Setting cluster size to match Live
Installing live components and test applications
Upgrading a live-like test cluster
Monitoring the upgrade
Final Tests
Tearing down
Provisioning EKS clusters
Pre-requisites
Environment Variables
Provisioning an EKS cluster with the cloud-platform CLI
Manually provisioning a cluster
1. VPC
2. Creating EKS cluster
3. Deploy core
4. Deploy components
Deleting your test cluster
Provisioning a custom cluster
Change load balancer alias to the interface IP’s in Route53.
Request AWS to restart the health check
Change load balancer alias
Expanding Persistent Volumes created using StatefulSets
Velero - Cluster backups and disaster recovery
Backups
Why?
What?
How?
Cloud Platform helm chart repository
Performing CRUD on Cloud Platform Helm repository
Access EKS cluster
Pre-requisites
Create kubeconfig using aws command
Create kubeconfig manually using a template
OpenSearch modsec setup
Get an audit log from modsec (when fluent-bit is not pushing to OpenSearch)
How do I check the audit log
Perform a search for the unique-id (obtained from the OpenSearch entry)
Create custom cluster
Deleting your custom cluster
 Rerunning stages in the create custom cluster pipeline
Run integration tests against custom cluster
Debugging AWS Console read-only access issues
GitHub teams Principal Tag character limit exceeded
Terminal access to EKS managed nodes
Overview
Pre-requisites
Steps to get terminal access via Console
Steps to get terminal access via AWS cli
Access via SSH /SCP
Open Policy Agent policies
Onboarding into the Cloud Platform Team
People Team
Service Desk
Sit down with DM
Cloud Platform team
Access
Custom default-backend
Background
Creating your own custom error page
1. Create your docker image
2. Creating a service and deployment
3. Define annotations in your ingress file.
Use the platform-level error page
Serve all errors from your custom default backend
Adding a route to connect to the MOJ Transit Gateway
Quick introduction
Transit Gateway
Making the change
Adding routes from live-2 VPC to MoJ Transit Gateway
Destroy Concourse Build Data
Overview
Steps to remove the build data
Leavers Guide
Revoking Access
Digital Services
AWS Accounts
Other 3rd Party Accounts access removal
Line manager actions
Pushing logs to the SOC team
1. Cloudtrail logs
Architecture
2. live-1 VPC FlowLogs
Architecture
3. Route53 logs
Architecture
4. EKS logs
Todo
IAM User access keys rotation
Blocking Public IP Address from EKS Cluster
Introduction
Adding deny rules to the public network ACL
Scheduled PR Reminders
Steps required for new repositories
Grafana Dashboards
Kubernetes Number of Pods per Node
Dashboard Layout
Troubleshooting
Fixing “failed to load dashboard” errors
Fixing “duplicate dashboard uid” errors
Going on call
What’s expected?
Expected:
Not expected:
Where do I start?
What do I get for being on call?
Civil servants
Cloud Platform Communications Plan
The Plan
Tips on format of communications
Examples
Things to include in incident communications
Example
Things to include in upgrade communications
Example
Sharing information with the wider Ministry of Justice and the Public
Google Search Console and Indexing
Tips and Tricks
Delete an RDS database snapshot
Check the expiration date of the SSL certificate for a live domain
Delete a “stuck” resource
Filter namespaces by specific label or annotation
Find all pods running on a specific worker node
Count pods running on all nodes
Add more RSS feeds to #cloud-platform-rss channel
Find files which don’t contain a particular string
Get AWS EC2 instance information for a node
Create a one-off job to repeat a cronjob
Hijack a concourse job container
Help when manually deleting AWS resources
Find out why your namespace isn’t deleting
Modsec false positives
Identifying and Managing Untagged AWS Resources
Overview
Step-by-Step Process
1. Identify untagged resources by type
2. Investigate services not visible in Tag Editor
3. Exceptions – organisation-level resources
4. Verifying the Change
Investigating blocked ingress spikes
Communication
Other Links
Debugging 101
Grafana Dashboards
Kubernetes / View / Nodes
Kubernetes / Compute Resources / Node (Pods)
Kubernetes / Views / Pods
Kubernetes Nginx Ingress Controller NextGen - DevOps Nirvana2
How to Investigate cert-manager Errors
Troubleshooting
Invalid Change Batch
Current fix
Add a new runbook
OpenSearch Storage Issues
Hot/Warm Node Parity
Manually migrating failed indexes
Further Reading
Cloud Platform Runbooks
These runbooks are used and maintained by the MoJ Digital Cloud Platform team.