RDS Snapshots

RDS snapshots have multiple purposes: migrations, backups, etc. When RDS DB instances are created using the Cloud Platform RDS terraform module, an IAM user account is created for management purposes. This user can create, delete, copy, and restore RDS snapshots.

This team runbook provides technical guidance for testing and verifying the RDS snapshot restoration process.

Warning Where a team asks for help restoring a production database, advise them to rehearse the same restore in a non-production environment first and to proceed in production only once that has succeeded. Record that advice in the support ticket.

Prerequisites

Before performing an RDS restore from snapshot:

Confirm the team has rehearsed the restore in a non-production environment
Ensure you have AWS CLI access with appropriate permissions
Verify the existence of snapshots for the target database
Identify the correct DB instance identifier
Check any existing snapshots with potential naming conflicts

Confirm the snapshot belongs to the database being restored

Check the snapshot’s source instance, engine version, storage and master username before using it:

aws rds describe-db-snapshots \
  --db-snapshot-identifier <snapshot-identifier> \
  --query 'DBSnapshots[0].[DBInstanceIdentifier,InstanceCreateTime,EngineVersion,AllocatedStorage,MasterUsername,KmsKeyId]' \
  --output text

InstanceCreateTime is the creation time of the database the snapshot came from, not the time the snapshot was taken. It is the most reliable way to confirm a snapshot’s lineage when several snapshots share a similar name.

A snapshot taken from a different database carries that database’s master username and KMS key. See Restored Database Has the Wrong Credentials.

Use a manual snapshot

Automated snapshots, prefixed rds:, are tied to the lifecycle of their source instance and are deleted with it. Terraform destroys the existing instance before creating its replacement, so an automated snapshot can be removed part-way through a restore.

Copy an automated snapshot to a manual one before using it:

aws rds copy-db-snapshot \
  --source-db-snapshot-identifier <automated-snapshot-id> \
  --target-db-snapshot-identifier <new-manual-snapshot-name> \
  --copy-tags \
  --region eu-west-2

Copying is also the safest way to preserve a snapshot that Terraform would otherwise overwrite, or that a team needs held under a stable name. Copies are independent, so deleting the source does not affect the copy.

Considerations

The amount of manual snapshots per AWS account is limited, so it’s important to cleanup old snapshots
Daily snapshots are provided “out of the box”, and do not count towards the “Manual Snapshots” total
Managing snapshots is the teams’ responsibility (as are snapshot restores), so teams are responsible for cleaning up unneeded manual snapshots in order to avoid hitting our AWS account limits

Restoring live services from an RDS DB snapshot

If you want to restore your production RDS DB instance from a previous snapshot, either because the database is corrupted or the whole database was deleted, here is the procedure:

1. Get the Database Instance Identifier

aws rds describe-db-instances --query 'DBInstances[*].[DBInstanceIdentifier]' --output text

Or, alternatively:

cloud-platform decode-secret -n <your namespace> -s <the secret storing your RDS details>

Look for a line like this in the output:

{
  "rds_instance_address": "cloud-platform-abcdefghij.abcdefghijkl.eu-west-2.rds.amazonaws.com"
}

The db-instance-identifier is the cloud-platform-abcdefghij part.

For more detailed information about a specific instance:

aws rds describe-db-instances --db-instance-identifier <db-instance-identifier>

Example output will include critical information:

{
  "DBInstanceIdentifier": "cloud-platform-klmno12345",
  "DBInstanceClass": "db.t4g.micro",
  "Engine": "mysql",
  "DBInstanceStatus": "available",
  "MasterUsername": "cpuser123",
  "DBName": "dbexample123",
  "Endpoint": {
    "Address": "cloud-platform-klmno12345.cdwm328dlye6.eu-west-2.rds.amazonaws.com",
    "Port": 3306,
    "HostedZoneId": "Z1TTGA775OQIYO"
  },
  "AllocatedStorage": 20,
  "InstanceCreateTime": "2025-02-18T15:32:07.509000+00:00",
  "PreferredBackupWindow": "03:29-03:59",
  "BackupRetentionPeriod": 7
}

2. List Available Snapshots

aws rds describe-db-snapshots --db-instance-identifier <db-instance-identifier> --query 'DBSnapshots[*].[DBSnapshotIdentifier,SnapshotCreateTime]' --output text

This will list all available snapshots for the specified database along with their creation timestamps.

3. Pausing the Pipeline Is Currently Unavailable

Earlier revisions of this runbook instructed teams to add an APPLY_PIPELINE_SKIP_THIS_NAMESPACE file at the root of the namespace to stop the apply pipeline running against it during a restoration.

Warning: The pipeline mechanism this depends on is currently out of service. Adding the file has no effect, so do not ask teams to add it and do not rely on it to hold a namespace still. Remove this notice once the mechanism is back in service.

Agree directly with the team instead that no further changes are merged for the namespace until the restoration is complete and verified. An unrelated apply landing part-way through a replacement is difficult to unpick, because a replacement destroys the instance before recreating it.

4. Create a Test Database with Sample Data (For Testing)

Before proceeding with any live restoration, it’s recommended to test the process:

4.1 Seed Test Data Process

Create a port-forward pod to establish connection with the RDS database:

   kubectl -n <namespace> run -it --rm --image=mysql:5.7 mysql-client -- bash

Connect to the RDS database:

   mysql -h127.0.0.1 -P3306 -u<username> -p<password>

Example: bash mysql -h127.0.0.1 -P3306 -ucpuser123 -pmysecretpassword

Select your database:

   USE <database_name>;

Example: sql USE dbexample123;

Create a test table:

   CREATE TABLE IF NOT EXISTS restore_test (
     id INT AUTO_INCREMENT PRIMARY KEY,
     info VARCHAR(255)
   );

Seed test data:

   INSERT INTO restore_test (info) VALUES ('Test entry 1'), ('Test entry 2');

Verify data insertion:

   SELECT * FROM restore_test;

4.2 Create a Manual Snapshot

Before making any changes, create a manual snapshot as a safety measure:

aws rds create-db-snapshot \
  --db-instance-identifier <db-instance-identifier> \
  --db-snapshot-identifier <custom-identifier-with-date>

5. Verify Storage Configuration

Check the current storage allocation settings for the database:

aws rds describe-db-instances --db-instance-identifier <db-instance-identifier>

Note the AllocatedStorage and MaxAllocatedStorage values. By default, these are set to 20GB and 500GB respectively if not explicitly defined in the module.

The target database’s db_allocated_storage must be at least equal to the AllocatedStorage of the database the snapshot came from. A restore cannot be used to reduce storage, and the apply fails with:

InvalidParameterCombination: Invalid storage size for engine name postgres and storage type gp3: <size>

If db_allocated_storage has to be raised to accommodate the restore, the increase must be at least 10 percent of the current value. AWS rejects anything smaller: “You can increase the allocated space on a storage volume by a minimum of 10%. You can’t deallocate space.” See Increasing DB instance storage capacity.

Establishing how much storage is actually needed

Teams often quote the size shown against a snapshot in the RDS console and ask to restore into something smaller. That figure is not returned by the API and is not a safe basis for sizing the target.

describe-db-snapshots reports only AllocatedStorage, which is the volume provisioned on the source instance rather than the data held in the snapshot:

aws rds describe-db-snapshots \
  --db-snapshot-identifier <snapshot-identifier> \
  --query 'DBSnapshots[0].[DBSnapshotIdentifier,AllocatedStorage,InstanceCreateTime,EngineVersion]' \
  --output text

To establish how much data a running instance actually holds, subtract the FreeStorageSpace CloudWatch metric from its AllocatedStorage:

aws cloudwatch get-metric-statistics \
  --namespace AWS/RDS \
  --metric-name FreeStorageSpace \
  --dimensions Name=DBInstanceIdentifier,Value=<db-instance-identifier> \
  --start-time "$(date -u -v-3H '+%Y-%m-%dT%H:%M:%SZ')" \
  --end-time "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" \
  --period 300 \
  --statistics Minimum \
  --region eu-west-2 \
  --query 'sort_by(Datapoints,&Timestamp)[-1]'

The value is returned in bytes. The date -u -v-3H syntax is macOS; on Linux use date -u -d '3 hours ago'.

This establishes whether a shrink is worth pursuing, but it cannot be delivered through a snapshot restore. Point teams at the Migrating an RDS instance guide, which covers a logical dump and reload.

6. Modify the RDS Module for Restoration

Create a PR to modify the RDS module in the rds.tf file to include:

module "example_team_rds" {
  source = "github.com/ministryofjustice/cloud-platform-terraform-rds-instance?ref=5.1"
  # ... other parameters ...
  snapshot_identifier = "<your-snapshot-identifier>"
  # ... other parameters ...
}

Important: When using a snapshot for restoration, do not attempt to change storage parameters at the same time. The RDS instance will adopt the storage configuration from the snapshot.

It is crucial to retain other original settings associated with the DB instance when restoring from the snapshot. Details such as db-engine-version, db-engine, rds-family must be the same as the original instance when the snapshot was taken.

Do not merge this PR yet - there are additional steps required first.

7. Rename the Current Database Instance Before Merging

As a Cloud Platform team member handling this restoration request:

Identify the DB instance identifier provided by the user requesting the restoration
Rename the current database in the AWS console:
- Navigate to AWS Console → RDS → Databases
- Select the target database
- Choose “Modify”
- Append .old or a similar suffix to the DB instance identifier
- Apply the change immediately
Once the DB has been renamed, merge the PR with the RDS changes once approved.

This process preserves the database endpoint and credentials when the new instance is created from the snapshot.

8. Handle Final Snapshot Conflicts

The pipeline may fail due to final snapshot name conflicts. This occurs because Terraform attempts to create a final snapshot with the same name during recreation.

Why This Happens:

When you delete an RDS instance, AWS/Terraform automatically names the final snapshot as {db-instance-identifier}-finalsnapshot.

Example: If your DB instance is named mydb, the final snapshot becomes mydb-finalsnapshot.

Conflict Scenario:
- You delete mydb → Final snapshot mydb-finalsnapshot is created.
- Later, you create a new mydb instance, modify it, and try to delete it again.
- Terraform/AWS attempts to create mydb-finalsnapshot again → Conflict.

If the pipeline fails with the error DBSnapshotAlreadyExists, execute:

aws rds delete-db-snapshot --db-snapshot-identifier <db-instance-identifier>-finalsnapshot

Then rerun the failed pipeline.

Warning: Check what the conflicting snapshot contains before deleting it. Where a database has already been replaced once, <db-instance-identifier>-finalsnapshot can hold the only copy of the data from before that happened. Compare its InstanceCreateTime against the database it is expected to have come from, and copy it to a safely named manual snapshot first if there is any doubt.

Deleting the final snapshot also unblocks whatever is currently merged to main. Confirm that the merged configuration contains the snapshot_identifier intended for the restore. If it does not, clearing the conflict allows the destroy to succeed and the instance is rebuilt empty.

Setting skip_final_snapshot does not resolve the conflict

Adding skip_final_snapshot = true in the same PR as snapshot_identifier does not prevent the error.

Changing snapshot_identifier forces a replacement, and the destroy half of a replacement uses the values already recorded in state, not the values in the new PR. skip_final_snapshot = true only takes effect once it has been applied and stored in state, which cannot happen in the same run.

For it to take effect it has to reach state before the replacement is planned, which means applying it on its own first:

Raise a PR changing only skip_final_snapshot = true. Nothing else in the module changes, so this is an update in place rather than a replacement, and the value is recorded in state.
Once that has applied, raise the PR adding snapshot_identifier. The destroy now reads skip_final_snapshot = true from state and writes no final snapshot, so there is nothing to conflict with.

This costs an extra apply and gives up the final snapshot, which is the last copy of the data as it stood before the restore. In most cases deleting the conflicting snapshot is the better option.

Expected Timeline and Status Changes

Total downtime: ~26 minutes from Deleting to Available. This may vary depending on how much data is being restored.

Verification After Restoration

Once the new DB instance is restored and available, verify that the data has been successfully restored:

Create a port-forward pod to establish connection with the restored RDS database:

   kubectl -n <namespace> run -it --rm --image=mysql:5.7 mysql-client -- bash

Connect to the database to verify data restoration:

   mysql -h $DB_HOST -u $DB_USERNAME -p$DB_PASSWORD $DB_NAME

Or use a direct connection:

   mysql -h127.0.0.1 -P3306 -u<username> -p<password>

Run basic queries to confirm data accessibility:

   SHOW DATABASES;
   USE <database_name>;
   SHOW TABLES;
   SELECT * FROM restore_test;  # Check if test data is present

Verify the connection string remains the same:

   cloud-platform decode-secret -n <your namespace> -s <the secret storing your RDS details>

Look for the rds_instance_endpoint value and confirm it matches the original endpoint.

Version Downgrade Limitations and Process

Important: Version Downgrades Cannot Be Done In-Place

AWS does not support downgrading a database to an earlier version on the same instance. For example, if a database has been upgraded from PostgreSQL 16.3 to 17, it cannot be downgraded back to 16.3 using the standard snapshot restoration process outlined above.

Example of a Failed Downgrade Scenario

The following failed scenario was observed and documented:

User had upgraded their RDS from PostgreSQL 16.3 to 17
User wanted to restore back to 16.3 using the process:
- Found an older snapshot identifier from before the upgrade
- Raised PR with the snapshot identifier
- Manually renamed the database in the console
- Merged the PR

What actually happened: * Terraform apply reverted the manual rename * Started deleting the database, which also deleted all automatic snapshots * First apply pipeline failed with a parameter group error * Second attempt failed because the snapshot no longer existed (deleted during first apply)

Correct Process for Version Downgrades

To properly downgrade an RDS database to a previous version:

Take a snapshot of the current database to preserve it (in case restoration is needed later):

   aws rds create-db-snapshot \
     --db-instance-identifier <current-db-instance-identifier> \
     --db-snapshot-identifier <current-version-snapshot-name>

Create a new database instance with:
- A different identifier than the current database
- Engine version matching the snapshot you want to restore from
- RDS family matching the snapshot you want to restore from
- All other configuration parameters consistent with the source of the snapshot
In the terraform module:

   module "example_team_rds" {
     source               = "github.com/ministryofjustice/cloud-platform-terraform-rds-instance?ref=5.1"
     # ... other parameters ...
     db_instance_identifier = "<new-instance-name>"  # Must be different from current DB
     db_engine_version      = "16.3"  # Earlier version
     rds_family             = "postgres16"  # Family matching earlier version
     snapshot_identifier    = "<old-version-snapshot>"
     # ... other parameters ...
   }

Note: This approach requires a cutover as the new database will have a different endpoint and connection string.

After verifying the new downgraded database works properly, the old one can be deleted if needed.

Known Issues and Solutions

Final Snapshot Naming Conflicts

Issue: The pipeline fails because it cannot create a final snapshot with the same name as an existing snapshot.

Error Message: DBSnapshotAlreadyExists

Solution: - Delete the conflicting snapshot manually:

  aws rds delete-db-snapshot --db-snapshot-identifier <db-instance-identifier>-finalsnapshot

Engine Version Conflicts

Issue: Sometimes the pipeline throws an error: InvalidParameterCombination: Cannot upgrade mariadb from 10.4.13 to 10.4.8

Solution: This will happen if AWS upgraded the minor version automatically and terraform stored the exact db_engine_version in its state.

To fix this, change the db_engine_version in your rds.tf from 10.4 to the one AWS upgraded to, in this case 10.4.13, and repeat the previous steps again.

Once the terraform apply is successful, and the new DB instance has been created, revert the changes made for db_engine_version.

Read Replica Issues

Issue: If there is an RDS read replica for the DB instance that is being restored, Terraform will throw the error: Error: cannot elect new source database for replication

Why This Happens: A replacement destroys the source database before recreating it. AWS documents that “if you delete a source DB instance without deleting its read replicas in the same AWS Region, each replica is promoted to a standalone DB instance” (Considerations when deleting replicas).

Promotion is one way. A promoted instance “ceases to function as a read replica and becomes a standalone DB instance”, and “you can’t use the DB instance as a replication target because it is no longer a read replica” (Characteristics of a promoted read replica).

Terraform therefore has no way of reattaching it to the restored database, and rerunning the pipeline will not clear the error.

Solution: Set count = 0 on the read replica module and on any Kubernetes secrets relating to the read replica. Both must change in the same PR, otherwise the plan fails with module.read_replica is empty tuple. After the restoration is complete and verified, set both back to count = 1.

Note that the read replica will have a new database identifier and endpoint once it is recreated, so any pods reading from it need restarting.

Warning: The module sets skip_final_snapshot = true and db_backup_retention_period = 0 on read replicas, so nothing is retained when one is deleted. A promoted replica holds a live copy of the data as it stood when its source was destroyed, which is often the most convenient thing to verify against during an incident. Snapshot it before setting count = 0.

Restored Database Has the Wrong Credentials

Issue: The restore reports success but the application cannot connect.

Why This Happens: The master username and password are generated by the module and held in Terraform state, but a restored instance keeps the master username from the snapshot. Restoring from a snapshot belonging to another database brings that database’s credentials across, and they will not match what the application expects.

A snapshot from another database also carries its KMS key, and the restored instance inherits that key rather than the one the module created. This leaves the database dependent on a key belonging to another namespace, which will become unrecoverable if that namespace is torn down.

Solution: Confirm the snapshot’s MasterUsername and KmsKeyId against the target database before restoring. A snapshot taken from another database carries its own KMS key, and the restored instance inherits that key rather than the one belonging to its own module. A database left encrypted under another namespace’s key is a risk in its own right, because that key is outside the owning team’s control.

To recover from a restore that used the wrong snapshot, restore again from a snapshot belonging to the correct database. The final snapshot written when the original instance was destroyed is usually the best source. Copy it to a stable name before using it, so that it is not overwritten by a later destroy.

Removing snapshot_identifier Destroys the Database

Issue: A clean-up PR removing snapshot_identifier triggers a destroy and recreate, and the replacement is built empty.

Why This Happens: snapshot_identifier forces replacement, and the module does not set ignore_changes on it. Removing the attribute is a change like any other, so Terraform replaces the instance and creates the new one with no snapshot to restore from.

Solution: Advise teams to leave snapshot_identifier in place. While its value is unchanged it produces no plan diff and has no effect on the running database. Where it genuinely has to come out, clear it from Terraform state rather than through a pipeline apply.

Recommend deletion_protection = true on restored production databases. With it enabled AWS refuses the delete, so an accidental replacement fails in the pipeline rather than destroying the data.

Permission Denied Copying a Snapshot

Issue: A team reports an access denied error running aws rds copy-db-snapshot from their service pod, despite being able to create snapshots.

Why This Happens: The module’s IRSA policy grants rds:CopyDBSnapshot, but copying an encrypted snapshot also requires KMS permissions on the source key, which the policy does not grant. All Cloud Platform databases are encrypted.

Solution: Carry out the copy on the team’s behalf, and record the request so the gap in the IRSA policy can be addressed.

Metrics and Observations

Based on our testing:

Downtime: Expect ~26 minutes of database unavailability. This may vary depending on how much data is being restored.
Connection String: Remains unchanged after restoration (provided the restore is on the same database instance)
Storage Configuration: Inherits from the snapshot
Final Snapshots: May require manual cleanup to prevent pipeline failures
Status Progression: Follows predictable pattern: Deleting → Creating → Modifying → Rebooting → Backing-up → Modifying → Available

This page was last reviewed on 16 October 2026. It needs to be reviewed again on 16 April 2027 by the page owner #cloud-platform-notify .