Skip to main content

Incident on 2025-07-23 - Multiple secrets leaked to public

  • Key events

    • First detected 2025-07-23 15:35
    • Incident declared 2025-07-23 16:48
    • Repaired 2025-07-24 12:34
    • Resolved 2025-07-24 12:34

  • Identified:

Via Engineer

  • Impact:

A terraform plan was pushed to a branch within the Cloud Platform Environments repository. Terraform plans are binary files which can be unzipped exposing a Github Personal Access token and other credentials.

  • Context:

15:35 – Plan pushed to git branch

16:08 – Initial finding of secret leakage via human inspection

16:48 – More secrets found:

  • K8s secrets
  • Service account
  • RDS

16:59 – Incident declared in #cloud-platform

17:03 – Teams Meeting created

17:05 - looking through tokens to see which token could’ve been leaked

17:20 - Discovered $TOKEN$ is Tony’s Personal Github token for CP_GITHUB_ACTIONS_SECRETS_WRITER in 1pass. Confirmed owned by Tony with (curl -H "Authorization: token " https://api.github.com/user)

17:33 - Attempted to revoke token using GitHub API using the token to remove - Unsuccessful

17:40 - Mike attempted to revoke token using github api Personal credentials - Unsuccessful

17:56 - Rosie Brigham had access to see token as she has owner in organisation owner group. Promoted Julia to owner in group too. Token has been revoked but GitHub still recognises it. Could have only revoked session not the token.

18:04 - Github API token revoke endpoint worked - documentation is wrong, need to remove authentication line:

curl -L
-X POST
-H "Accept: application/vnd.github+json"
-H "X-GitHub-Api-Version: 2022-11-28"

https://api.github.com/credentials/revoke
-d '{"credentials":[""]} Confirmed revoked - curl -H "Authorization: token $TOKEN$" https://api.github.com/user { "message": "Bad credentials", "documentation_url": "https://docs.github.com/rest", "status": "401" }

18:23 - RDS and service account is being destroyed.

18:25 - Checking audit log to ensure no unexpected token uses.

18:33 - RDS and service account destroyed.

18:38 - Secret manager being deleted (empty)

18:44 - Deleted secret from Azure

19:00 – Slack channel created

19:08 - Git history purged

20:45 – Incident declared resolved.

24/07/25

12:34 – PR, plan file and commit removed from Github site entirely via Support ticket

  • Resolution:

The team and engineers from other teams worked closely to triage and remediate the credential leak. The Github Personal Access token was revoked and tested to ensure that access was no longer possible. Kubernetes service accounts were revoked and AWS resources were deleted.

  • Review actions:
  1. Speak to namespace owner
  2. Find where token was being used and what’s broken
  3. Nobody to use the token going forward
  4. Runbook/User Guide for revoking token using the API
  5. Can we search the audit logs without an admin and how?
  6. User guide on how to purge git history (inc github support step if required)
  7. Add *.out to gitignore
  8. Leads to review/audit Github token usage across Cloud Platform and plan migration to sustainable methods of authentication.
    1. Personal tokens in shared vaults in 1Password
    2. Tokens in Concourse
    3. Tokens in Cloud Platform Apps
    4. Tokens in Cloud Platform Github Actions
    5. Replace Sablubot with Cloud Platform controlled alternative
    6. Break glass access for Cloud Platform team members
    7. Add report to Runbook incident log.
    8. Spike usage of pre commit hooks
      • File size
      • Binary files