Skip to main content

Incident on 2023-01-05 - CircleCI Security Incident

  • Key events

    • First detected 2023-01-04 (Time TBC)
    • Incident declared: 2022-01-05 08:56
    • Repaired 2023-02-01 10:30
    • Resolved 2022-02-01 10:30

  • Time to repair: 673h 34m

  • Time to resolve: 673h 34m

  • Identified: CircleCI announced a security alert on 4th January 2023. Their advice was for any and all secrets stored in CircleCI to be rotated immediately as a cautionary measure.

  • Impact: Exposure of secrets stored within CircleCI for running various services associated with applications running on the Cloud Platform.

  • Context: Users of the Cloud Platform use CircleCI for CI/CD including deployments into the Cloud Platform. Access for CircleCI into the Cloud Platform is granted by generating a namespace enclosed service-account with required permission set by individual teams/users. As all service-account access/permissions were set based on user need, some service-accounts had access to all stored secrets within the namespace it was created in. As part of our preliminary investigation, it was also discovered service-accounts were shared between namespaces which exposed this incident wider than first anticipated. We made the decision that we need to rotate any and all secrets used within the cluster.

  • Resolution: Due to the unknown nature opf some of the secrets that may have been exposed a prioritised phased approach was created:

    • Phase 1 Rotate the secret access key all service-accounts named “circle-*” Rotate the secret access key for all other service-accounts Rotate all IRSA service-accounts
    • Phase 2 Rotate all AWS keys within namespaces which had a CircleCI service-account
    • Phase 3 Rotate all AWS keys within all other namespaces not in Phase 2
    • Phase 4 Create and publish guidance for users to rotate all other secrets within namespaces and AWS keys generated via a Cloud Platform Module
    • Phase 5 Clean up any other IAM/Access keys not managed via code within the AWS account.

Full detailed breakdown of events can be found in the postmortem notes.

  • Review actions:
    • Implement Trivy scanning for container vulnerability (Done)
    • Implement Secrets Manager
    • Propose more code to be managed in cloud-platform-environments repository
    • Look into a Terraform resource for CircleCI
    • Use IRSA instead of AWS Keys