Skip to main content

Incident on 2021-09-30 - SSL Certificate Issue in browsers

  • Key events

    • First detected 2021-09-30 15:31
    • Repaired 2021-10-01 10:29
    • Incident declared 2021-09-30 17:26
    • Resolved 2021-10-01 13:09

  • Time to repair: 5h 3m

  • Time to resolve: 7h 43m

  • Identified: User reported that they are getting SSL certificate errors when browsing sites which are hosted on Cloud Platform

  • Impact:

    • 300 LAA caseworkers and thousands of DOM1 users using CP-based digital services if it was during office hours. They had Firefox as a fallback and no actual reports.
    • Public users - No reports.

  • Context:

  • Resolution:

    • The new certificate was pushed to DOM1 and Quantum machines by the engineers who have been contracted to manage these devices

  • Review actions:

    • How to get latest announcements/ releases of components used in CP stack? Ticket raised #3262
    • Can we use AWS Certificate Manager instead of Letsencrypt? Ticket raised #3263
    • How would the team escalate a major incident e.g. CP goes down. Runbook page here
    • How we can get visibility of ServiceNow service issues for CP-hosted services. Ticket raised 3264