Incident on 2020-04-15 Nginx/TLS

• Key events

  • Fault occurs 2020-04-15 07:15
  • Fault detected 2020-04-15 13:45
  • Incident declared 2020-04-15 14:39
  • Resolved 2020-04-15 15:09

• Status: Resolved at 2020-04-15 15:09 UTC

• Time to repair: 0h 30m

• Time to resolve: 5h 09m (during support hours 10:00-17:00)

• Identified: After an upgrade of the Nginx ingresses, support for legacy TLS was dropped.

• Impact:

  • IE11 users could not access any services running on the Cloud Platform
  • A few teams came forward with the issue :
  • LAA
  • Correspondence Tool
  • Prisoner Money

• Context:

  • After an upgrade of the Nginx Helm chart v1.24.0 to v1.35
  • The current version of Nginx has deprecated support for TLS 1.3 and lower
  • The issue was spotted on IE11 browsers
  • Timeline: https://docs.google.com/document/d/1SCf1WT82IlBYWozWN_FXZqL5h0KUcul_QAkxE84YDw0/edit?usp=sharing
  • Slack thread: https://mojdt.slack.com/archives/C57UPMZLY/p1586954463298700

• Resolution: The Nginx configuration was modified to enable TLSv1, TLSv1.1 and TLSv1.2