Skip to main content

Searching logs in OpenSearch

Overview

Container Platform 3 (CP3) provides full-text log search through a shared Amazon OpenSearch Service domain. Each environment has one shared managed cluster — one for non-live, one for live — used by every business unit: each unit’s logs live in their own set of indexes, and fine-grained access control keeps a unit’s view to its own logs. You search and build visualisations through OpenSearch Dashboards, reached from an AWS-hosted OpenSearch user interface (OpenSearch UI) that you sign into with your MoJ identity through IAM Identity Center — the same single sign-on as the AWS access portal and Grafana.

This page is a developer and platform-engineer guide to how log search works and how to use it. The Terraform that implements it lives in modernisation-platform-environments under terraform/environments/cloud-platform (the root component, opensearch.tf), and the design is recorded in ADR-017 Logging and Log Management Architecture (see its OpenSearch section).

Status: this is a development proof of concept. The cluster runs in the cloud-platform development account only, on a public (internet-facing) endpoint locked down by access control and HTTPS. A move to a private, in-VPC endpoint is planned once VPN access exists. Log ingestion from clusters is still being rolled out, so not every unit’s logs are flowing yet.

How it works

Each environment’s shared, managed OpenSearch domain holds that environment’s logs. Business units are separated inside the cluster rather than by separate clusters per unit:

  • Per-unit indexes — each unit’s logs go to indexes named <business-unit>-<environment>-* (for example hmpps-non-live-*).
  • Fine-grained access control (FGAC) — a unit’s role is scoped to its own index pattern, so a user sees only their own logs.
  • Encryption and HTTPS — encryption at rest and in transit, TLS 1.3, audit logging to CloudWatch.

The domain runs OpenSearch 3.7, right-sized for the proof of concept. Why one shared cluster rather than one per unit (cost, operational overhead, and the isolation model) is set out in ADR-017.

flowchart TB
  subgraph ENV["One shared cluster per environment (non-live / live)"]
    OS["OpenSearch domain"]
    IDXA["hmpps-non-live-*"]
    IDXB["laa-non-live-*"]
    OS --- IDXA
    OS --- IDXB
  end
  UA["HMPPS engineer"]
  UB["LAA engineer"]
  UA -->|"sign in (SSO)"| OS
  UB -->|"sign in (SSO)"| OS
  OS -. "FGAC: HMPPS sees hmpps-* only" .-> IDXA
  OS -. "FGAC: LAA sees laa-* only" .-> IDXB

How logs get here

Your application writes to stdout; Fluent Bit on the cluster collects it and sends it to two places — CloudWatch Logs for recent/operational searches, and an S3 archive. Logs reach OpenSearch from that S3 archive, through an event-driven pipeline, so nothing is lost if the search cluster is briefly unavailable.

flowchart LR
  APP["Your app<br/>(pods, stdout)"]
  FB["Fluent Bit<br/>(on the cluster)"]
  S3["S3 log archive<br/>(durable copy)"]
  CW["CloudWatch Logs<br/>(recent, ~30 days)"]
  OS["OpenSearch<br/>(full-text search, older logs)"]
  APP --> FB
  FB --> CW
  FB --> S3
  S3 -->|event-driven pipeline| OS

Signing in

Access is through the OpenSearch UI application, not the domain’s own Dashboards address (that has no interactive sign-in).

  1. Open the OpenSearch UI application URL (published by the platform team).
  2. You are redirected to IAM Identity Center; sign in with your MoJ identity if you are not already signed in.
  3. You land in OpenSearch Dashboards.

Your access follows your Identity Center group membership — being in the right group is enough, there is nothing to request per person. Your group does need to be added to the application by the platform team first; once it is, everyone in that group can sign in.

Access model

Role Who What you can do
Read-only Every business unit, by default Search your unit’s logs; build and read visualisations and dashboards.
Elevated Units that opt in Read-only, plus safe self-service such as exporting search results.
Admin Platform team Full management of the cluster.

Two things follow:

  • You see only your own unit’s logs — your role is scoped to your unit’s <business-unit>-<environment>-* indexes.
  • The elevated role is opt-in per unit, and each unit maps its own teams to it, so onboarding an engineer needs no platform-team change.

The platform team defines and provisions the roles and manages the cluster; business units map their own teams and own index-level access within their own indexes.

Finding your logs

Create an index pattern for your unit — <business-unit>-<environment>-* — then use Discover to search and the visualisation tools to build charts and dashboards.

Programmatic access

For scripts and tools, call the domain’s REST API directly with an AWS-signed (SigV4) request using your platform-engineer credentials, rather than the browser UI. Browser sign-in is for interactive use only.

Troubleshooting

  • “You don’t have authorization to access dashboards” — your Identity Center group has not been added to the application yet. Contact the platform team with your group name so they can add it.
  • Signed in but no indexes — logs may not be flowing for your unit yet, or your index pattern does not match <business-unit>-<environment>-*.
  • Cannot reach the URL — use the OpenSearch UI application URL, not the cluster’s own Dashboards address.

For the cluster’s managed-component logs (Karpenter, EBS CSI, load balancer controller, VPC CNI), which go to CloudWatch rather than OpenSearch, see EKS Auto Mode component logs.

This page was last reviewed on 20 October 2026. It needs to be reviewed again on 20 April 2027 by the page owner #cloud-platform-notify .