OpenSearch modsec setup
We have introduced an OpenSearch dashboard which collects all modsec logs and has document level security enabled. This means users can only access the logs for the github team they are in see here for more details. With this feature in place users can self serve and access their own modsec logs. In the case of a rare error and logs aren’t flowing to OpenSearch, then you must use the instructions below to access modsec logs on behalf of the user.
Get an audit log from modsec (when fluent-bit is not pushing to OpenSearch)
In the event that audit logs have failed to ship to OpenSearch, you’ll need to fetch this information from the pod that generated the log.
How do I check the audit log
- An OpenSearch ingress event from the user. A request will come into the ask-cloud-platform channel asking something like:
Good afternoon, could I ask for the detailed logs for this block from ModSecurity, please?
https://app-logs.cloud-platform.service.justice.gov.uk/_dashboards/app/data-explorer/discover#?[SOME-SEARCH-QUERY.....]
(I need to find out which rules triggered the block)
- The OpenSearch event above should provide you with the following key information
kubernetes.pod_name (optional): This will allow you to hone in on the modsec ingress correct pod.
unique_id: This is a hash of the event in question, e.g. 16494071776.005464, and can be located in the log entry.
- Kubectl access to the live cluster and access to the
ingress-controllersnamespace.
Perform a search for the unique-id (obtained from the OpenSearch entry)
# assuming the event id is 16494071776.005464
$ event="16494071776.005464"
$ for k in `kubectl -n ingress-controllers get pods | grep modsec01-nx-controller | awk '{print $1}'` ; do echo $k ; kubectl -n ingress-controllers exec -ti $k -- bash -c "cat /var/log/modsec_audit.log ; grep -r $event *" && break ; done
Which should return the event.
If you want to copy the filtered events of a particular search pattern to a folder and download to the local machine to extract the data, use the commands in bash
cd /var/log/
cat modsec_audit.log | grep -r "String or hostname to search" > list.txt
mkdir copy
cp list.txt ./copy
This should copy the filtered folders and files into the copy folder
Exit the modsec pod and from your machine, enter the below command
kubectl -n ingress-controllers cp <modsec-pod-name>:/var/log/copy .
If you want to send the log data to a user, make sure you filter the data specific to that application and encrypt the file. From mac, you can use gpgkeychain to encrypt the file using the receiver’s public key before sending it to the user.