Pushing logs to the SOC team
Cloud Platform logs from various sources are being pushed to the SOC team’s security monitoring solution Palo Alto Cortex XSIAM.
We are currently pushing the following logs:
- Cloudtrail logs
live-1VPC FlowLogs- Route 53 logs
liveEKS cluster logsliveApplication logs
1. Cloudtrail logs
Architecture
We have followed Palo Alto’s documentation on implementation to allow Cortex XSIAM to injest logs from Cloudtrail.
Cloudtrail logs are written to a S3 bucket. The implementation consist of enabling the S3 bucket to trigger event notifications to an SQS queue. An IAM user with access keys has been created to grant Cortex XSIAM to access the SQS queue and receives these messages. Cortex XSIAM then uses the references in these messages to retrieve the logs from the S3 bucket. The terraform code for this implementation is found in the cloud-platform-terraform-infrastructure repository.
2. live-1 VPC FlowLogs
Architecture
We have followed Palo Alto’s documentation on implementation to allow Cortex XSIAM to injest VPC flowlogs from S3.
This implementation follows the same architectural pattern as pushing CloudTrail logs above. VPC flowlogs are written to a S3 bucket and triggers event notifications to an SQS queue. Cortex XSIAM then uses the references in these messages to retrieve the logs from the S3 bucket. The same IAM user access keys are used here. The terraform code for this implementation is found in the cloud-platform-terraform-infrastructure repository.
3. Route53 logs
Architecture
We have followed Palo Alto’s documentation on implementation to allow Cortex XSIAM to injest Route53 from S3.
Since Route53 Resolver query logging is VPC bound, we are calling the Route53 logs module at the VPC layer within infrastructure and referencing the S3 bucket for the related SQS resources in the same way we do so for VPC FlowLogs, in the cloud-platform-terraform-infrastructure repository.
4. live EKS logs
Architecture
We have followed Palo Alto’s documentation on implementation to allow Cortex XSIAM to injest EKS cluster logs from Cloudwatch.
Our live EKS cluster is configured to send cluster Control Plane logs to CloudWatch. These include API server, audit, authenticator, controller manager, and scheduler logs.
We have set up AWS Firehose to stream these logs from CloudWatch to Cortex XSIAM. The terraform code for this implementation is found in firehose.tf in the cloud-platform-terraform-infrastructure repository.
5. live Application logs
Architecture
By default, Cloud Platform collects application logs via Fluentbit, then pushes them to OpenSearch. We have set up an additional output data pipeline in Fluentbit which also pushes these logs concurrently to a S3 bucket. From there, it follows the same architectural pattern as pushing CloudTrail logs above, where the logs that are written to the S3 bucket will trigger event notifications to an SQS queue. Cortex XSIAM then uses the references in these messages to retrieve the logs from the S3 bucket. The same IAM user access keys are used here. The terraform code for this implementation is found in the cloud-platform-terraform-infrastructure repository.
IAM User access keys rotation
We need put in a mechanism to periodically rotate the IAM User access keys created for Cortex XSIAM to receive the logs. Suggestion and issue for this has been raised.