Tips and Tricks
A collection of useful one-liners and code snippets.
Delete an RDS database snapshot
$ aws rds delete-db-snapshot --db-snapshot-identifier cloud-platform-2f8c3c1ceaf0a05d-finalsnapshot
Check the expiration date of the SSL certificate for a live domain
$ echo | openssl s_client -connect google.com:443 | openssl x509 -noout -enddate
depth=1 C = US, O = Google Trust Services, CN = Google Internet Authority G3
verify error:num=20:unable to get local issuer certificate
verify return:0
DONE
notAfter=Oct 21 18:23:00 2019 GMT
Delete a “stuck” resource
kubectl can freeze with resources in status “Terminating”, these workarounds can force a cleanup:
Identify what resources are still present in a namespace:
ns=my-app ; for api in `kubectl api-resources --verbs=list --namespaced -o name` ; do echo $api ; kubectl get $api -n $ns ; done
If a regular delete does not remove the resource, try
kubectl delete <resource> --grace-period=0 --force
and/or
kubectl patch <resource> -p '{"metadata":{"finalizers":[]}}' --type=merge
Where “resource” can be namespace, crd, pod etc
Filter namespaces by specific label or annotation
For the label cloud-platform.justice.gov.uk/is-production with value true
kubectl get ns -l=cloud-platform.justice.gov.uk/is-production=true
You can use awk to output specific column
kubectl get namespaces -l=cloud-platform.justice.gov.uk/is-production=true | awk '{print $1,$2}'
To filter namespaces by label and annotation and display as namespace / annotation format
kubectl get namespaces -l=cloud-platform.justice.gov.uk/is-production=true -ojson | jq -r '.items[] | .metadata.name + " / " + .metadata.annotations["cloud-platform.justice.gov.uk/source-code"] '
If you need to install jq, do brew install jq
Find all pods running on a specific worker node
kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=ip-172-20-80-62.eu-west-2.compute.internal
Count pods running on all nodes
Paste this into the search field on Prometheus:
max by(node) (max by(instance) (kubelet_running_pod_count{job="kubelet",metrics_path="/metrics"}) * on(instance) group_left(node) kubelet_node_name{job="kubelet",metrics_path="/metrics"})
Add more RSS feeds to #cloud-platform-rss channel
/feed subscribe <feed address>
There are more slash commands for rss feed which can be used to list, remove and help. Type /feed help in the #cloud-platform-rss` channel to get more details.
To validate your feed url, test using the w3c validator
To get the RSS feed url for github projects, use <github repo url>/releases.atom
To add your first feed to a new slack channel, follow steps provided by slack
Find files which don’t contain a particular string
for file in $(find * -name '*.erb'); do grep -q last_reviewed_on $file || echo $file; done
Get AWS EC2 instance information for a node
aws ec2 describe-instances --filters 'Name=private-dns-name,Values=ip-172-20-127-140.eu-west-2.compute.internal'
Create a one-off job to repeat a cronjob
kubectl -n concourse-main create job ds-update-orphaned-statefiles --from=cronjob/orphaned-terraform-statefiles
Hijack a concourse job container
fly -t manager hijack -j environments-live/apply-namespace-changes-live
You’ll be presented with a list of containers to choose from.
Help when manually deleting AWS resources
Occasionally you’ll need to manually delete AWS components. A neat trick is to use the AWS Tag Editor.
- Open the Tag Editor.
- Select “All Supported Resources” from the “Resource type” drop down.
- Click “Search Resources”.
- Use the filter to search for a tag, for example a cluster name:
cp-0909-1202.
You’ll be presented with all resources with your tag associated, click the link to navigate to each resource and delete as appropriate.
Find out why your namespace isn’t deleting
kubectl api-resources --verbs=list --namespaced -o name \
| xargs -n 1 kubectl get --show-kind --ignore-not-found -n <namespace-name>
It should show you the reason the API doesn’t want to delete. For example:
Error from server: conversion webhook for acme.cert-manager.io/v1alpha2, Kind=Order failed: Post https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s: service "cert-manager-webhook" not found
Error from server: conversion webhook for cert-manager.io/v1alpha2, Kind=CertificateRequest failed: Post https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s: service "cert-manager-webhook" not found
Error from server: conversion webhook for cert-manager.io/v1alpha2, Kind=Certificate failed: Post https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s: service "cert-manager-webhook" not found
Modsec false positives
When a genuine transaction causes a rule from the Core Rule Set to match in error it is described as a false positive. The OWASP rules are enabled on the ingress-controller level on the dedicated ModSec ingress-controller. The common problem with standard OWASP (CRS) is that it gives some false positive results. The SQL Injection rules in particular are very prone to over alerting, open (issue)[https://github.com/SpiderLabs/owasp-modsecurity-crs/issues/794] related to sql injection.
Every rule which is defined in the Core Rule Set (CRS) is identified by the ID. False positive rule ID’s can be disabled using the example below.
annotations:
nginx.ingress.kubernetes.io/enable-modsecurity: "true"
nginx.ingress.kubernetes.io/modsecurity-snippet: |
SecRuleEngine On
SecRuleRemoveById 111111
Some of the common issues and rules are given here
Note: Turning off these rules completely will reduce the effectiveness of ModSecurity, so users should not use same SecRuleRemoveById’s across applications, and before turning off any rules, users should make sure the exceptions apply to their particular use case.