Grafana Dashboards
Kubernetes Number of Pods per Node
This dashboard was created to show the current number of pods per node in the cluster.
Dashboard Layout
Each box on the dashboard has the name of the node and the number of pods on the node - This number includes pods with a statuses such as “DeadlineExceeded” and completed".
The exception is the Max Pods per Node box. This is a constant number set on creation for the maximum allowed pods per node.
The current architecture does not allow instance group id to be viewed on the dashboard:
We currently have 5 instance groups:
- Masters (one per each of the 3 availability zones in the London region)
- Nodes
- 2xlarge Nodes
As the dashboard is set in descending order, the last two boxes are normally from the 2xlarge Nodes group (2 instances), the next 3 boxes are normally the masters, and the rest are from the Nodes group.
You can run the following command to confirm this and get more information about a node:
kubectl describe node <node_name>
Troubleshooting
If a customer is reporting their dashboards are failing to load, this is usually due to a duplicate entry. You can see errors from the Grafana pod by running:
kubectl logs -n monitoring prometheus-operator-grafana-<pod-id> -f -c grafana
You’ll see an error similar to:
t=2021-12-03T13:37:35+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=sidecarProvider file=/tmp/dashboards/<MY-DASHBOARD>.json error="invalid character 'c' looking for beginning of value"
Identify that dashboard and fix the error in question.