Skip to main content

Grafana Dashboards

Kubernetes Number of Pods per Node

This dashboard was created to show the current number of pods per node in the cluster.

Dashboard Layout

Each box on the dashboard has the name of the node and the number of pods on the node - This number includes pods with a statuses such as “DeadlineExceeded” and completed".

The exception is the Max Pods per Node box. This is a constant number set on creation for the maximum allowed pods per node.

The current architecture does not allow instance group id to be viewed on the dashboard:

We currently have 5 instance groups:

  • Masters (one per each of the 3 availability zones in the London region)
  • Nodes
  • 2xlarge Nodes

As the dashboard is set in descending order, the last two boxes are normally from the 2xlarge Nodes group (2 instances), the next 3 boxes are normally the masters, and the rest are from the Nodes group.

You can run the following command to confirm this and get more information about a node:

kubectl describe node <node_name>

Troubleshooting

If a customer is reporting their dashboards are failing to load, this is usually due to a duplicate entry. You can see errors from the Grafana pod by running:

kubectl logs -n monitoring prometheus-operator-grafana-<pod-id> -f -c grafana

You’ll see an error similar to:

t=2021-12-03T13:37:35+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=sidecarProvider file=/tmp/dashboards/<MY-DASHBOARD>.json error="invalid character 'c' looking for beginning of value"

Identify that dashboard and fix the error in question.

This page was last reviewed on 18 August 2023. It needs to be reviewed again on 18 November 2023 by the page owner #cloud-platform .
This page was set to be reviewed before 18 November 2023 by the page owner #cloud-platform. This might mean the content is out of date.