Skip to main content

Switch ingress to the new nginx-ingress-controller v1.2

Introduction

An ingress-controller is a specialised load balancer for Kubernetes. The Cloud Platform use the nginx-ingress-controller to accept traffic from outside the Kubernetes platform and load balance it to pods running in Namespaces.

Currently all service teams are using 1 of 2 ingress-controllers using one of the following annotations:

  • kubernetes.io/ingress.class: nginx - This is the default ingress-controller
  • kubernetes.io/ingress.class: modsec01 - This ingress-controller has the modsec WAF enabled.

These ingress-controllers are running on version v0.47.0

Unfortunately, there is a breaking change in the upgrade path of the nginx-ingress-controller to v1.0 and above. This would require all service teams to make changes to all ingress manifests at the same time as the upgrade.

To avoid this, the Cloud Platform team have released a new set of nginx-ingress-controllers (v1.2) alongside the current nginx-ingress-controllers (v0.47.0).

Service teams using the Cloud Platform are asked to switch to the new releases of the nginx-ingress-controller(v1.2).

Pre-requisite - Ingress API Version

This is a critical change/check all users should complete before switching to the new nginx-ingress-controller(v1.2).

This change is also in preparation for the upgrade of the Cloud-Platform to Kubernetes 1.22 which has deprecated all beta versions of the Ingress API.

Please follow the Deprecated Ingress API document on how to check and if required, change to the stable Ingress API version.

Switch to the new nginx-ingress-controller

IMPORTANT: There may be downtime of up to 10 minutes while switching to the new ingress controller due to DNS propagation. Please test in non-production environments before making the change on your production applications.

Alternatively you can follow the steps later in this document to switch to the new nginx-ingress-controller(v1.2) without downtime.

There are 2 releases of the new nginx-ingress-controller(v1.2). They can be selected using the new IngressClassName field:

  • ingressClassName: default - This is the default ingress-controller
  • ingressClassName: modsec - This ingress-controller has the modsec WAF enabled.

To switch to the new nginx-ingress-controller(v1.2):

  • Remove the kubernetes.io/ingress.class annotation
  • Add the ingressClassName field under spec (using the default or modsec value)

The changes should look like the below:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: helloworld
  annotations:
    external-dns.alpha.kubernetes.io/set-identifier: helloworld-mynamespace-green
    external-dns.alpha.kubernetes.io/aws-weight: "100"
spec:
  ingressClassName: default OR modsec
  tls:
  - hosts:
    - helloworld-app.apps.live.cloud-platform.service.justice.gov.uk
  rules:
  - host: helloworld-app.apps.live.cloud-platform.service.justice.gov.uk
    http:
      paths:
      - path: /
        pathType: ImplementationSpecific
        backend:
          service:
            name: helloworld
            port:
              number: 4567

Switch to the new ingress controller with zero downtime

The following steps describe how to switch ingress controllers without downtime. We strongly recommend teams test these steps on their non-production environments before attempting on production.

Step 1 - Changes to your current ingress resource

You must set external-dns.alpha.kubernetes.io/aws-weight value to “0” for the current ingress. This does not mean traffic will stop, 0 is a valid value for a single ingress.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: helloworld
  annotations:
    kubernetes.io/ingress.class: nginx
    external-dns.alpha.kubernetes.io/set-identifier: helloworld-mynamespace-green
    external-dns.alpha.kubernetes.io/aws-weight: "0"
spec:
  tls:
  - hosts:
    - helloworld-app.apps.live.cloud-platform.service.justice.gov.uk
  rules:
  - host: helloworld-app.apps.live.cloud-platform.service.justice.gov.uk
    http:
      paths:
      - path: /
        pathType: ImplementationSpecific
        backend:
          service:
            name: helloworld
            port:
              number: 4567

Step 2 - Create a new ingress resource

Create a new Ingress with a different ingress name to the existing one with the following changes:

  1. Create a new ingress name different to the existing one:

    name: helloworld-new

  2. Update set-identifier annotation to match with the new ingress name (-green remains the same):

    external-dns.alpha.kubernetes.io/set-identifier: helloworld-new-mynamespace-green

  3. Update aws-weight annotation to “100”:

    external-dns.alpha.kubernetes.io/aws-weight: "100"

  4. Remove the kubernetes.io/ingress.class annotation.

  5. Use the new ingress controller:

    Add ingressClassName under spec, using the value of default or modsec

The new manifest after changes will look like:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: helloworld-new
  annotations:
    external-dns.alpha.kubernetes.io/set-identifier: helloworld-new-mynamespace-green
    external-dns.alpha.kubernetes.io/aws-weight: "100"
spec:
  ingressClassName: default OR modsec
  tls:
  - hosts:
    - helloworld-app.apps.live.cloud-platform.service.justice.gov.uk
  rules:
  - host: helloworld-app.apps.live.cloud-platform.service.justice.gov.uk
    http:
      paths:
      - path: /
        pathType: ImplementationSpecific
        backend:
          service:
            name: helloworld
            port:
              number: 4567

To apply the changes to the ingress, as a quick alternative to running your application’s CI/CD pipeline, you can apply them directly:

kubectl -n <namespace-name> apply -f ingress-new.yaml

Verify traffic sent to the new ingress

After you have applied new ingress with external-dns.alpha.kubernetes.io/aws-weight value set to “100”, you can verify traffic is being sent to the new ingress using grafana

Get the “hostname” used in your ingress for your namespace

  kubectl get ingress <ingress-name> -n <namespace-name> -o json | jq -r '.spec.rules[].host'

Using the “hostname” from the above command, make a cURL call every 5 seconds to send traffic to your service

  while sleep 5; do curl -I https://<host-name>; done

To view your ingress traffic, login to grafana-live, in Kubernetes-ingress traffic dashboard, select your ingress-name and namespace-name from the drop-down and do a search, you will see traffic for your new ingress.

Your new ingress is successfully switched to the new ingress controller.

Step 3 - Cleanup old ingress

Once you have successfully switched to the new ingress controller, you can tidy up by deleting the old ingress and updating your CI pipeline/job with new ingress manifest that deploys to the cluster.

Getting help

If you have any questions, please contact us on #ask-cloud-platform Slack channel.

This page was last reviewed on 26 May 2022. It needs to be reviewed again on 26 August 2022 by the page owner #cloud-platform .
This page was set to be reviewed before 26 August 2022 by the page owner #cloud-platform. This might mean the content is out of date.