Skip to main content

Provisioning EKS clusters

IMPORTANT: All cluster’s names must be globally unique. Each one creates a unique Route53 hostzone which is unique to the name

You can create a new EKS test cluster using the cluster build pipeline.

Alternatively, if you want to create a cluster manually, follow the steps below.

Pre-requisites

  • Make sure you have access to Cloud Platform AWS account
  • Set your environment variables (you might also want to include AWS_* envs):
  • You have an AWS CLI profile moj-cp, with suitable credentials
    export AWS_PROFILE=moj-cp

Environment Variables

See the file example.env.create-cluster in the infrastructure repo.

You can get the auth0 values from the terraform-provider-auth0 application on justice-cloud-platform - auth0.

or you can export these env vars in your shell:

export AUTH0_DOMAIN=
export AUTH0_CLIENT_ID=
export AUTH0_CLIENT_SECRET=

Provisioning an EKS cluster with the cloud-platform CLI

  • In your terminal, navigate to the root of the cloud-platform-infrastructure repository.

  • Ensure your local git-crypted files are decryped with git-crypt unlock

  • Execute the cloud-platform command to create a new cluster:

cloud-platform cluster create --name <cluster-name>

NB: Your cluster name must be no more than 12 characters. Any longer, and some of the computed strings which include the cluster name will exceed their maximum allowed values. The error messages you get if this happens are unhelpful. In order to prevent this, the build script will fail immediately if you supply a name which is too long.

See our cluster naming policy for information on how to choose a suitable name for your cluster.

By default, the script will create a small cluster. This means the worker EC2 instances will be less powerful machine types than in our production cluster.

You can see more options to use when creating the cluster by running:

cloud-platform cluster create --help

The script takes around 30 minutes to execute. At the end, you should see output like this:

2022-05-04 12:10:27 kubectl cluster-info
Kubernetes control plane is running at https://B04C5FC7828A0515457E141A9610815D.sk1.eu-west-2.eks.amazonaws.com
CoreDNS is running at https://B04C5FC7828A0515457E141A9610815D.sk1.eu-west-2.eks.amazonaws.com

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

The section before kubectl cluster-info is the cluster integration tests, which run at the end of the build process. The number of tests will change, so the output will vary from what is shown here.

Alternatively, if you need more control over the test cluster parameters, or you just prefer to do it manually, the rest of this document describes the process.

Manually provisioning a cluster

1. VPC

We need to create a VPC to deploy the cluster, VPCs are deployed using terraform code under cloud-platform-infrastructure/terraform/aws-accounts/cloud-platform-aws/vpc folder:

cd cloud-platform-infrastructure/terraform/aws-acounts/cloud-platform-aws/vpc
terraform init
terraform workspace new <WorkspaceName>
terraform apply

You should be able to see your new VPC (called WorkspaceName) inside the AWS Console. Check it before jumping to the next step.

NOTE: It is important that you use the same workspace name at each terraform level during this process. Please pick a workspace name that will make your cluster easily identifiable.

2. Creating EKS cluster

Now it’s time to provision the cluster itself, we change directory to terraform/aws-accounts/cloud-platform-aws/vpc/eks and follow very similar terraform workflow:

If you are running on m1 mac then you may experience this error:

â•·
│ Error: Incompatible provider version
│
│ Provider registry.terraform.io/hashicorp/template v2.2.0 does not have a package available for your current platform, darwin_arm64.
│
│ Provider releases are separate from Terraform CLI releases, so not all providers are available for all platforms. Other versions of this provider may have different platforms supported.

This is because the template provider was deprecated and archived before the existence of m1 (darwin_arm64) releases. To overcome this error we must compile the provider ourselves for your Apple Silicon. Luckily, there’s this nifty cli tool that will download and compile the provider for us:

brew install kreuzwerker/taps/m1-terraform-provider-helper
m1-terraform-provider-helper activate
m1-terraform-provider-helper install hashicorp/template -v v2.2.0

Now we can continue with our terraform commands:

terraform init
terraform workspace new <WorkspaceName>
git-crypt unlock
terraform apply

Once terraform finishes you should be able to generate your kubeadmin file (used by kubectl to establish a connection to the cluster) using aws-cli in the following way:

aws eks --region eu-west-2 update-kubeconfig --name mogaal-eks

NOTE: Replace “mogaal-eks” with the cluster name (workspace name)

Final check for this step consists in cluster connection

$ kubectl cluster-info
Kubernetes master is running at https://B04C5FC7828A0515457E141A9610815D.sk1.eu-west-2.eks.amazonaws.com
CoreDNS is running at https://B04C5FC7828A0515457E141A9610815D.sk1.eu-west-2.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

3. Deploy core

Core installation refers to a set of components that need to deployed to the cluster before other components are installed. Things like networking, policy engines, monitoring etc. We need to have the KUBECONFIG variable exported and make sure you have access to the cluster. To deploy core we go to terraform/aws-accounts/cloud-platform-aws/vpc/eks/core and follow exactly the same terraform workflow:

terraform init
terraform workspace new <WorkspaceName>
terraform apply

4. Deploy components

Components installation is the last step, for that we need to have the KUBECONFIG variable exported and make sure you have access to the cluster. To deploy components we go to terraform/aws-accounts/cloud-platform-aws/vpc/eks/core/components and follow exactly the same terraform workflow:

terraform init
terraform workspace new <WorkspaceName>
terraform apply

Deleting your test cluster

Refer to this runbook for various methods of deleting your cluster.

Provisioning a custom cluster

If you’re working on a task that involves changing/adding cluster configuration or components, you can have Concourse execute a clean cluster build for you based off your working infrastructure branch. Refer to the instructions detailed in the Concourse repository pipelines README for the steps to follow.

Once you’re done with your customised cluster, you can pass your infrastructure branch name to the branch flag of the CLI delete-cluster command.

This page was last reviewed on 8 August 2024. It needs to be reviewed again on 8 February 2025 by the page owner #cloud-platform .
This page was set to be reviewed before 8 February 2025 by the page owner #cloud-platform. This might mean the content is out of date.