Problem

When setting up a Kubernetes cluster on EKS for the first time, you’ll likely encounter a pod limit issue due to node IP restrictions!
This limit depends on the instance type and the number of ENIs it can handle (+ENIs = +IPs). However, if you need to use smaller instances (to reduce costs) and deploy many pods (with low resource consumption), it’s beneficial to increase this default limit.

Why?

The documentation for amazon-vpc-cni-k8s explains (ENI + IP limits per instance and calculations) how to determine the limit.

For instance, with a t3a.medium instance, you are limited to a maximum of 17 pods because:

• It supports up to 3 network interfaces (ENIs).
• Each ENI can have up to 6 IP addresses, but 1 is reserved for the instance itself.

The calculation is as follows:

• 2 ENIs available for pods, with 5 usable IP addresses per ENI.
• Total of (2 ENIs x 5 IPs) = 10 IP addresses for additional pods.

Including the primary IP, there are 17 total IP addresses: 1 for the instance and 16 for the pods.

Solution

The simplest and quickest solution is to install the amazon-vpc-cni-k8s plugin and configure it to enable prefix delegation for ENIs. This allows the addition of /28 ranges (14 usable IP addresses out of 16) to each ENI assigned to an instance. For a t3a.medium, this configuration increases the capacity to 42 pods based on the previous calculation.

When applying this configuration, IT IS NECESSARY to delete and recreate your node groups so they adopt the changes. EKS will then display the standard Kubernetes limit of 110 pods per node.

Installation and Configuration

1. Log in to the AWS Console

2. Select the EKS Cluster

3. Access the Add-ons Page: In the left-hand navigation panel, click on “Add-ons.” You’ll see a list of installed add-ons and an option to add a new one.

4. Add the VPC CNI Add-on: Click the “Add Add-on” button. Under the “Add-on” section, select Amazon VPC CNI. Choose the latest version or your desired version under “Add-on Version.” Click Next.

5. Configure Add-on Parameters:

   {
     "env": {
       "ENABLE_PREFIX_DELEGATION": "true",
       "MINIMUM_IP_TARGET": "5",
       "WARM_ENI_TARGET": "1",
       "WARM_IP_TARGET": "2"
     }
   }
   

   • ENABLE_PREFIX_DELEGATION: Enables prefix delegation in subnets, allowing efficient allocation of IP blocks for pods.
   • MINIMUM_IP_TARGET: Sets the minimum number of IP addresses that must remain available on the node for new pods.
   • WARM_ENI_TARGET: Specifies how many additional ENIs should be kept “warm” (ready for rapid IP allocation).
   • WARM_IP_TARGET: Defines how many extra IP addresses should be kept “warm” on each ENI for quick pod scaling (e.g., 2).

6. Review and Confirm

7. Verify the Installation: After adding the add-on, verify its proper installation and that the pods are running. If a node group was already created, it must be deleted and recreated to adopt the changes.

(Translated using ChatGPT)