Modernize Amazon EKS managed nodes with gp3

A time ago in 2020, AWS announced the EBS volume type gp3, with this introduction allows EC2 compute users including Kubernetes administrators to potentially benefit from its baseline performance of 3,000 IOPS on smaller volumes and cost savings of up to 20% compared to its predecessor gp2. In my opinion, this presents an great opportunity for workloads like EKS nodes that usually has small volume size, where stateful pod data are stored externally. By operating within the gp3 baseline configuration, performance improvements can be experienced at a reduced price.

There is a great blog post on how cluster administrators and developers can transition to the EBS CSI driver and gp3 to provision PVs. However, this article outlines an approach on how cluster administrators can utilize EBS gp3 volumes as their node’s ephemeral storage (the worker node’s root volume).

I humbly share that the opinions in this article are my own. It’s important to note that this article may not consider clusters with Karpenter.

Managed node group with custom Launch Template

For workloads on an existing managed node group, we can consider creating a new managed node group with a custom Launch Template and migrating the workloads into the new node group. Using a custom Launch Template provides greater flexibility and customization once the managed node group is deployed.

Creating a new managed node-group

When creating a managed node group, the AWS EKS console will create the resource with specifications you provide, along with using some of its default values automatically (e.g., EBS volume type) for common parameters. For the highest level of customization, you can deploy managed nodes using your own launch template and a custom AMI.

Please note — When using your own launch template, there are some configurations that needs to be either specified on the launch template or the AWS EKS configuration. For future customization, please may refer to the latest Amazon EKS user guide on Launch template support for the requirements of each of those two to achieve the desired managed node group configuration.

Walk-through

1. Navigate to the Amazon EC2 console to Create an EC2 Launch Template,
2. Under Storage (volumes), select Advance on the right corner side,
3. Add new volume, and specify the Device Name as (Specify custom value) where custom value is ‘/dev/xvda’ and Volume type as ‘gp3’.
4. Specify the Size (GiB),
5. Double-check other EBS configurations, including the baseline IOPS as 3,000 and Throughput as 125.
6. Optionally, specify the Key pair name under Key pair (login) — for worker node secure shell access; or specify the Security groups under Network settings for the instance or Security groups under Network interfaces (Add network interface), but not both — interface ‘eth0’ using specific security group.
7. Review the summary and Proceed to Create Launch template.
8. Use your recently created Launch Template to create a Managed Nodegroup in your EKS cluster — During the node group creation in the Amazon EKS console, after choosing the desired Node IAM Role,
9. Toggle the option to Use launch template — Please may give the refresh button a push if Launch template is not found.
10. Continue in configuring the EKS cluster node as desired (ie. Instance Type, Taints, Networking/Subnets).
11. Double-check the Summary and Create the Managed Node-group.

Once the first node group has been created, the node(s) should be using gp3. Check EBS volume type is gp3 with the desired size (EC2 Storage).

SampleCloudFormationTemplate, EC2LaunchTemplate

To simplify steps 1 to 7, the following AWS CloudFormation template can be used as a reference to create the EC2 Launch Template resource:

AWSTemplateFormatVersion: '2010-09-09'
Description: 'Create an EC2 Launch Template to use the baseline EBS gp3 as the root block volume'
 
Parameters:
  launchTemplateNameParameter:
    Type: String
    Default: 'my-eks-gp3-lt'
    Description: Enter a unique EC2 Launch Template name
  ebsVolumeSizeParameter:
    Type: Number
    Default: 20
    Description: The desired volume size in GiB of the EBS gp3 root volume
    MinValue: 1
    MaxValue: 500 # 16,384
 
Resources:
  eksLaunchTemplate:
    Type: 'AWS::EC2:LaunchTemplate'
    Properties:
      LaunchTemplateData:
        BlockDeviceMappings:
          - DeviceName: '/dev/xvda'
            Ebs:
              Iops: 3000
              Throughput: 125
              VolumeSize: !Ref ebsVolumeSizeParameter
              VolumeType: 'gp3'
      LaunchTemplateName: !Ref launchTemplateNameParameter
 
Outputs:
  myEksLaunchTemplate:
    Value: !Ref eksLaunchTemplate
    Description: Logical ID of the newly created EC2 Launch Template resource

Existing managed node-group

When we create a EKS manage node-group without specifying a custom launch template, an auto-generated launch template will be created by EKS to simplify the creation process. To reduce future operational headaches, it’s probably better to consider creating a new managed node-group. Preferably, consider whether a separate Launch template per each EKS managed node group is a manageable option if future customization may be required. While technically, it’s possible to apply a new Launch template version in order to use the desire EBS volume configuration for the nodes. Then, the changes could be executed via a rolling update to minimize the impact to the cluster.

Thoughts or suggestions?