Module 08 — Managing Storage in Amazon EKS

TOPIC 1 Stateless vs stateful apps

The core question for storage design is: does your app need to remember data between restarts? Stateless apps can use ephemeral storage and be freely replaced. Stateful apps require that data outlive the container, the pod, and even the node.

Stateless

All replicas are interchangeable
Data is transient — can be rebuilt
Ephemeral storage is fine
Examples: web frontends, REST APIs, batch processors
Scale freely: kill/replace any replica

Stateful

Each replica has a unique identity
Data must persist across restarts
Persistent volume required
Examples: databases, message brokers, caches
Storage must match app design exactly

What are the three stateful requirements the course identifies?

1. Consistent access per instance — each pod accesses its own data, not shared data
2. Data persistence independent of container/node — data survives crashes and rescheduling
3. Storage suited for app design — block vs file vs object depending on the workload

Can you use ephemeral storage (emptyDir) for a stateful database pod?

No. An emptyDir volume is created when the pod is scheduled and deleted when the pod is removed — even if the pod restarts. All database data would be lost. Use a PersistentVolumeClaim backed by EBS or EFS instead.

What kind of storage is appropriate for a scratch/cache volume that doesn't need to persist?

Ephemeral volume (emptyDir). Declared in the pod spec under volumes: with emptyDir: {}. It is fast (can be backed by memory with medium: Memory), temporary, and requires no provisioning — perfect for caching or inter-container scratch space.

TOPIC 2 StatefulSets

A StatefulSet is like a Deployment, but for workloads that need stable identities. Each pod gets a persistent ordinal name (e.g., mysql-0, mysql-1) that never changes, and its own PersistentVolumeClaim that travels with it across reschedules.

Deployment

Pods are interchangeable
Names are random hashes
All pods share one PVC (or no PVC)
Parallel scale-up/scale-down
Any pod can be deleted/replaced

StatefulSet

Pods have unique, stable identities
Names: name-0, name-1…
Each pod gets its own PVC
Ordered deploy, scale, update
PVC retained even if pod is deleted

Name four use cases that call for a StatefulSet over a Deployment.

1. Stable network IDs — pod DNS name must be predictable (e.g., mysql-0.mysql-svc)
2. Persistent storage per pod — each replica owns its data
3. Ordered deployment/scaling — pod N must be Running before pod N+1 starts
4. Ordered rolling updates — updates proceed in reverse ordinal order

What happens to a StatefulSet pod's PVC when the pod is deleted?

The PVC is retained — it is NOT deleted with the pod. When the pod is rescheduled (same ordinal), Kubernetes re-attaches the same PVC. This guarantees data continuity across reschedules and node failures.

How does a StatefulSet pod name differ from a Deployment pod name?

StatefulSet: <name>-<ordinal> — e.g., mysql-0, mysql-1. The ordinal is stable; if mysql-1 is killed, its replacement is also named mysql-1.

Deployment: <name>-<replicaset-hash>-<random> — no stable identity.

TOPIC 3 Kubernetes storage objects

Four objects form the Kubernetes storage model. They separate the concerns of who provides the storage, who requests it, and how it's provisioned.

Animation · PersistentVolume lifecycle — from creation to pod mount

Object	What it represents	Created by	YAML kind
emptyDir	Ephemeral scratch volume; lives and dies with the pod	Declared inline in pod spec	`volumes[].emptyDir`
PersistentVolume (PV)	Cluster-level resource representing actual storage (EBS vol, EFS mount, etc.)	Admin or dynamically by StorageClass	`PersistentVolume`
PersistentVolumeClaim (PVC)	User request for storage (capacity, access mode). Binds to a matching PV.	Developer / pod spec	`PersistentVolumeClaim`
StorageClass	Defines a "tier" of storage. Enables dynamic PV provisioning on demand.	Cluster admin	`StorageClass`

# StorageClass — defines how EBS volumes are provisioned
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-gp3
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer  # wait until pod is scheduled
allowVolumeExpansion: true
parameters:
  type: gp3
  encrypted: "true"
    

What is the difference between a PV and a PVC?

PV (PersistentVolume) is a cluster-level resource that represents actual storage — an EBS volume, EFS mount point, etc. It is provisioned by an admin or dynamically.

PVC (PersistentVolumeClaim) is a request for storage by a user — specifying size and access mode. Kubernetes binds the PVC to a suitable PV automatically.

What does volumeBindingMode: WaitForFirstConsumer do?

It tells Kubernetes to delay PV creation until a pod using the PVC is actually scheduled to a node. This ensures the EBS volume is created in the same Availability Zone as the pod — without this, the volume might be created in a different AZ, causing the pod to be unable to mount it.

Which object should an app developer reference in a pod manifest — PV or PVC?

PVC only. Developers use PVCs in pod specs (volumes[].persistentVolumeClaim.claimName). PVs are an admin/infrastructure concern. This separation of concerns means developers don't need to know the underlying storage type or backend.

What are the three access modes for a PersistentVolume?

ReadWriteOnce (RWO) — one node mounts read-write (EBS)
ReadOnlyMany (ROX) — many nodes mount read-only
ReadWriteMany (RWX) — many nodes mount read-write (EFS)

Note: EBS only supports RWO. EFS supports RWX.

What does allowVolumeExpansion: true in a StorageClass enable?

It allows users to resize an existing PVC by editing its spec.resources.requests.storage to a larger value. The EBS CSI driver will expand the underlying EBS volume without needing to delete and recreate the PVC or the pod.

TOPIC 4 AWS storage options — EBS, EFS, FSx

EKS supports three families of AWS persistent storage. Choosing the right one depends on whether your workload needs block storage, shared file access, or specialized file system features.

Animation · EBS (ReadWriteOnce) vs EFS (ReadWriteMany)

AWS EBS

Elastic Block Store

Block storage. One EC2 instance at a time (RWO). SSD + HDD tiers. Independent lifecycle from EC2. Dynamic resize.

AWS EFS

Elastic File System

Shared NFS (v4). Multiple pods/nodes simultaneously (RWX). Elastic — scales automatically. Fargate compatible.

AWS FSx

Amazon FSx

Four types: Lustre, Windows File Server, NetApp ONTAP, OpenZFS. For specialized HPC and enterprise file system needs.

Feature	EBS	EFS
Type	Block storage (SSD/HDD)	Managed NFS file system
Access mode	ReadWriteOnce (one node)	ReadWriteMany (many pods/nodes)
Protocol	Block device	NFSv4
Fargate support	✗ Not supported	✓ Supported
Capacity	Fixed at creation (expandable)	Elastic — grows/shrinks automatically
Best for	Databases, single-replica stateful apps	Shared content, ML training datasets, CMS

EBS volume types

Type	Category	Use case
`gp2 / gp3`	General purpose SSD	Most workloads — default choice
`io1 / io2`	Provisioned IOPS SSD	High-performance databases (io2 includes Block Express)
`st1`	Throughput HDD	Big data, log processing, large sequential reads
`sc1`	Cold HDD	Infrequently accessed data, lowest cost

You have a Fargate pod that needs persistent storage. Which AWS service can you use?

Amazon EFS only. EBS is not supported on Fargate pods — EBS requires attaching to an EC2 instance. EFS uses NFS over the network and is fully compatible with Fargate. You must use the EFS CSI driver and mount an EFS access point.

What is the default recommended EBS volume type for EKS StorageClasses?

gp3 — the newest general-purpose SSD type. It provides better baseline performance than gp2 and lets you provision IOPS and throughput independently. AWS recommends gp3 as the default in new StorageClass configurations (parameters.type: gp3).

Multiple replicas of a machine-learning training job need to read the same dataset simultaneously. Which storage?

Amazon EFS — because it supports ReadWriteMany, allowing multiple pods across multiple nodes to mount and read the same file system concurrently. EBS (RWO) can only be mounted by one node at a time.

What are the four FSx types supported by EKS?

1. FSx for Lustre — HPC, ML, high-throughput parallel access
2. FSx for Windows File Server — SMB/NTFS, Windows workloads
3. FSx for NetApp ONTAP — enterprise NAS features, multi-protocol
4. FSx for OpenZFS — ZFS features, low-latency NFS access

TOPIC 5 CSI drivers

The Container Storage Interface (CSI) is a standard API that lets Kubernetes communicate with any storage system through a consistent interface — without needing storage-specific code inside Kubernetes itself. Each storage type installs its own CSI driver as an EKS add-on.

Animation · How the EBS CSI driver provisions a volume

EBS CSI Driver

Provisioner: ebs.csi.aws.com
EKS managed add-on
Supports gp2, gp3, io1, io2, st1, sc1
Requires IRSA for IAM permissions
Supports volume snapshots
Supports volume expansion

EFS CSI Driver

Provisioner: efs.csi.aws.com
EKS managed add-on
Supports ReadWriteMany across pods
Works on Fargate pods
Requires IRSA for IAM permissions
Uses EFS access points per PVC

What is the Container Storage Interface (CSI)?

CSI is a standardized interface between container orchestrators (Kubernetes) and storage providers. Before CSI, storage code was baked into the Kubernetes source. Now, each storage vendor ships their own CSI driver — installed separately — and Kubernetes calls it through the standard API.

What is the provisioner string for the EBS CSI driver?

ebs.csi.aws.com

Used in the StorageClass.provisioner field. Similarly for EFS: efs.csi.aws.com. These strings tell the dynamic provisioner which CSI driver to call when a PVC is created.

How does the EBS CSI driver authenticate to AWS APIs?

Via IRSA (IAM Roles for Service Accounts). The EBS CSI driver runs as a pod with a Kubernetes service account annotated with an IAM role ARN. The role has permissions to call EC2 APIs (ec2:CreateVolume, ec2:AttachVolume, etc.). No long-lived credentials needed.

How do you install the EBS CSI driver in EKS?

Install it as an EKS managed add-on: via the AWS Console, eksctl, or the AWS CLI (aws eks create-addon --addon-name aws-ebs-csi-driver). EKS handles version upgrades. You must also configure an IRSA role for the driver's service account before or after installation.

TOPIC 6 Storage scenario decision guide

The course gives four canonical scenarios that map directly to storage choices. Know these patterns — they are common exam questions.

Scenario	Storage type	Why
Stateless multi-replica app needs scratch space	Volume (emptyDir)	No persistence needed; data is transient per pod
Stateful replicas need shared data access	Amazon EFS	ReadWriteMany; all replicas mount the same file system
Each stateful replica needs independent storage with expansion	Amazon EBS	ReadWriteOnce per pod; supports dynamic resizing
Fargate pods need persistent storage	Amazon EFS	EBS is not supported on Fargate; EFS NFS works over network

App developer guidelines

Use PVC in manifests (not PV directly)
Specify the correct StorageClass
Adhere to namespace storage quotas
Request minimum storage needed

Cluster admin guidelines

Define and maintain StorageClasses
Set a default StorageClass for the cluster
Monitor unbound PVCs (misconfigured requests)
Define and enforce storage quotas per namespace

KC scenario: a MySQL StatefulSet has 3 replicas. Each needs its own 20Gi database volume that survives pod restarts and can grow. What storage?

Amazon EBS (gp3) with allowVolumeExpansion: true in the StorageClass.
Each StatefulSet replica gets its own PVC via volumeClaimTemplates → one EBS volume per pod, independent lifecycle, expandable. EFS would be shared across all replicas (wrong for database isolation).

KC scenario: a content management system has 10 web server pods that all need to read/write shared media files. What storage?

Amazon EFS — ReadWriteMany access mode lets all 10 pods mount the same EFS file system simultaneously. EBS (RWO) can only be mounted by one node at a time and would not allow shared writes across replicas.

TOPIC 7 Kubernetes Secrets

Kubernetes Secrets store sensitive data — passwords, API tokens, SSH keys, TLS certificates. The most important thing to know: Kubernetes Secrets are NOT encrypted by default — they are only base64 encoded, which is trivially reversible.

What Secrets store

Database credentials
API tokens & OAuth keys
SSH private keys
TLS certificates
Docker registry credentials (imagePullSecret)

How Secrets work by default

Values are base64 encoded (not encrypted)
Stored in etcd (on encrypted volume)
Injected into pods as env vars or mounted files
Optional: enable KMS envelope encryption for etcd at rest
base64 ≠ encryption — anyone with etcd access can decode

KC — What is the default behavior for Kubernetes Secrets?
A. Plaintext in etcd B. AES-256 encrypted C. base64 encoded, stored in etcd on encrypted volume D. Stored in AWS KMS

✓ C — base64 encoded, stored in etcd on encrypted volume

The etcd volume is encrypted (disk-level), but the secret values themselves are only base64 encoded — not encrypted. KMS envelope encryption (option D) is an optional additional layer you can enable, but it is not the default.

Why is "base64 encoded" NOT the same as "encrypted"?

Base64 is an encoding scheme, not an encryption scheme. It has no key and is trivially reversible: echo "dXNlcjpwYXNz" | base64 -d → user:pass. Anyone who can read the etcd data or the Kubernetes API can immediately decode any Secret value without any key material.

How can you add true encryption for Kubernetes Secrets at rest?

Enable KMS envelope encryption for etcd. In EKS, configure a EncryptionConfig with an AWS KMS key ARN when creating the cluster. Kubernetes then encrypts secret values with a data key, and AWS KMS protects the envelope key. This adds a true encryption layer on top of the base64 encoding.

What are the two ways to expose a Kubernetes Secret to a pod?

1. Environment variable — env.valueFrom.secretKeyRef. Simple, but the value is visible in kubectl describe pod and in the process environment.
2. Mounted volume (file) — volumes[].secret.secretName + volumeMounts. The secret value appears as a file in the container; more secure and supports automatic rotation with ASCP.

TOPIC 8 AWS Secrets Manager + ASCP

The AWS Secrets and Configuration Provider (ASCP) bridges AWS Secrets Manager (and SSM Parameter Store) with Kubernetes pods. Secrets are stored and rotated in AWS — pods see them as mounted files or Kubernetes Secret objects, without manual syncing.

Animation · ASCP: how a pod gets a secret from AWS Secrets Manager

# SecretProviderClass — tells ASCP where the secret lives
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: my-aws-secret
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "MyDbPassword"
        objectType: "secretsmanager"
    

What ASCP provides

Mounts secrets as files in pod volumes
OR syncs to Kubernetes Secret objects
Supports automatic key rotation
Works with Secrets Manager and SSM Parameter Store
Uses IRSA — no IAM keys in the cluster

Components required

Secrets Store CSI Driver (secrets-store.csi.k8s.io)
ASCP provider plugin for AWS
IRSA role on the pod's service account
SecretProviderClass custom resource
Pod spec: volumes[].csi.driver: secrets-store.csi.k8s.io

What is ASCP and what problem does it solve?

ASCP (AWS Secrets & Configuration Provider) is a plugin for the Secrets Store CSI Driver that fetches secrets from AWS Secrets Manager or SSM Parameter Store and delivers them to pods as volume-mounted files or as Kubernetes Secret objects. It solves the problem of storing sensitive data in Kubernetes etcd — secrets live in AWS with proper encryption and rotation.

How does ASCP restrict which pods can access a specific AWS secret?

Via IRSA (IAM Roles for Service Accounts). The pod's service account is annotated with an IAM role ARN. The IAM role has a policy that permits secretsmanager:GetSecretValue for only specific secret ARNs. Only pods with that service account can retrieve those secrets — no other pod can.

What is a SecretProviderClass?

A SecretProviderClass is a custom Kubernetes resource (CRD) that defines which secrets to fetch from AWS. It specifies: the provider: aws, the secret names/paths in Secrets Manager or SSM, and optional sync-to-Kubernetes-Secret configuration. Each pod that needs the secrets references a SecretProviderClass by name in its volume spec.

What CSI driver name does the Secrets Store use?

secrets-store.csi.k8s.io

Referenced in the pod's volume spec under volumes[].csi.driver. This is separate from the EBS CSI driver (ebs.csi.aws.com) and EFS CSI driver (efs.csi.aws.com) — it is a general-purpose secrets bridge, not a storage driver.

What two delivery modes can ASCP use to expose a secret to a pod?

1. Volume mount (file) — the secret value appears as a file at a path in the container (e.g., /mnt/secrets/MyDbPassword). This is the primary ASCP delivery method.
2. Kubernetes Secret sync — ASCP can optionally create a Kubernetes Secret object that mirrors the AWS secret, so existing code that reads from env vars or K8s Secrets continues to work unchanged.

Does ASCP support automatic secret rotation?

Yes. ASCP periodically re-fetches secrets from Secrets Manager. When AWS Secrets Manager rotates a secret, ASCP will pick up the new value on its next refresh cycle and update the mounted file. The application can re-read the file without restarting. This is a significant advantage over baking secrets into Kubernetes Secret objects manually.

TOPIC 9 Knowledge checks (from the course)

Official course knowledge-check questions plus common exam traps for Module 08. Try to answer before flipping.

KC — What is the default behavior for a Kubernetes Secret?
A. Plaintext B. AES-256 encrypted C. base64 encoded, stored in etcd on encrypted volume D. AWS KMS encrypted

✓ C — base64 encoded, stored in etcd on encrypted volume.

The etcd disk is encrypted but the secret values are only base64 encoded — reversible without a key. KMS encryption (D) is an optional add-on, not the default.

Which Kubernetes workload controller gives each pod a stable, persistent identity?

StatefulSet. Each pod gets an ordinal name (name-0, name-1) and its own PersistentVolumeClaim that survives pod deletion. Deployments give pods random names and share (or forgo) PVCs.

You need persistent storage for a Fargate pod. Which AWS storage service must you use?

Amazon EFS. EBS requires EC2 instance attachment and is not available on Fargate. EFS uses NFS over the network, which works with Fargate pods. Ensure you use the EFS CSI driver and configure an EFS access point per PVC.

What does the ASCP use to ensure only specific pods can access a secret?

IRSA (IAM Roles for Service Accounts). The pod's Kubernetes service account is annotated with an IAM role. The IAM policy grants secretsmanager:GetSecretValue only for specific secret ARNs. Pods without that service account cannot retrieve the secret.

What is the access mode for Amazon EBS when used with Kubernetes?

ReadWriteOnce (RWO) — one node can mount the volume read-write at a time. EBS is block storage and must be attached to a single EC2 instance. If you need multiple pods to write simultaneously, use EFS (ReadWriteMany).

What is the provisioner name for the Amazon EBS CSI driver?

ebs.csi.aws.com — referenced in the StorageClass provisioner field.
EFS: efs.csi.aws.com
Secrets Store: secrets-store.csi.k8s.io

Lab 5 preview: what does Lab 5 exercise?

Lab 5 exercises the full PVC lifecycle: (1) create a PVC using pre-created StorageClasses, (2) assign the PVC to a pod, (3) write data through the mounted volume, (4) delete and recreate the pod to confirm data persists, (5) manage storage lifecycle — cleanup and verification.

💾 Module 08 — Managing Storage in Amazon EKS

What's in this module