ops0ops0

Storage

Manage Kubernetes storage resources including PersistentVolumes, PersistentVolumeClaims, StorageClasses, and ResourceQuotas for persistent data storage.

Storage Resources

PersistentVolumes (PV)
Cluster-wide storage resources provisioned by admin
PersistentVolumeClaims (PVC)
User requests for storage with specific size and access mode
StorageClasses
Storage provisioning templates for dynamic volume creation
ResourceQuotas
Namespace-level storage consumption limits

PersistentVolumes (PV)

Cluster-wide storage resources that exist independently of pods and persist beyond pod lifecycle.

Viewing PersistentVolumes

ColumnDescription
NamePV name
StatusAvailable, Bound, Released, Failed
ClaimPVC using this PV (if bound)
Storage ClassStorageClass used for provisioning
CapacityStorage size (e.g., 100Gi)
Access ModesReadWriteOnce, ReadOnlyMany, ReadWriteMany
Reclaim PolicyRetain, Delete, or Recycle
AgeTime since creation

PV Status Lifecycle

StatusMeaning
AvailablePV is free and not yet bound to a PVC
BoundPV is bound to a PVC and in use
ReleasedPVC was deleted but PV not yet reclaimed
FailedAutomatic reclamation failed

PV Details

Click a PV to view:

  • Capacity: Storage size in Gi
  • Access Modes:
    • ReadWriteOnce (RWO) - Single node read/write
    • ReadOnlyMany (ROX) - Multiple nodes read-only
    • ReadWriteMany (RWX) - Multiple nodes read/write
  • Reclaim Policy: What happens when PVC is deleted
  • Storage Class: Provisioner used to create PV
  • Volume Source: AWS EBS, GCE PD, NFS, etc.
  • Node Affinity: Node constraints for volume attachment

Reclaim Policies

PolicyBehavior
RetainPV remains after PVC deletion (manual cleanup needed)
DeletePV and underlying storage deleted automatically
RecycleData scrubbed, PV becomes available again (deprecated)

PersistentVolumeClaims (PVC)

User requests for storage that bind to PersistentVolumes or trigger dynamic provisioning.

Viewing PersistentVolumeClaims

ColumnDescription
NamePVC name
NamespaceKubernetes namespace
StatusPending, Bound, Lost
VolumeBound PV name
CapacityRequested storage size
Access ModesRWO, ROX, RWX
Storage ClassStorageClass for dynamic provisioning
AgeTime since creation

PVC Status

StatusMeaningAction Needed
BoundPVC bound to PV, ready to useNone
PendingWaiting for PV or provisioningCheck StorageClass, capacity, node availability
LostBound PV deleted or unavailableRecreate PV or restore volume

PVC Binding Process

PVC Created

User creates PVC requesting specific capacity and access mode.

Kubernetes looks for available PV matching requirements.

Bind or Provision

  • Static Provisioning: Bind to existing PV
  • Dynamic Provisioning: Create new PV using StorageClass

Bound

PVC status changes to Bound, ready for pod to use.

Using PVCs in Pods

PVCs are mounted into pods to provide persistent storage:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: nginx
    volumeMounts:
    - name: html
      mountPath: /usr/share/nginx/html
  volumes:
  - name: html
    persistentVolumeClaim:
      claimName: nginx-pvc

Expanding PVCs

Some StorageClasses allow volume expansion:

Check StorageClass

Verify allowVolumeExpansion: true is set.

Edit PVC

Increase the spec.resources.requests.storage value.

Expansion Triggers

Kubernetes expands the underlying volume.

Wait for Completion

PVC status shows FileSystemResizePending until complete.

Verify New Size

Check pod sees expanded storage capacity.

Expansion Limitations
  • Volume expansion is one-way only (cannot shrink)
  • Some volume types require pod restart to recognize new size
  • Not all StorageClasses support expansion

StorageClasses

Templates for dynamic PersistentVolume provisioning with cloud provider-specific parameters.

Viewing StorageClasses

ColumnDescription
NameStorageClass name
ProvisionerVolume plugin (ebs.csi.aws.com, pd.csi.storage.gke.io, etc.)
Reclaim PolicyDelete or Retain
Volume Binding ModeImmediate or WaitForFirstConsumer
Allow ExpansionWhether volumes can be resized
DefaultIs this the default StorageClass

Common StorageClasses by Provider

AWS EKS

NameProvisionerTypeIOPSUse Case
gp3ebs.csi.aws.comGeneral Purpose SSD3000-16000Default, balanced performance
gp2ebs.csi.aws.comGeneral Purpose SSDBurstableLegacy default
io2ebs.csi.aws.comProvisioned IOPS SSD100-64000High performance databases
st1ebs.csi.aws.comThroughput Optimized HDD-Big data, log processing

Google GKE

NameProvisionerTypeUse Case
standardpd.csi.storage.gke.ioStandard PDDefault, cost-effective
balancedpd.csi.storage.gke.ioBalanced PDGeneral purpose workloads
ssdpd.csi.storage.gke.ioSSD PDHigh performance
extremepd.csi.storage.gke.ioExtreme PDHighest performance

Azure AKS

NameProvisionerTypeUse Case
defaultdisk.csi.azure.comStandard HDDLow cost, infrequent access
managed-premiumdisk.csi.azure.comPremium SSDProduction workloads
managed-csi-premiumdisk.csi.azure.comPremium SSDHigh performance
azurefilefile.csi.azure.comAzure FilesReadWriteMany support

Volume Binding Modes

ModeBehaviorUse When
ImmediatePV provisioned immediately when PVC createdSingle-zone clusters, default behavior
WaitForFirstConsumerPV provisioned when pod using PVC is scheduledMulti-zone clusters, ensures volume in same zone as pod

StorageClass Parameters

AWS EBS Example:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iops: "5000"
  throughput: "250"
  encrypted: "true"
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
reclaimPolicy: Delete

Google PD Example:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ssd
provisioner: pd.csi.storage.gke.io
parameters:
  type: pd-ssd
  replication-type: regional-pd
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

Default StorageClass

Mark a StorageClass as default for PVCs without explicit class:

metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"

Behavior:

  • PVCs without storageClassName use default class
  • Only one default class should exist per cluster
  • Helps simplify PVC creation for users

ResourceQuotas

Limit storage consumption per namespace to prevent resource exhaustion.

Viewing ResourceQuotas

ColumnDescription
NameQuota name
NamespaceKubernetes namespace
ResourceWhat is limited (storage, PVCs, etc.)
UsedCurrent consumption
HardMaximum allowed

Storage Quota Types

Quota TypeDescription
requests.storageTotal storage requested across all PVCs
persistentvolumeclaimsMaximum number of PVCs allowed
<storageclass>.storageclass.storage.k8s.io/requests.storageStorage limit for specific StorageClass
<storageclass>.storageclass.storage.k8s.io/persistentvolumeclaimsPVC count limit for specific StorageClass

Example ResourceQuota

apiVersion: v1
kind: ResourceQuota
metadata:
  name: storage-quota
  namespace: development
spec:
  hard:
    requests.storage: "500Gi"
    persistentvolumeclaims: "20"
    gp3.storageclass.storage.k8s.io/requests.storage: "300Gi"
    gp3.storageclass.storage.k8s.io/persistentvolumeclaims: "15"

Enforcement:

Used / Hard
─────────────────────────────────────
Storage (All):          250Gi / 500Gi
PVC Count:              12 / 20
Storage (gp3):          180Gi / 300Gi
PVC Count (gp3):        10 / 15

Quota Exceeded Behavior

When quota is exceeded, PVC creation fails:

Error creating PVC:
Exceeded quota: storage-quota
Requested: requests.storage=100Gi
Used: 450Gi
Hard: 500Gi

Resolution:

  • Request quota increase
  • Delete unused PVCs
  • Use different StorageClass with available quota
  • Archive data to cheaper storage

Example: StatefulSet with Persistent Storage

Scenario

Deploy PostgreSQL database with persistent storage using StatefulSet and PVC.

StatefulSet Configuration

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgresql
  namespace: production
spec:
  serviceName: postgresql
  replicas: 3
  selector:
    matchLabels:
      app: postgresql
  template:
    metadata:
      labels:
        app: postgresql
    spec:
      containers:
      - name: postgresql
        image: postgres:15
        ports:
        - containerPort: 5432
        env:
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: postgresql-secret
              key: password
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "gp3"
      resources:
        requests:
          storage: 100Gi

Created Resources

PVCs Created:

NAME                  STATUS   VOLUME                 CAPACITY   STORAGE CLASS
data-postgresql-0     Bound    pvc-abc123             100Gi      gp3
data-postgresql-1     Bound    pvc-def456             100Gi      gp3
data-postgresql-2     Bound    pvc-ghi789             100Gi      gp3

PVs Auto-Provisioned:

NAME         CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM
pvc-abc123   100Gi      RWO            Delete           Bound    production/data-postgresql-0
pvc-def456   100Gi      RWO            Delete           Bound    production/data-postgresql-1
pvc-ghi789   100Gi      RWO            Delete           Bound    production/data-postgresql-2

Scaling Behavior

Scale to 5 replicas:

kubectl scale statefulset postgresql --replicas=5

New PVCs created automatically:

data-postgresql-3     Pending   -                      -          gp3
data-postgresql-4     Pending   -                      -          gp3

After provisioning:

data-postgresql-3     Bound     pvc-jkl012             100Gi      gp3
data-postgresql-4     Bound     pvc-mno345             100Gi      gp3

Total Storage Used: 5 × 100Gi = 500Gi


Troubleshooting Storage Issues

PVC Stuck in Pending
Causes:
  • No PV matches requested capacity/access mode
  • StorageClass not found or misconfigured
  • ResourceQuota exceeded
  • Volume provisioning failed (check events)

Solution: Check PVC events with kubectl describe pvc [name]. Verify StorageClass exists and provisioner is healthy.

Pod Stuck: FailedAttachVolume
Causes:
  • Volume already attached to different node
  • Volume doesn't exist in cloud provider
  • Node doesn't have permissions to attach volume
  • Volume zone doesn't match node zone

Solution: Check pod events. Verify PV exists in cloud console. Ensure node and volume are in same availability zone.

PVC Won't Delete
Causes:
  • PVC is still in use by a pod (protection finalizer)
  • PVC has deletion finalizer preventing removal

Solution: Delete or scale down pods using the PVC first. Check for finalizers with kubectl get pvc [name] -o yaml. Never force-remove finalizers without understanding implications.

Volume Expansion Failed
Causes:
  • StorageClass doesn't allow expansion
  • Requested size exceeds cloud provider limits
  • Volume type doesn't support online resizing

Solution: Verify allowVolumeExpansion: true in StorageClass. Some volumes require pod restart after expansion. Check cloud provider volume limits.

Best Practices
Use WaitForFirstConsumer - Ensures volume provisioned in same zone as pod (multi-zone clusters)
Set Resource Quotas - Prevent runaway storage costs with namespace quotas
Enable Volume Expansion - Allow resizing without recreating PVCs
Choose Appropriate StorageClass - Match performance requirements (gp3 vs io2) to workload needs
Monitor PV Usage - Set up alerts for storage capacity nearing limits
Backup Critical Data - PVs can fail; use volume snapshots or backups for databases