Kubernetes Persistent Storage & Deployment

By default any volume that we create in a pod is transient in nature which means pods are there are to execute a task and it will just die once the task is completed so if we wish to keep data perm we need a pvc..

What is Persistent Storage in Kubernetes?

A piece of disk attached to a pod or container for data consumption. Without it, all data inside a container is lost when the pod restarts as the container filesystem is ephemeral by default.

Persistent storage lives in the cluster as YAML objects and survives pod restarts, redeployments, and upgrades.

https://kubernetes.io/docs/concepts/storage/persistent-volumes/

PersistentVolume (PV)

The actual disk. A resource in the cluster that represents a piece of storage — either pre-provisioned manually by an admin or dynamically created by a storage class.

Think of it like a LUN in VMware  the raw storage resource.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  local:
    path: /mnt/data

In most modern setups you never write a PV manually — the StorageClass creates it for you automatically.

PersistentVolumeClaim (PVC)

The request for storage. A pod doesn’t talk to a PV directly and it talks to a PVC. The PVC says “I need 500Mi of ReadWriteOnce storage” and Kubernetes finds or creates a PV that matches.

Think of it like a datastore request , you just that you get the storage you asked for.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mealie-pvc
  namespace: mealie
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 500Mi
  storageClassName: local-path

Once bound, the PVC gets a PV assigned and shows Bound status:

kubectl get pvc -n mealie
NAME         STATUS   VOLUME                                     CAPACITY
mealie-pvc   Bound    pvc-923af421-441e-4555-a335-f28e9c77e6c5   500Mi

StorageClass

The provisioner. A YAML object that defines how storage is created — and depending on the platform it runs on, it can have completely different backends:

In your k3s cluster the default is local-path — it provisions storage from /var/lib/rancher/k3s/storage/ on whichever node the pod lands on.

kubectl get storageclass
NAME                   PROVISIONER
local-path (default)   rancher.io/local-path
longhorn               driver.longhorn.io
local-storage          kubernetes.io/no-provisioner

Access Modes

RWO is what you use for most apps. If you try RollingUpdate with RWO, the new pod can’t mount the volume while the old pod still has it — use Recreate strategy instead (like Forgejo does).

How the Volume Gets Linked to a Pod

Three pieces must connect:

volumeMounts (inside container)
    └── name: mealie-data  ←─────────────────┐
                                              │ internal name — must match
volumes (inside spec)                         │
    └── name: mealie-data  ←─────────────────┘
        └── persistentVolumeClaim:
                └── claimName: mealie-pvc  ←── points to the PVC object

name in volumeMounts and volumes is just an internal reference — they must be identical but can be anything you want. claimName points to the actual PVC object name in the cluster.

Deployment Order — The Right Way

Always apply in this order. Each object depends on the one before it.

1. Namespace

kubectl apply -f namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: mealie

Everything lives inside a namespace. Create it first or everything else fails.

2. Storage

kubectl apply -f storage.yaml
kubectl get pvc -n mealie   # wait for Bound

PVC must exist and be Bound before the deployment can schedule. If the PVC doesn’t exist the pod stays Pending with persistentvolumeclaim "name" not found.

3. Deployment

kubectl apply -f deployment.yaml
kubectl get pods -n mealie -w

Pod schedules, pulls the image, mounts the PVC, starts the app.

4. Service

kubectl apply -f service.yaml
kubectl get svc -n mealie

Exposes the app. LoadBalancer on k3s uses ServiceLB to assign node IPs automatically.

File Structure

homelab/mealie/
├── namespace.yaml     # Namespace
├── storage.yaml       # PersistentVolumeClaim
├── deployment.yaml    # Deployment
└── services.yaml      # Service

You can also apply the whole directory at once — Kubernetes figures out the order:

kubectl apply -f ~/homelab/mealie/

Useful Commands

# check PVC status
kubectl get pvc -n mealie
kubectl describe pvc mealie-pvc -n mealie


# check where data physically lives
kubectl describe pv <pv-name>
# look for: Path: /var/lib/rancher/k3s/storage/pvc-{uid}_mealie_mealie-pvc



# check the pvc claims 
kubectl describe persistentvolumeclaims mealie-pvc
# look for: Path: /var/lib/rancher/k3s/storage/pvc-{uid}_mealie_mealie-pvc



# delete svc (data survives)
kubectl delete svc mealie -n mealie
kubectl apply -f services.yaml


# delete everything except the PVC (data survives)
kubectl delete deployment mealie -n mealie
kubectl apply -f deployment.yaml

# Deletes everything including data
kubectl delete namespace mealie

Access Mealie at http://172.16.77.10:9000
Location of the path will be on /var/lib/rancher/k3s/storage/ as local-path PVC which Kubernetes determines.
awk@k3s-worker-01:~$ sudo ls -l /var/lib/rancher/k3s/storage/
total 20
drwxrwxrwx 7 911 911 4096 Apr 20 19:16 pvc-32e9c071-a037-4f40-a20a-a8ff8e7fa4d6_mealie_mealie-data
drwxrwsrwx 3 root 1001 4096 Apr 1 15:53 pvc-56eb0a38-4754-413e-86c9-149c807c697e_database_data-my-postgresql-0
drwxrwxrwx 2 root root 4096 Apr 20 19:16 pvc-97935f46-0a83-4435-b58f-66e34fb4f317_mealie_mealie-temp
drwxrwxrwx 2 911 911 4096 Apr 20 19:16 pvc-d070384d-b974-4a3c-b126-395c7f43c888_mealie_mealie-logs
drwxrwxrwx 35 root root 4096 Apr 11 22:26 pvc-d464d034-97a7-47f2-b1e6-6f7f9211459a_stock-platform_prometheus-pvc
awk@k3s-worker-01:~$