Configuring K8s Persistent Storage using Longhorn

If we need persistent storage for our Kubernetes clusters, we need the kubernetes volumes to be sitting on shared storage so here is where Longhorn comes in .

Longhorn provides the actual storage backend and turns our cluster’s disks into a distributed storage system so this is like ceph , distrubuted file system/vSAN but for lab longhorn is just fine.

Each volume is split into replicas (default: 3) just like our vSAN but this uses iSCSI to expose the volume to the pod

Install dependencies on all nodes

Enable ISCSI ID on all nodes

Install helm just on the control node

Helm is the package manager for kubernetes just like apt is for ubuntu

Download helm

curl -o /tmp/get-helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3

Now install helm:

sudo bash /tmp/get-helm.sh

Then add autocompletion:

echo 'source <(helm completion bash)' >> ~/.bashrc
source ~/.bashrc

To add the Longhorn to Helm so its like adding a new software source in Ubuntu we use the below command

helm install longhorn longhorn/longhorn --namespace longhorn-system --create-namespace

When you run kubectl it works because K3s automatically set it up for you. But Helm looks in a different place for the config file so it can’t find it. This tells Helm the Kubernetes config is at this location.

echo 'export KUBECONFIG=/etc/rancher/k3s/k3s.yaml' >> ~/.bashrc

Reload

source ~/.bashrc

Rerun longhorn install

helm install longhorn longhorn/longhorn --namespace longhorn-system --create-namespace

Run get pods

k get pods -n longhorn-system

Run get svc to check the ports exposed if any

Export the yaml to make the edit so here the front end pod needs to be done

kubectl get svc longhorn-frontend -n longhorn-system -o yaml > /tmp/longhorn-svc.yaml

Open the file in nano and Change cluster type : to Node Port and also add node port 30088 to expose the port

Apply the config

kubectl get svc longhorn-frontend -n longhorn-system -o yaml > /tmp/longhorn-svc.yaml

kubectl get svc longhorn-frontend -n longhorn-system -o yaml > /tmp/longhorn-svc.yaml

Our UI is here.

Total storage pool: ~78Gi across 3 nodes

Each node has 1 replica allocated already

This is our distributed storage working

Longhorn DaemonSet agents

Longhorn runs DaemonSet pods on every node like NSX VIBs on ESXi host and these pods cannot be drained — they stay on every node just like NSX VIBs stay on ESXi during maintenance mode.