Part 5 – Scaling Up a new Proxmox/Ceph Cluster

In this blog, we will configure a new node into our Ceph Cluster.

Rather than doing it via the UI,we can easily deploy a new VM shell from my base proxmox server.

qm create 144 --name pve-n4 --ostype l26 --machine q35 \
  --cpu host --sockets 1 --cores 4 --memory 16384 --balloon 0 \
  --scsihw virtio-scsi-single \
  --scsi0 Local-2TB:32,iothread=1,ssd=1,discard=on \
  --scsi1 Local-2TB:64,iothread=1,ssd=1,discard=on \
  --net0 virtio,bridge=vmbr1,firewall=0 \
  --net1 virtio,bridge=vmbr1,tag=1612,firewall=0 \
  --net2 virtio,bridge=vmbr1,tag=1613,firewall=0 \
  --ide2 local:iso/proxmox-ve_9.2-1.iso,media=cdrom \
  --boot 'order=scsi0;ide2' --agent 1

qm create 144 --name pve-n4 --ostype l26 --machine q35 \
  --cpu host --sockets 1 --cores 4 --memory 16384 --balloon 0 \
  --scsihw virtio-scsi-single \
  --scsi0 Local-2TB:32,iothread=1,ssd=1,discard=on \
  --scsi1 Local-2TB:64,iothread=1,ssd=1,discard=on \
  --net0 virtio,bridge=vmbr1,firewall=0 \
  --net1 virtio,bridge=vmbr1,tag=1612,firewall=0 \
  --net2 virtio,bridge=vmbr1,tag=1613,firewall=0 \
  --ide2 local:iso/proxmox-ve_9.2-1.iso,media=cdrom \
  --boot 'order=scsi0;ide2' --agent 1

After the template shell is created, just power on the VM and Install Proxmox VE as we did in chapter1.

Reboot when done and login as https://172.16.11.44:8006

The next step is to update the repositories . A fresh install of Proxmox will want to look at an enterprise repository by default so in our case we are a free subscriber, so we just need to update the depot to disable the enterprise ones and enable the free ones.

On all the nodes, just go to Updates → Repositories

Select all the enterprise subscriptions and disable it

Now, Click Add, choose No-Subscription from the drop-down, then Add and Choose Ceph Squid No-Subscription

Now, Click Add, choose No-Subscription from the drop-down,

As you add it, we will get the message that we are free tier and we are now able to get updates from Proxmox

Go to Refresh → Upgrade → Y

This actually triggers apt-get update

Since our Promox nodes are now on Virtual Environment 9.2.21 we will also just upgrade the version to 9.2.2.1 and then it should be an easy task to join to our Ceph cluster.

Add the host entry into /etc/hosts file on all hosts

cat >> /etc/hosts <<‘EOF’
172.16.11.41 pve-n1.lab.internal pve-n1
172.16.11.42 pve-n2.lab.internal pve-n2
172.16.11.43 pve-n3.lab.internal pve-n3
EOF

Login to pve-n4 and go to the network section and we can see three NIC’s here. Here we will need to configure NIC1 as our Migration IP and NIC2 as our Ceph IP

Name	Alternative name	VM NIC	Use
nic0	enp6s18	net0	Management ,1611
nic1	enp6s19	net1	Migration, 1612
nic2	enp6s20	net2	Ceph, 1613

Set the migration IP to 1612

Double-click enp6s19
IPv4/CIDR: 172.16.12.44/24
Gateway: leave empty
Autostart: ticked
Comment: Migration (vMotion)

Set the Ceph IP to 1613

Double-click enp6s20
IPv4/CIDR: 172.16.13.44/24
Gateway: leave empty
Autostart: ticked
Comment: Ceph (vSAN)

Click Apply Configuration and then Review the configuration from the UI

To join the new node to cluster go to Datacenter → Cluster → Join Cluster

Under join, we copy the code into the information section

Password: pve-n1’s root password
Link 0: 172.16.11.44
Link 1: 172.16.12.44
Click Join nested-cluster

Run pvcem status from any node to see the detailed stats

root@pve-n1:~# pvecm status
Cluster information
-------------------
Name:             nested-cluster
Config Version:   4
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Tue Oct 20 20:35:58 2026
Quorum provider:  corosync_votequorum
Nodes:            4
Node ID:          0x00000001
Ring ID:          1.79
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   4
Highest expected: 4
Total votes:      4
Quorum:           3
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 172.16.11.41 (local)
0x00000002          1 172.16.11.42
0x00000003          1 172.16.11.43
0x00000004          1 172.16.11.44
root@pve-n1:~#

Install Ceph on pve-n4

OSD: pve-n4 → Ceph → OSD → Create: OSD

Choose 64 GB disk as /dev/sdb as our Capacity Disk
DB disk = choose as /dev/sda, this is where indexing goes so this is like Cache tier
WAL disk = this is a write ahead log for incoming writers, so just keep it on our /dev/sda
Under Device Class , choose NVMe

Our 4th OSD shows up node 4

Next step is to add a Monitor on every node in Ceph so click on pve-n4 and go to Ceph > Montior. In this node montior and Manager are running

Ceph monitors work by majority vote. Before Ceph changes anything, more than half the monitors have to agree. That’s what stops a “split brain”, where two halves of the cluster each think they’re in charge so we just 3 monitors.

Manager table → click Create → Host pve-n4 → Create → TASK OK.

Run ceph status to check the health of ceph so we can now see 4 monitor Deamons are active and also the manager nodes showing Active/Standy states

ot@pve-n4:~# ceph status
  cluster:
    id:     63400913-953c-4b9f-bee6-f6bbb259245a
    health: HEALTH_OK

  services:
    mon: 4 daemons, quorum pve-n1,pve-n2,pve-n3,pve-n4 (age 50s) [leader: pve-n1]
    mgr: pve-n2(active, since 2h), standbys: pve-n1, pve-n4, pve-n3
    osd: 4 osds: 4 up (since 3m), 4 in (since 3m)

  data:
    pools:   2 pools, 33 pgs
    objects: 1.35k objects, 4.9 GiB
    usage:   13 GiB used, 243 GiB / 256 GiB avail
    pgs:     33 active+clean

root@pve-n4:~#

(Visited 6 times, 1 visits today)