Ceph is software defined distributed storage system, designed for excellent performance, reliability, and scalability. Ceph will use local disks available on each of our nodes in cluster to create a shared storage.
The config file for ceph is /etc/pve/ceph.conf.
The role of monitor is to keep information of ceph clusters and there is 1 monitor and for production its 3. it has to be odd. The second component is called OSD is a deamon that makes it possible and it performs the r and writes.. 3 HDD means 3 osd process.
Login to pve-n1 and go to the network section and we can see three NIC’s here. Here we will need to configure NIC1 as our Migration IP and NIC2 as our Ceph IP
Lab Topology
Install Ceph
To install Ceph, Go to Datacenter → Select the first node pve-n1 > Install Ceph
Choose the following
- Ceph version:
tentacle (20.2) - Repository:
No-Subscription - Click Start tentacle installation
Click Yes to Continue
Under the Configuration tab, Choose the Public network as our Ceph Network 172.16.13.0/24, number of replicas =3, min replicas=2
Review and Click Finish
Our Ceph is now ready
Click Ceph on pve-n1 > OSD
- Choose 64 GB disk as /dev/sdb as our Capacity Disk
- DB disk = choose as /dev/sda, this is where indexing goes so this is like Cache tier
- WAL disk = this is a write ahead log for incoming writers, so just keep it on our /dev/sda
- Under Device Class , choose NVMe
To install Ceph on pve-n2 and pve-n3, following the same procedure. We will have 3 disks holding a osd.
Next step is to add a Monitor on every node in Ceph so click on pve-n1 and go to Ceph > Montior. In this node montior and Manager are running
Hover over to pve-n2 and pve-n3 adn this shows still we have only these roles installed on node 1 so lets install it on node 2 and node 3
To Install the Montor and Manager on pve-n2 go to Ceph > Montior > Click Create
wait for task OK
Manager table → click Create → Host pve-n2 → Create → TASK OK.
Repeat the same on the final node
Run ceph status to check the health of ceph so we can now see 3 monitor Deamons are active and also the manager nodes showing Active/Standy states
root@pve-n1:~# ceph status
cluster:
id: 63400913-953c-4b9f-bee6-f6bbb259245a
health: HEALTH_OK
services:
mon: 3 daemons, quorum pve-n1,pve-n2,pve-n3 (age 60s) [leader: pve-n1]
mgr: pve-n1(active, since 44m), standbys: pve-n2, pve-n3
osd: 3 osds: 3 up (since 16m), 3 in (since 16m)
data:
pools: 1 pools, 1 pgs
objects: 2 objects, 577 KiB
usage: 81 MiB used, 192 GiB / 192 GiB avail
pgs: 1 active+clean
root@pve-n1:~#
Create Ceph Pools
Assign the OSD’s we created into Storage Pools so click on Any node → Ceph → Pools → Create:
Choose the following and Click Create
- Select the name of the pool as VM-Pool
- Size / Min. Size – 3 copies, stay online with 2
- Crush Value- pick replicated_rule which means use all OSD’s
- PG = is placement groups which 128 by default
- Target Size – This is just a measure of how big we expect the pool to become, we will skip it
The pool is done, and VM-Pool now appears in the tree under pve-n1 (and every other node) and this is our shared datastore or in short, our vSAN datastore, live.
There are two pools here.
- mgr Pool – Ceph’s internal pool for the manager.
- VM-Pool – our pool for VM Storag
Right Click a VM and migrate the VM to our new vmpool
That worked flawlessly…
Testing HA in Proxmox
Unlike VMware, where all VM’s in a cluster are protected by HA when the feature is turned in Proxmox HA Is per server and this needs to be manually configured.
To add a VM to HA go to Datacenter → HA → Resources → Add all the VM’s we need to be protected with HA
Node Affinity rule in Proxmox is the same as VM-Host affinity rule so the VM should run on this host
Resource affinity rule is the same as VM-VM affinity / anti-affinity in VMware
Let’s do one controlled failure of a node and watch both Ceph and HA react. It’s the vSAN + vSphere HA “pull a host” test so we will shut one node down
After few moments vm03 bounces back on another pve-node1
That’s it to all the proxmox HA level tests.