vCSA: /storage/log partition full

Loading

Your monitoring platform would pick the error as /storage/ log is full and this is something you will need to investigate to figure out what’s causing it. The common issue for this partition to become full is vCenter support logs present on the directory.

In this blog, we will extend our /storage/log filesystem. The procedure to expand any filesystem on vCSA is by identifying the correct disk, extending the disk from VM Console settings and running the autogrow script to grow the filesystem.

List of VMDKs/Partitions for a vCenter Server Appliance 6.7 – Size Mount point and Purpose

Disk
(VMDK)
Default Size
(Tiny with Default Storage Size)
Mount PointPurpose
VMDK112GB/ (10GB)
/boot (132MB)
SWAP (1GB)
Directory where the kernel images and boot loader configurations are stored.
VMDK21.8GB/tmpDirectory used to store temporary files generated or used by services from vCenter Server
VMDK325GBSWAPDirectory used when the system is out of memory to swap to disk
VMDK425GB/storage/coreDirectory where core dumps from VPXD process from the vCenter Server are stored
VMDK510GB/storage/logDirectory where vCenter Server and Platform Services Controller store all logs for the environment
VMDK610GB/storage/dbVMware Postgres database storage location
VMDK75GB/storage/dblogVMware Postgres database logging location
VMDK810GB/storage/seatStats, Events, Alarms and Tasks (SEAT) directory for VMware Postgres
VMDK91GB/storage/netdumpVMware Netdump collector repository that stores ESXi dumps
VMDK1010GB/storage/autodeployVMware Auto Deploy repository that stores the thinpackages used for stateless booting of ESXi hosts
VMDK1110GB/storage/imagebuilderVMware Image Builder repository that stores the vSphere image profiles, software depots and VIB packages, such as driver VIBs and update VIBs.
VMDK12100GB/storage/updatemgrVMware Update Manager repository where patches and updates are stored for Virtual Machine and ESXi hosts
VMDK1350GB/storage/archiveVMware Postgres database’s Write-Ahead Logging (WAL) location

List of VMDKs/Partitions for a vCenter Server Appliance 7.0Size Mount point and Purpose

Disk
(VMDK)
Default Size
(Tiny with Default Storage Size)
Mount PointPurpose
VMDK148GB/ (46GB)
/boot (132MB)
SWAP (1GB)
Directory where the kernel images and boot loader configurations are stored.
VMDK21.8GB/tmpDirectory used to store temporary files generated or used by services from vCenter Server
VMDK325GBSWAPDirectory used when the system is out of memory to swap to disk
VMDK425GB/storage/coreDirectory where core dumps from VPXD process from the vCenter Server are stored
VMDK510GB/storage/logDirectory where vCenter Server and Platform Services Controller store all logs for the environment
VMDK610GB/storage/dbVMware Postgres database storage location
VMDK75GB/storage/dblogVMware Postgres database logging location
VMDK810GB/storage/seatStats, Events, Alarms and Tasks (SEAT) directory for VMware Postgres
VMDK91GB/storage/netdumpVMware Netdump collector repository that stores ESXi dumps
VMDK1010GB/storage/autodeployVMware Auto Deploy repository that stores the thinpackages used for stateless booting of ESXi hosts
VMDK1110GB/storage/imagebuilderVMware Image Builder repository that stores the vSphere image profiles, software depots and VIB packages, such as driver VIBs and update VIBs.
VMDK12100GB/storage/updatemgrVMware Update Manager repository where patches and updates are stored for Virtual Machine and ESXi hosts
VMDK1350GB/storage/archiveVMware Postgres database’s Write-Ahead Logging (WAL) location
VMDK1410GB/storage/vtsdbVMware vTSDB Service Repository that stores the stats
VMDK155GB/storage/vtsdblogVMware vTSDB Service Repository that stores the logs of the service
VMDK16100GB/storage/lifecycleWorkload Control Plane service stage directory or software depot, this stores the binaries for install and update/upgrade.

Verification

  • Login to the VAMI UI as shown will root password and verify if all services are up . As seen all the vCenter services will be running fine.

  • Login to the vCenter via ssh session.
  • Execute the command df -H to view the file system space

Solution

Locate the disk holding the /storage/log by running the command

df-h;lsblk;lsscsi


The /storage/log mount point is disk label sdf..

The sdf is located on disk 13. Typically it’s not a best practice to extend disks on a Linux based system and the ideal scenario would have been to add a new disk and extend the partition via lvm but on the vCenter system, this is okay

On the vCenter VM, Click Edit settings, Locate disk 4 and choose to extend space

Fix for the vCenter /storage/log storage partition full error

Establish an SSH session to the vCSA and run the autogrow script as shown to automatically expand any logical volume that has space added. This script looks for changes in the underlying storage and expands the volume without the need to reboot.

/usr/lib/applmgmt/support/scripts/autogrow.sh

Execute the command df -H again to verify the size of the/storage/archive partition and we can see the partition has been extended.

References

Increasing the disk space for the vCenter Server Appliance in vSphere 6.5, 6.7 and 7.0 (2145603)

List of VMDKs/Partitions for a vCenter Server Appliance 6.7

List of VMDKs/Partitions for a vCenter Server Appliance 7.0

(Visited 4,587 times, 1 visits today)

Leave a Reply