Your monitoring platform would pick the error as /storage/ log is full and this is something you will need to investigate to figure out what’s causing it. The common issue for this partition to become full is vCenter support logs present on the directory.
In this blog, we will extend our /storage/log filesystem. The procedure to expand any filesystem on vCSA is by identifying the correct disk, extending the disk from VM Console settings and running the autogrow script to grow the filesystem.
By default, when the some disks reach 95% full, the main vCenter service, vmware-vpxd, is turned off automatically and this is to protect the database from corruption and may cause vCenter Server to be inaccessible with 503 Service Unavailable errors.
List of VMDKs/Partitions for a vCenter Server Appliance 6.7 – Size Mount point and Purpose
| Disk (VMDK) | Default Size (Tiny with Default Storage Size) | Mount Point | Purpose |
| VMDK1 | 12GB | / (10GB) /boot (132MB) SWAP (1GB) | Directory where the kernel images and boot loader configurations are stored. |
| VMDK2 | 1.8GB | /tmp | Directory used to store temporary files generated or used by services from vCenter Server |
| VMDK3 | 25GB | SWAP | Directory used when the system is out of memory to swap to disk |
| VMDK4 | 25GB | /storage/core | Directory where core dumps from VPXD process from the vCenter Server are stored |
| VMDK5 | 10GB | /storage/log | Directory where vCenter Server and Platform Services Controller store all logs for the environment |
| VMDK6 | 10GB | /storage/db | VMware Postgres database storage location |
| VMDK7 | 5GB | /storage/dblog | VMware Postgres database logging location |
| VMDK8 | 10GB | /storage/seat | Stats, Events, Alarms and Tasks (SEAT) directory for VMware Postgres |
| VMDK9 | 1GB | /storage/netdump | VMware Netdump collector repository that stores ESXi dumps |
| VMDK10 | 10GB | /storage/autodeploy | VMware Auto Deploy repository that stores the thinpackages used for stateless booting of ESXi hosts |
| VMDK11 | 10GB | /storage/imagebuilder | VMware Image Builder repository that stores the vSphere image profiles, software depots and VIB packages, such as driver VIBs and update VIBs. |
| VMDK12 | 100GB | /storage/updatemgr | VMware Update Manager repository where patches and updates are stored for Virtual Machine and ESXi hosts |
| VMDK13 | 50GB | /storage/archive | VMware Postgres database’s Write-Ahead Logging (WAL) location |
List of VMDKs/Partitions for a vCenter Server Appliance 7.0 – Size Mount point and Purpose
| Disk (VMDK) | Default Size (Tiny with Default Storage Size) | Mount Point | Purpose |
| VMDK1 | 48GB | / (46GB) /boot (132MB) SWAP (1GB) | Directory where the kernel images and boot loader configurations are stored. |
| VMDK2 | 1.8GB | /tmp | Directory used to store temporary files generated or used by services from vCenter Server |
| VMDK3 | 25GB | SWAP | Directory used when the system is out of memory to swap to disk |
| VMDK4 | 25GB | /storage/core | Directory where core dumps from VPXD process from the vCenter Server are stored |
| VMDK5 | 10GB | /storage/log | Directory where vCenter Server and Platform Services Controller store all logs for the environment |
| VMDK6 | 10GB | /storage/db | VMware Postgres database storage location |
| VMDK7 | 5GB | /storage/dblog | VMware Postgres database logging location |
| VMDK8 | 10GB | /storage/seat | Stats, Events, Alarms and Tasks (SEAT) directory for VMware Postgres |
| VMDK9 | 1GB | /storage/netdump | VMware Netdump collector repository that stores ESXi dumps |
| VMDK10 | 10GB | /storage/autodeploy | VMware Auto Deploy repository that stores the thinpackages used for stateless booting of ESXi hosts |
| VMDK11 | 10GB | /storage/imagebuilder | VMware Image Builder repository that stores the vSphere image profiles, software depots and VIB packages, such as driver VIBs and update VIBs. |
| VMDK12 | 100GB | /storage/updatemgr | VMware Update Manager repository where patches and updates are stored for Virtual Machine and ESXi hosts |
| VMDK13 | 50GB | /storage/archive | VMware Postgres database’s Write-Ahead Logging (WAL) location |
| VMDK14 | 10GB | /storage/vtsdb | VMware vTSDB Service Repository that stores the stats |
| VMDK15 | 5GB | /storage/vtsdblog | VMware vTSDB Service Repository that stores the logs of the service |
| VMDK16 | 100GB | /storage/lifecycle | Workload Control Plane service stage directory or software depot, this stores the binaries for install and update/upgrade. |
Verification
- Login to the VAMI UI as shown will root password and verify if all services are up . As seen all the vCenter services will be running fine.
- Login to the vCenter via ssh session.
- Execute the command df -H to view the file system space
Solution
Locate the disk holding the /storage/log by running the command
df-h;lsblk;lsscsi
The /storage/log mount point is disk label sdf..
The sdf is located on disk 13. Typically it’s not a best practice to extend disks on a Linux based system and the ideal scenario would have been to add a new disk and extend the partition via lvm but on the vCenter system, this is okay
On the vCenter VM, Click Edit settings, Locate disk 4 and choose to extend space
Establish an ssh session to the vCenter and run the below commands to clear up the log files taking up space on the /log partition
find /storage/log/vmware/ -mtime +1 -type f -name "localhost*" | while read file; do rm "$file" ; done
find /storage/log/vmware/ -mtime +1 -type f -name "catalina*log" | while read file; do rm "$file" ; done
find /storage/log/vmware/vmware-sps -mtime +1 -type f -name "sps-access-log.*" | while read file; do rm "$file" ; done
find /storage/log/vmware/ -mtime +7 -type f -name "sps.log*" | while read file; do truncate -s 0 "$file" ; done
find /storage/log/vmware/ -mtime +14 -type f -name "*.gz" | while read file; do rm "$file" ; done
Fix for the vCenter /storage/log storage partition full error
Establish an SSH session to the vCSA and run the autogrow script as shown to automatically expand any logical volume that has space added. This script looks for changes in the underlying storage and expands the volume without the need to reboot.
/usr/lib/applmgmt/support/scripts/autogrow.shExecute the command df -H again to verify the size of the/storage/archive partition and we can see the partition has been extended.
References
Increasing the disk space for the vCenter Server Appliance in vSphere 6.5, 6.7 and 7.0 (2145603)
List of VMDKs/Partitions for a vCenter Server Appliance 6.7
List of VMDKs/Partitions for a vCenter Server Appliance 7.0