Managing ESXi Locked VM files

In an ideal scenario, our healthy VM tries to create lock files to prevent reads/writes from other hosts accessing the disk thus preventing data corruption. In this blog, we will identify the owner of the file lock which is not letting us power up a VM.

We will show you two ways to figure out the locks on

1. Identifying the locked file using vmkfstools -D
2. Identifying the Locked file using vmfsfilelockinfo

Principle of Lock Files

There are two types of Lock Files

• Reader – A disk in nonpersistent mode is protected by reader-lock files and these can be read by more than one VM at a given point so this could include our RDM’s.
• Writer – Disks in persistent use writer lock files that can be accessed by one single VM.

During a normal operating scenario, when the VM is powered off, the lock is removed however when the VM crashes before it had its time to remove the lock file, the stale file remains. When such VMs are started, it does a check to see if the lock was created on the same ESXi host where the VM was running before and if the process that created the lock in the first place is not running anymore. When these conditions are met, the VM releases the lock by itself and VM can power on else it throws the warning

• Failed to add disk scsi0:1. Failed to power on scsi0:1
• Cannot power on the virtual machine after deploying it from a template.
• Powering on a virtual machine fails with an error:
• Unable to open Swap File
• Unable to access a file since it is locked
• Unable to access a file filename since it is locked
• Unable to access Virtual machine configuration

Find locked files using vmkfstools

We will need to establish this info to unlock the disk files.

• Find the IP address of the locking host or the lock owner
• Find the host locking the file by IP address
• Find the locking process and release the file lock

Open the SSH console, and browse to the datastore location holding the VM. An easy way is to run the touch * command to change the datestamp of all files in this directory and If you have a lock you will get the output as shown.

root@esxlab:/vmfs/volumes/607c2b1c-13cf66f2-f2f7-b8ac6f19f4ba/CV_CS_Master] touch *
touch: CV_CS_Master-flat.vmdk: Device or resource busy
touch: CV_CS_Master_1-flat.vmdk: Device or resource busy
touch: CV_CS_Master_2-flat.vmdk: Device or resource busy

Who owns the lock?

In this example, the 3 files above are in some kind of lock and cannot be written to and without resolving this our VMs won’t power up.

vmkfstools – A tool to check which host is holding locks on VM files.

• The mode indicates the type of lock that is on the file. The list of mode locks are:
  1. mode 0 = no lock
  2. mode 1 = is an exclusive lock (vmx file of a powered-on VM, the currently used disk (flat or delta), *vswp, etc.)
  3. mode 2 = is a read-only lock (e.g. on the ..-flat.vmdk of a running VM with snapshots)
  4. mode 3 = is a multi-writer lock (e.g. used for MSCS clusters disks or FT VMs).

In this example, mode 2 indicates a file lock, the owner of the file 00000000-00000000-0000-000000000000 is of no use to us but as you go below we notice RO Owner and also the bit highlighted in orange b8ac6f19f4ba, it’s our MAC ID of the ESXI host holding the VM files.

The read-only owner of this lock is MAC address b8ac6f19f4ba.

Locate the ESX MAC ID

Get-Cluster < Clustername> | Get-VMHost | Get-VMHostNetworkAdapter | ?{$_.Mac -eq “b8:ac:6f:19:f4:ba“}

Once you locate the host, place this host in maintenance and reboot.

Find locked files using vmfsfilelockinfo

Let’s just assume that the vCenter is dead at this time so lets run a find command to locate the VM files

[root@esxlab:~] find / -name CV_CS_Master.vmx
/vmfs/volumes/607c2b1c-13cf66f2-f2f7-b8ac6f19f4ba/CV_CS_Master/CV_CS_Master.vmx

The location of the file is now obtained so let’s browse to the path and run the vmfsfilelockinfo command.

vmfsfilelockinfo takes these parameters:

• File to be tested
• Username and password for accessing VMware vCenter Server (when tracing MAC address to ESX host.)

Open the SSH console, and browse to the datastore location holding the VM. An easy way is to run the touch * command to change the datestamp of all files in this directory and If you have a lock you will get the output as shown.

root@esxlab:/vmfs/volumes/607c2b1c-13cf66f2-f2f7-b8ac6f19f4ba/CV_CS_Master] touch *
touch: CV_CS_Master-flat.vmdk: Device or resource busy
touch: CV_CS_Master_1-flat.vmdk: Device or resource busy
touch: CV_CS_Master_2-flat.vmdk: Device or resource busy

[root@esxlab:/vmfs/volumes/607c2b1c-13cf66f2-f2f7-b8ac6f19f4ba/CV_CS_Master] lsof | grep -i CV_CS_Master


If it returns an output, there is a service or task locking the disks at the moment.

If the above steps don’t work, there is a script file attached to VMware KB 10051 that does the trick.

Removing the Lock

If the VM has no clear process locking the files (no output from the command lsof | grep -i “VM_Name”) then this is a stale process and the locking ESXi Host has to be rebooted to release the lock.

You could else just restart these processes and you should be able to powerup the VM.

# /etc/init.d/hostd stop && /etc/init.d/hostd start
# /etc/init.d/vpxd stop && /etc/init.d/vpxd start

References

https://kb.vmware.com/s/article/84475

https://kb.vmware.com/s/article/10051

https://kb.vmware.com/s/article/2107795