Duplicate VM UUID because of VMware Cloud Director

Introduction

The other day I received a ticket from a customer that mentioned that they had several VM’s with the same VM UUID. They noticed this while executing some checks on their VM’s. As you can figure out yourselfes, you should not want any duplicate UUID’s in your environment. So while troubleshooting I found the following VMware KB specifically for VMware Cloud Director (VCD) environments. I couldn’t find any specific versions that would be affected, so I figured I’d check this just to be sure. We are running VCD version 10.0.0.1 on the environment that was having the issues described above.

The KB mentions that when you use VMware Cloud Director all virtual machines that are created from a vApp Template receive the same VM UUID. I figured I’d test this. I deployed a VM from my very own vApp Template from within VCD. Once deployed I checked the vCenter event logs to try and figure out what happens.

It turns out, VCD forcefully changes the VM BIOS UUID right after the cloning operation completed, eventhough the cloning operation generates a new fresh UUID. So first up the below event log is for the cloning operation itself:

We can see that at the 1st red dot the new VM BIOS UUID (ending with f05) is configured on the cloned VM. The 2nd red dot is the original VM BIOS UUID (ending with 711). The direct next event in vCenter is the following:

We can now see that VCD changed the VM BIOS UUID directly after the cloning operation was finished. And it changes it back to the original UUID, which belongs to the template. If you want to find the UUID through PowerCLI, you can use the following script:

$VMSource = Get-VM -Name "Source"
$VMSource.ExtensionData.Summary.Config.Uuid

$VMTarget = Get-VM -Name "Target"
$VMTarget.ExtensionData.Summary.Config.Uuid

Like I said before, you can understand that we don’t want duplicate UUID’s on the platform. Looking at the KB I mentioned before it looks like VCD is using this as a default. Which I find odd to be honest. It also mentions that “Starting vCD 9.0 a new optimized implementation of the Move operation is introduced, refer to the KB 2151743  for additional details.”. However, this KB doesn’t work anymore. Wondering what should’ve changed starting from vCD 9.0 I browsed through all vCD 9.0 release notes, but couldn’t find anything that was related to this issue.

The fix

But don’t be afraid, luckily we can fix this. We can change a VCD configuration value on the cell(s) with the “cell-management-tool” to change this default behaviour. We have to use the subcommand “manage-config”. The change itself is very easy. I will guide you through it:

1. Login on the Primary VCD Cell (You can find this through logging into a random VCD Cell on the VAMI) with SSH.
2. Get the current configuration value to check if everything is how it is supposed to be:

$VCLOUD_HOME/bin/cell-management-tool manage-config --lookup --name backend.cloneBiosUuidOnVmCopy
Property "backend.cloneBiosUuidOnVmCopy" has value "1"

3. Change the configuration value with the following command:

$VCLOUD_HOME/bin/cell-management-tool manage-config -n backend.cloneBiosUuidOnVmCopy -v 0

4. Repeat step 2 to doublecheck if the value is now “0”.
5. Now we need to restart all the VCD Cells. Since I have a multi-cell VCD environment, I want to do it without impact on the environment. Since there is no need for a failover, we can do this by executing the following:

Quiesce the Cell:
/opt/vmware/vcloud-director/bin/cell-management-tool -u administrator cell --quiesce true
Put the Cell in maintenance mode:
/opt/vmware/vcloud-director/bin/cell-management-tool -u administrator cell --maintenance true 
Clean shutdown the Cell:
/opt/vmware/vcloud-director/bin/cell-management-tool -u administrator cell --shutdown
Doublecheck the Cell state (Result should be: State: Inactive, Maintenance: True):
/opt/vmware/vcloud-director/bin/cell-management-tool -u administrator cell -t
Check the "vmware-vcd" service state (it should be inactive):
service vmware-vcd status
Start the "vmware-vcd" service:
service vmware-vcd start
Check the "vmware-vcd" service state (it should be active):
service vmware-vcd status
Doublecheck the VCD Cell state again with:
/opt/vmware/vcloud-director/bin/cell-management-tool -u administrator cell -t

At this point the configuration value is changed. Now I tried the same procedure as before to validate the change we made to the Cell(s). This time however the VM BIOS UUID was different from the template. So a great success. Something that I later noticed in the KB is that it is recommended to change this value especially on vSAN environments. Since the vSAN Health check has a requirement for unique VM BIOS UUID’s. Duplicate ID’s on a vSAN environment will give false positives in the “vsan.check_state find_inconsistent_vms” health check.

This will be all for today. I hope this helped. I hoped it wouldn’t be needed since the KB doesn’t mention a VCD version. But it seems that this “issue” is still present on 10.0.0.1.