Virtual Machine BootTime property is not being populated

Ever so often you come around something that you haven’t seen before. Well this issue is one of those things. In our environments we use loads of metrics and properties that we gather from the vCenter Server and expose them to the customer in our own custom front-end portal. One of these properties is the following VirtualMachine “BootTime” property. This property is located in the VirtualMachineRuntimeInfo(vim.vm.RuntimeInfo) data object and provides the requestor with a timestamp on which the Virtual Machine was recently powered on. It is updated when the virtual machine is powered on from a hard “Powered Off” state and it does not get updated if you resume a Virtual Machine from a suspended state.

This property can be requested in the following way with PowerCLI:

Get-VM  -Name "VMNAME" | Select Name, @{N="BootTime";E={$_.ExtensionData.Summary.Runtime.BootTime}}

This will provide you with the following output:

Get-VM  -Name "VMNAME" | Select Name, @{N="BootTime";E={$_.ExtensionData.Summary.Runtime.BootTime}}

Name                            BootTime
----                                --------
VMNAME                   7-8-2020 05:49:57

So looking at the above piece of output we can see that the VM called “VMNAME” was last booted on 7-8-2020 05:49:57 AM. This information is very valuable for our customers, with this piece of information they can for example quickly see if there’s been a disruption or a reboot on the VM’s.

Unfortunately in one our environments the property is not being populated on VM’s anymore. This seemed odd to me because it was being populated on some VM’s, but yet again on some it wasn’t. All of these VM’s were in the same cluster, even on the same host. I’ts really easy to look up VM’s that don’t have the property populated with the following script:

Get-VM  | ?{$_.PowerState -eq "PoweredOn"} |?{$_.ExtensionData.Summary.Runtime.Boottime -eq $NULL} | Select Name, @{N="BootTime";E={$_.ExtensionData.Summary.Runtime.BootTime}}

This piece will provide you a list of VM’s that have nothing stored in the property. After troubleshooting with VMware GSS we found out that this issue is a known problem, without any KB’s or any public documentation, but there is a PR within VMware. I can confirm that this issue is not because of PowerCLI (any version for that matter), but rather due to a bug in the ESXi version we are running. This version would be VMware ESXi, 6.7.0, 16075168. This is the second to latest ESXi 6.7 build. VMware also confirmed it is not fixed within the latest ESXi build with version 16316930.

The circumstances as to why this issue is happening only on one environment rather than on all, are unknown (even to VMware). But the fix is rather easy untill VMware brings out a patch on ESXi 6.7. If you want to populated the property again you will simply have to do a Guest OS Reboot, Reset or PowerOff continued with a PowerOn action. Remember the first part of the blog where I explained how this property gets populated? The mentioned actions will simply reset the value.

This is unfortunate if the VM’s are production VM’s that cannot be easily rebooted, but hey at least there is a “fix”.

That would be all for now guys! I hope this helped.