In my previous blogpost where we deleted stuck vSAN objects due to a vSAN bug in the remediation workflow I mentioned the use of the ‘objtool’ to delete objects from the vSAN filesystem. In that specific blogpost I had issues removing the objects, even while using the ‘force’ parameter. For this blogpost we wanted to remove some ‘Unknown object type’ objects which we were seeing in the vSphere Client. In the next part of the blogpost will guide you through what eventually turned out to be old vSAN.stats folders.

The issue and solution

As you can see below we had some leftover ‘Unknown object type’ objects in the Virtual Objects tab in the vSphere Client that were not known to us.

vSAN Cluster Unknown Object type objects
vSAN Cluster Unknown Object type objects

While having a quick look into the file browser in the vSAN datastore I noticed a lot of ‘vsan.stats’ folders.

vSAN Cluster old '.vsan.stats' folders.
vSAN Cluster old ‘.vsan.stats’ folders.

These folders are created to hold the vSAN Statistics database files, these files contain the data analyzed and collected by the vSAN Performance Service. This data is the data that can be displayed within the vSphere Client on the ‘Performance’ Tab under the ‘vSAN’ tab on the cluster.

I’ve read a couple of mentions that disabling the service under Cluster -> Configure -> vSAN Services -> Performance Service would remove any folder remaining on the vSAN datastore. We tried this but unfortunately this didn’t work. When all else fails, let’s pull out the ‘objtool’ command. This command allows users to execute tasks against the objects present on the object based vSAN datastore. This command is also the most dangerous in that case because once you delete something, or alter it incorrectly, you can potentially corrupt your object (and therefor VM).

However, if you know what you can do with it you can use it to for example remove stuck or unknown objects from vSAN. Let’s dive into the objects themselfes first before we do anything:

[root@esx10-am:~] /usr/lib/vmware/osfs/bin/objtool getAttr -u xxxxxx-348d-275e-964d-b88303594c10
Object Attributes --


Object type:vsan

Object size:273804165120

Allocation type:Thin

Policy:((\"stripeWidth\" i1) (\"cacheReservation\" i0) (\"proportionalCapacity\" (i0 i100)) (\"hostFailuresToTolerate\" i1) (\"forceProvisio\" \"xxxxxxxx-c388-4c51-a7df-xxxxxxxx\") (\"spbmProfileGenerationNumber\" l+0) (\"storageType\" \"Allflash\") (\"replicaPreference\" \"C0) (\"checksumDisabled\" i0) (\"spbmProfileName\" \"vSAN Default Storage Policy Perfomance Service\"))

Object class:vmnamespace

Object capabilities:NONE

Object path:/vmfs/volumes/vsan:xxxxxxxxx-9318dee8c8ec30dc/.vsan.stats

Group uuid:00000000-0000-0000-0000-000000000000

Container uuid:00000000-0000-0000-0000-000000000000


User friendly name:.vsan.stats

HA metadata:(null)

Filesystem type: vmfs6d

As you can see in the below piece of output, we asked the ‘objtool’ to display the object known by one of the UUID’s from the ‘Unknown Object Types’. The tool will display some information and in our case it mentions the specific object path by which we can see that this is indeed an old ‘.vsan.stats’ folder with files in it. Now that we know that this is old information (by disabling the Performance Service to remove the current files) we can go ahead and remove them with the following command (beware, this does permanently remove the object!):

/usr/lib/vmware/osfs/bin/objtool delete -u xxxxxxx-348d-275e-964d-b88303594c10

This command will not give you any feedback. Once executed in our case on most ‘Unknown Object Type’ objects our list was clean once more! The ‘Unknown Object Type’ objects that were left were part of the Zerto installation. Zerto places a couple of files on the datastore which are not connected to any VM’s. If this is the case, vSAN also marks them as ‘Unknown Object Type’ objects.

Once again, be careful executing commands with the ‘objtool’ on a vSAN Cluster. You can seriously break your environment with this. However, in our case we were succesful in removing unknown objects.

Categories: VMware

Bryan van Eeden

Bryan is an ambitious and seasoned IT professional with almost a decade of experience in designing, building and operating complex (virtual) IT environments. In his current role he tackles customers, complex issues and design questions on a daily basis. Bryan holds several certifications such as VCIX-DCV, VCAP-DCA, VCAP-DCD, V(T)SP and vSAN and vCloud Specialist badges.


Leave a Reply

Avatar placeholder

Your email address will not be published. Required fields are marked *