How to detach Provider VDC Resource Pool in VMware Cloud Director

Introduction

Once in a while you will have to remove a Resource Pool (vSphere Cluster in vCenter Server) in VMware Cloud Director (VCD). In this specific case we are using Elastic Provider VDC’s (pVDC’s). This means that a single pVDC can hold several physical backed vSphere Clusters on the back-end environment. You will come across the scenario to remove a Resource Pool for several reasons but primarily while doing lifecycle management. This blogpost will discuss an issue we’ve found while detaching (removing) a Resource Pool from VCD.

Troubleshooting

The issue itself shows when you try to detach a Resource Pool from a pVDC. The error can be seen below:

[ c5ee9dle-f426-426c-803c-xxxxxx ] Unable to detach resource pools) 'moref://xxxxxx-xxxxxx-xxxxxx-xxxxxx-xxxxxx/ResourcePool#resgroup-xxxxxx' because storage policies would no longer be supported. Here are the first 1 (out of 1 storage policies) which would become unsupported: ONBOARDING XXXX-02

The error itself is quite strange. Reason being, the mentioned Storage Policy is not used in the Resource Pool we are trying to remove, it’s from a different Resource Pool.

Looking at the Storage Policy on the pVDC we can see the following:

In this screenshot we can see that the Storage Policy is displaying “Unlimited”, which it should now. That means there is something wrong with the Storage Policy, and not quite with the Resource Pool we are trying to delete.

While looking the Storage Policy up in vCenter Server we can see that the underlying datastores are removed from the policy. Therefor it is an empty Storage Policy. VCD cannot handle that situation and displays the Storage Policy as a policy with “Unlimited” storage due to not having any at all.

We will have to remove the Storage Policy first before we can continue to detach the Resource Pool. Since something was forgotten, let’s start with removing all remaining entries from the back-end. This would mean:

1. Remove the Storage Policy in vCenter Server.
2. Remove the Tag connected to the Storage Policy in vCenter Server.

Once that is done we can remove the Storage Policy in VCD:

Storage policy "ONBOARDING-02" cannot be deleted since it is currently in use.

Or not? Ok it seems it is still in use, however there are no VM’s on the datastores, there aren’t even datastores left for this Storage Policy.

The reason this message is being displayed is because the Storage Policy is still connected to OrgVDC’s. Since this is not easily accessible, I’ve reverse engineerd yet again the VCD database. If you execute the following query on the VCD database it will provide you with a list of OrgVDC’s that has the Storage Policy configured.

select storage_class.id, storage_class.name AS StoragePolicyName, vdc_logical_resource.vdc_id AS OrgVDC_ID, org_prov_vdc.name AS ORGVDC_NAME
from storage_class
INNER JOIN vdc_logical_resource ON storage_class.id = vdc_logical_resource.fo_id
INNER JOIN org_prov_vdc ON vdc_logical_resource.vdc_id = org_prov_vdc.id
where storage_class.name = 'ONBOARDING-02';

                 id                  | storagepolicyname  |              orgvdc_id               |                 orgvdc_name
--------------------------------------+--------------------+--------------------------------------+----------------------------------------------
XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX| ONBOARDING-02 | XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX | VDC-XXXXX-45
XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX | ONBOARDING-02 | XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX | VDC-XXXXX-50

This query can show any number of OrgVDC’s, I’ve left 2 behind as an example. Now you can easily go to the OrgVDC from the VCD GUI -> Remove the Storage Policy on every one and once that is done you can remove the Storage Policy from the pVDC and as a last step detach the Resource Pool from the pVDC.

Nice little tidbit, if you execute the following query you can also find any VM’s where the policy is still connected:

select storage_class.id, storage_class.name AS StoragePolicyName, vm.name AS VM_NAME
from storage_class
INNER JOIN vm on storage_class.id = vm.sclass_id
where storage_class.name ='STORAGE POLICY';

Conclusion

Now that we’ve resolved the issue we can conclude that we need to maintain a correct order in removing items from our infrastructure. If you are removing or lifecylce managing the back-end in a VMware Cloud Director environment the sequence should start at cleaning the VMware Cloud Director inventory and only after this we should move on to the next steps and remove the actual infrastructure in the back-end. This makes sense, but sometimes can be forgotten.

I hope this blogpost helped!