VMware Cloud Director datastore thresholds prevent deploying more virtual machines

Today I want to talk about an issue we’ve encountered within our VMware Cloud Director (VCD) environment before. I figured it might be usefull for everybody that is looking at this problem in their environment.

The situtation is as follows: You or your customers are trying to extent an existing disk or trying to add a fresh new disk to a Virtual Machine within VCD. This fails and provides you with the following error in the UI, it actually doesn’t log any failed tasks, since the task has not been initiated yet. This error comes from the pre-check that VCD does to ensure that you are using the correct environment and policies that allow you to edit or create a disk on a Virtual Machine.

[ 9615a150 ] One or more exceptions have occurred
 - The operation failed because no suitable resource was found. Could not find a suitable datastore in hub "moref://aabb9ff2/ResourcePool#resgroup-27"; cause(s): There is no usable datastore.
Cannot use datastore [vcId=aabb9ff2-653d-4202-8961-fc6e0aa314f1, moref=datastore-34932] for subject [type:VIRTUAL_MACHINE, URI:03e85ed7-e2ab-45a9-9bb2-35fc9f4e1711] because the container reached RED threshold of available capacity.

This pre-check is actually done many times, on each and every edit on the Virtual Machine that is! The above error message is explaining to us that there are no available datastores to deploy a new disk on, or to edit the current disk on because the datastore has reached a certain threshold. If you didn’t know that VCD enforces it’s own thresholds within the portal to ensure that the environment is not filled up to 100% and causes impact, you know it now! Fortunately these can easily be tuned for us Provider users.

You should be able to edit the thresholds if you go to the datastores in the VCD UI -> Provider VDCs -> Select a pVDC -> Storage Containers. In this window you will see something like this:

Now you don’t see an edit button. In certain VCD 10.2.x builds there is a bug which prevents you from clicking the datastore name. This is only present if you are using Chrome (Bug page). If you are having this issue, you can use any other browser. If you don’t have this bug, you can click on the datastore and the following page comes up:

On this page there are two categories “Threshold Yellow” and “Threshold Red”. By default the Yellow threshold is 25% of the capacity of the datastore. The Red threshold is 15% by default. You can also have a look at how this works at the following page. The way this works is that once a datastore reaches the 25% (Yellow) stage VCD starts to e-mail warnings to the configured e-mail adres. Once the threshold reaches the 15% (Red) stage, the Virtual Machine placement engine stops working for this datastore and no more Virtual machines can be placed on it.

Now that you know this, you can probably already see what is happing here. We have seperate capacity thresholds on the back-end environment which we monitor and act upon. VCD has other thresholds configured by default which are far more than what we use. So we’ve edited this to our own thresholds and pressed on Save, such as below:

There you have it. Once you have saved the new configuration you and your customers can deploy new disks/Virtual machines again.