How to fix VMware Cloud Director ‘Index 0 out of bounds for length 0 error’

Introduction

The other day we received a fresh new warning in VMware Cloud Director (VCD) while editing a VM to add more disk space. The request failed with an java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0 message. In this blogpost I will discuss how to fix this issue.

java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
    at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
    at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
    at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
    at java.base/java.util.Objects.checkIndex(Objects.java:372)
    at java.base/java.util.ArrayList.get(ArrayList.java:459)
    at com.vmware.ssdc.backend.dao.impl.VAppVirtualMachineDaoImpl.getStorageContainerValRefForVappVm(VAppVirtualMachineDaoImpl.java:127)
    at jdk.internal.reflect.GeneratedMethodAccessor17558.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:220)
    at com.sun.proxy.$Proxy303.getStorageContainerValRefForVappVm(Unknown Source)
    at com.vmware.ssdc.backend.services.impl.VmManagerImpl.updateVmStorageClassAndVdcComputePolicy(VmManagerImpl.java:3306)
    at jdk.internal.reflect.GeneratedMethodAccessor17557.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
    at com.sun.proxy.$Proxy309.updateVmStorageClassAndVdcComputePolicy(Unknown Source)
    at com.vmware.ssdc.backend.services.impl.VAppManagerImpl.updateVAppVm(VAppManagerImpl.java:4760)
    at jdk.internal.reflect.GeneratedMethodAccessor17556.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
    at com.sun.proxy.$Proxy322.updateVAppVm(Unknown Source)
    at com.vmware.vcloud.vapp.impl.VAppServiceImpl.executeTask(VAppServiceImpl.java:743)
    at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase$1.doInSecurityContext(TaskActivity.java:848)
    at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase$1.doInSecurityContext(TaskActivity.java:843)
    at com.vmware.vcloud.backendbase.management.system.SecurityContextTemplate.executeForOrgAndUser(SecurityContextTemplate.java:49)
    at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase.execute(TaskActivity.java:850)
    at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase.invokeInner(TaskActivity.java:746)
    at com.vmware.vcloud.backendbase.management.system.TaskActivity$TaskActivityBasePhase.invoke(TaskActivity.java:361)
    at com.vmware.vcloud.activity.executors.ActivityRunner.runPhase(ActivityRunner.java:175)
    at com.vmware.vcloud.activity.executors.ActivityRunner.run(ActivityRunner.java:112)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

Troubleshooting

Upon revisiting the VM in VCD we did notice a couple of things that were happening on the VM. First, this only seemed to be an issue with the problematic VM in question, no new VM’s or existing (random) other VM’s in this tenant, or any other tenant I tested had this issue. This is usually good news since that means that only this VM has something wrong with it.

We looked up the VM in the vCenter Server and did notice that the VM was not present in it’s appropriate VCD Resource Pool, which is a bit strange since that should always be the case, if it is not the VM will not be able to be managed. Also, the VM did not have any ‘datastore’ information in the GUI.

Next up, I had a look in the database for the VM, this resulted in the following data:

vcloud=# select * from vm where moref = 'vm-XXXXX';
-[ RECORD 1 ]-------+-------------------------------------------------
id                  | XXXXX-XXXX-XXXX-XXXX-XXXXXX
name                | XXXXX
vc_id               | XXXXX-XXXX-XXXX-XXXX-XXXXXX
sclass_id           | XXXXX-XXXX-XXXX-XXXX-XXXXXX
validity_status     | 1
guest_os            | windows9Server64Guest
is_vrouter          | f
is_published        | t
date_created        | 2023-09-11 20:33:27.164
template_id         |
is_template         | f
guest_cust_vm_id    | XXXXX-XXXX-XXXX-XXXX-XXXXXX
moref               | vm-XXXXX
location_path       | [XXXX-XXXX-06] XXXXXX/XXXXXX.vmx
cloud_uuid          | XXXXX-XXXX-XXXX-XXXX-XXXXXX
instance_uuid       | XXXXX-XXXX-XXXX-XXXX-XXXXXX
creation_status     | RESOLVED
version_number      | 8
suspend_proctype_id |
dstore_moref        |

The output triggered something with me because as you can see, the dstore_moref is empty, as was in the GUI! This should not be empty since this field tells VCD where the VM is located. VCD has many other tables in its database where this information is stored, but the specifically the Virtual Machines this is stored together in the ‘vm’ table. After a bit of digging I found that this is a known issue for specific scenario’s in which VCD cannot provision VM’s anymore that have this reference set to NULL. So we have to do some database editing to fix the issue.

1. First we get the VM information and note down the location_path and id value:

select * from vm where moref = 'vm-XXXXX';

2. Now we find the specific datastore moref value in the VCD database and note it down:

select * from datastore_inv where vc_display_name = 'location_path datastore name from step 1';

3. Now we use this in the following update query to update the information on the VM in the ‘vm’ table (I’ve build in some safeguards so that you can only update the specific VM you are targeting):

update vm set dstore_moref = 'datastore-moref from step 2' where creation_status = 'RESOLVED' and dstore_moref IS NULL and id='vm id from step 1';

4. Now redo step 1 and you will see that the dstore_moref is now no longer empty. Now you can retry your operation in VCD and it should succeed. You will now also see an entry in the datastore field in the VCD UI.

Conclusion

Within VMware Cloud Director (VCD) you might come across this issue that you are not able to edit an existing VM. The reason behind this is unclear, however it might be because of earlier issues on the underlying platform (such as storage or ESXi host issues). VCD seems to have some safeguards present to nullify some database entries (I don’t think this should happen).

However, this is easily fixed with a small database edit for the specific VM. Once this is done you can edit the VM again. I hope this helped!