Introduction
Starting with VMware Cloud Director (VCD) 10.5 VMware made a new construct possible within the product. A replication tracking (RT) VM object. This is something that users that have seen VMware Cloud Director Availability (VCDA) do and use within VCD. However, there is not much documentation online that explains how to interact with these replication tracking objects or how it works. The official documentation does tell us what a RT VM is:
A replication tracking virtual machine (RT VM) represents a shadow VM for replication placement that allows per-disk placement. An RT vApp is a vApp that consists of RT VMs. As a provider, you can configure the placement solution of the existing vApp and VM replications to RT VM.
Since VMware Cloud Director Availability 4.7, when using a supported version of VMware Cloud Director in the destination site, then RT VM replaces the independent disk as the placement and reporting solution going forward. RT VM is a shadow VM residing in the destination VMware Cloud Director that represents the replication. Performing recovery promotes the shadow VM as the workload VM. This recovery process is faster and more robust, allowing for lower RTO. The RT vApp is composed at configuration time instead of recovery time. Also, the recoveries use reload from path, instead of registering a new VM in vCenter, allowing for better RTO and resilience.
A replication that uses RT VM allows placing the replica files in separate data stores for each disk and for the VM itself.
So in other words a RT VM is an object that is essentially invisible until the application that uses this technique make it visible to use in failover scenario’s. The initial implementation for this piece is something I really enjoyed because this would replace the original ‘independent-disk’ replication technique used together with VCD. It made sure that most of the VCD interaction was done before the actual (test)failover took place. Why is this good? Well you want the least amount of interactions to any solution if you do a failover. You would want to have the least amount of possible areas where things can go wrong, once everything is wrong already. Hence it would reduce the RTO during a failover.
Nothing is wrong with this, however, there is no real documentation online how to work with these invisible VM’s once you need to. For example in situations where you would want to delete the invisible VM.
Information to use
Originally this concept was also introduced into VCD to 1. remove the “C4” objects that VCDA creates in the vCenter Server and 2. Enhance the way VCD got VCDA Replication storage usage to VCD without blasting a lot of API calls to the vCenter Server. Now since these objects are essentially invisible, even for providers, why would you need to read this blogpost? Well:
- Invisible or not, the VM’s are still present in Organization VDC’s and on the platform. You can’t browse to them in the VCD UI, but you can access them from the API (Which I will explain in a bit).
- Since they are present in the Organization VDC and on the platform, there might be a scenario in which you want to delete an Organization VDC, but cannot since there are still objects left that cannot be removed. (This was the case for me)
- If you are really curious, I will show how you can see them inside the VCDA databases.
Let’s start with number 1. RT VM’s are invisible from Tenants (on the UI and API), invisible from Providers (on the UI), but visible through the API. RT VM’s are essentially copies from the original vApp. They have the same options and names, however there is a postfix with “(1)” in the name with which you can originally spot them. Let’s first give an example:
Looking at the screenshot above one might think there is a VM and a vApp inside the Organization VDC. However:
As you can see, no actual vApps are left. In this case there is only the invisible RT VM object left. You can see this as a provider with the API:
- Login through the API as a provider.
- List the Organization VDC with the following command
GET - https://vcd.fqdn/api/admin/vdc/ORGVDC-URN
You will find a <ResourceEntity> section in the response. In this section the VM that is left will be present:
<ResourceEntities>
<ResourceEntity href="https://vcd.fqdn/api/vApp/vapp-xxxx-xxxx-xxxx-xxxx-xxxxxxxx" id="urn:vcloud:vapp:xxxxx-xxxx-xxxxx-xxxx-xxxxxx" type="application/vnd.vmware.vcloud.vApp+xml" name="VCDA-Zerto-1 (1)"/>
</ResourceEntities>
You can see the name for it. In this part you can already see what I mentioned before. The RT VM object will have a ‘(1)’ in the name at the end. Now you can go one lever deeper and open up this VM with the following API call:
GET - https://vcd.fqdn/api/vApp/vapp-xxxxx-xxxx-xxxxx-xxxx-xxxxxx
Now the second part in which you can identify if this is actually a RT VM object and not a regular VM is through the properties you will get back with the previous API call. If you scroll all the way to the bottom, you will find the following information:
<ReplicationTrackingDiskSection>
<ReplicationTrackingDiskSettings>
<DiskUuid>xxxxx-xxxx-xxxxx-xxxx-xxxxxx</DiskUuid>
<StorageProvisionedKb>10485760</StorageProvisionedKb>
<StorageOverheadKb>0</StorageOverheadKb>
<StoragePolicy href="https://vcd.fqdn/api/vdcStorageProfile/xxxxx-xxxx-xxxxx-xxxx-xxxxxx" id="urn:vcloud:vdcstorageProfile:xxxxx-xxxx-xxxxx-xxxx-xxxxxx" type="application/vnd.vmware.vcloud.vdcStorageProfile+xml" name="VDC Storage"/>
<PlacementRecommendation>moref://xxxxx-xxxx-xxxxx-xxxx-xxxxxx/Datastore#datastore-xxxxx</PlacementRecommendation>
</ReplicationTrackingDiskSettings>
<ReplicationTrackingDiskSettings>
<DiskUuid>xxxxx-xxxx-xxxxx-xxxx-xxxxxx</DiskUuid>
<StorageProvisionedKb>62914560</StorageProvisionedKb>
<StorageOverheadKb>0</StorageOverheadKb>
<StoragePolicy href="https://vcd.fqdn/api/vdcStorageProfile/xxxxx-xxxx-xxxxx-xxxx-xxxxxx" id="urn:vcloud:vdcstorageProfile:xxxxx-xxxx-xxxxx-xxxx-xxxxxx" type="application/vnd.vmware.vcloud.vdcStorageProfile+xml" name="VDC Storage"/>
<PlacementRecommendation>moref://xxxxx-xxxx-xxxxx-xxxx-xxxxxx/Datastore#datastore-xxxxx</PlacementRecommendation>
</ReplicationTrackingDiskSettings>
</ReplicationTrackingDiskSection>
<ReplicationTrackingSettings>
<ReplicationTrackingVmState>REPLICATION_MODE</ReplicationTrackingVmState>
</ReplicationTrackingSettings>
For each disk on the original VM, and therefor the RT VM object there is an <ReplicationTrackingDiskSettings> section present. The fact that this information is present at all is already enough to see it as an RT VM object. Regular VM’s don’t have this section. Since this is the vAPP you can go another level to see the children of this object, also called the VM’s. You can find what VM’s are in the vApp in the <Children> section:
<Children>
<Vm encrypted="false" needsCustomization="false" nestedHypervisorEnabled="false" deployed="false" status="8" name="TestVMBVE2" id="urn:vcloud:vm:xxxxx-xxxx-xxxxx-xxxx-xxxxxx" href="https://vcd.fqdn/api/vApp/vm-xxxxx-xxxx-xxxxx-xxxx-xxxxxx" type="application/vnd.vmware.vcloud.vm+xml">
<VCloudExtension required="false">
Now execute the following API call to get the VM information:
GET - https://vcd.fqdn/api/vApp/vm-xxxxx-xxxx-xxxxx-xxxx-xxxxxx
Now if you don’t want to do much more, you can go ahead and do the same API call, but now instead of a GET you do a DELETE. This will remove the RT VM from VCD, remove the VM object in vCenter and remove the vAPP folder in vCenter. After this you can remove your Organization VDC.
BONUS
Now for the first bonus. If you have a close look in the RT VM API call you can find all sorts of other actions you can do with the RT VM:
<Link rel="changeReplicationState" href="https://vcd.fqdn/api/vApp/vm-xxxxx-xxxx-xxxxx-xxxx-xxxxxx/action/changeReplicationTrackingSettings" type="application/vnd.vmware.vcloud.replicationTrackingSettings+xml"/>
<Link rel="changeReplicationState" href="https://vcd.fqdn/api/vApp/vm-xxxxx-xxxx-xxxxx-xxxx-xxxxxx/action/changeReplicationTrackingSettings" type="application/vnd.vmware.vcloud.replicationTrackingSettings+json"/>
<Link rel="promoteReplicationTrackingVm" href="https://vcd.fqdn/api/vApp/vm-xxxxx-xxxx-xxxxx-xxxx-xxxxxxaction/failover" type="application/vnd.vmware.vcloud.vmFailoverParams+xml"/>
<Link rel="promoteReplicationTrackingVm" href="https://vcd.fqdn/api/vApp/vm-xxxxx-xxxx-xxxxx-xxxx-xxxxxx/action/failover" type="application/vnd.vmware.vcloud.vmFailoverParams+json"/>
To be completely honest I don’t even know how to use these, and since no official working documentation is to be found on the web, I will have to figure that out later. But it is quite obvious that these are used by VCDA to promote an invisible RT VM to an actual VM when a (test)failover takes place.
If you found the RT VM and would want to see some VCDA database information from them you can do that by doing the following:
- Login to the destination VCDA Manager appliance through a SSH session.
- Enter the
clouddatabase with the command:psql -U h4postgres h4cloud - List the VM with the following command:
select * from vmreplication where recovered_vm_moref = 'vm-XXXX';- You can find the vm_moref in the RT VM API Call.
h4cloud=# select * from vmreplication where recovered_vm_moref = 'vm-XXXX'; -[ RECORD 1 ]----------------------+-------------------------------------------- id | C4-xxxxx-xxxx-xxxxx-xxxx-xxxxxx created_at | 2024-08-06 15:21:18.476 datastore | xxxxx-xxxx-xxxxx-xxxx-xxxxxx datastore_moref | datastore-XXXXXX independent_disk_vm_moref | is_reversed | f last_updated | 2024-08-20 18:23:27.562 manager_replication_id | H4-xxxxx-xxxx-xxxxx-xxxx-xxxxxx owner | XXXXXXX@Availability primary_nic_index | 2 recovered_vm_id | xxxxx-xxxx-xxxxx-xxxx-xxxxxx recovered_vm_moref | vm-XXXXX recovery_state | COMPLETE source_vim_server_uuid | xxxxx-xxxx-xxxxx-xxxx-xxxxxx source_vm_moref | vm-XXXX start_action | powerOn start_delay | 0 start_order | 0 stop_action | powerOff stop_delay | 0 storage_profile | xxxxx-xxxx-xxxxx-xxxx-xxxxxx vim_server | xxxxx-xxxx-xxxxx-xxxx-xxxxxx vim_server_uuid | xxxxx-xxxx-xxxxx-xxxx-xxxxxx vm_description | admin_password | admin_password_auto | admin_password_enabled | change_sid | computer_name | customization_script | domain_name | domain_user_name | domain_user_password | enabled | join_domain_enabled | reset_password_required | use_org_settings | vm_id | vm-XXXXX vm_name | BVETESTVM sla_profile_id | v_app_replication_id | C4VAPP-xxxxx-xxxx-xxxxx-xxxx-xxxxxx encrypted_source_vm | f placement_policy | recovered_vm_template_id | recovered_vm_template_moref | sizing_policy | admin_auto_logon_count | domain_org_unit | source_template_overwrite_tracking | overwrite_destination_template | tracking_start_time | tracking_stop_time | source_vm_template_id | rt_vm_id | xxxxx-xxxx-xxxxx-xxxx-xxxxxx use_rt_vm | t recovered_storage_profile | xxxxx-xxxx-xxxxx-xxxx-xxxxxx copy_security_tags |
There is all kinds of information in this place that might be useful in troubleshooting scenario’s. You can find the source VM, source Organization VDC, VCDA Manager that is managing this replication and a lot more.
If you use the v_app_replication_id property from this table in the following command, you can find the vApp replication information:select * from vappreplication where id = 'C4VAPP-xxxxx-xxxx-xxxxx-xxxx-xxxxxx';
h4cloud=# select * from vappreplication where id = 'C4VAPP-xxxxx-xxxx-xxxxx-xxxx-xxxxxx'; -[ RECORD 1 ]------------------+-------------------------------------------- id | C4VAPP-xxxxx-xxxx-xxxxx-xxxx-xxxxxx destination_org | ORG-XXXXX destination_vdc_id | xxxxx-xxxx-xxxxx-xxxx-xxxxxx destination_vdc_name | VDC-1111 recoveredvapp_id | xxxxx-xxxx-xxxxx-xxxx-xxxxxx is_migration | t last_updated | 2024-08-20 18:23:27.562 owner | ORG-XXXXX@Availability source_org | source_vdc_id | source_vdc_name | v_app_id | BVETESTVM description | name | BVETESTVM version | 2 destination_site | Availability source_site | Systemec data_engine | HBRSRV is_template | f recoveredvapp_template_id | catalog_id | catalog_name | enabled_template_customization | is_golden | source_catalog_item_id | source_catalog_id | source_catalog_name | destination_catalog_item_id | replication_start_time | 2024-08-06 15:21:18.393 rtvapp_id | xxxxx-xxxx-xxxxx-xxxx-xxxxxx
You will find all sorts of information. You can also use this to troubleshoot the other way around. For example you can’t find the RT VM through the API, but you do think you relate back information from this table back to a specifiek Organization VDC to still find the RT VM object.
Conclusion
This new RT VM object that is used for a couple of enhancements in VCD can be quite useful, but they can also provide some issues because the objects are essentially invisible. I hope this blogpost will help you with your troubleshooting if it is needed!
0 Comments