During an issue I had with my lab environment while using the ‘cmsso-util’ command to join two of my lab vCenters to the same SSO domain, I also had another issue which I will talk about in this blog post. Once I fixed the previous issue, which was due to the fact that my vCenter was rolled out in another timezone which caused some chicken and egg problems in regards to certificates, I encountered the following issue:

This was the output after using the ‘cmsso-util’ command below:

Just as in the last post I talked about earlier, we can find logs for the ‘cmsso-util’ command in the following location: /var/log/vmware/cloudvm/domain_consolidator.log. If we go back and open this up we could read the following log snippet:

After looking at the log snippet I saw that it was referencing another log file for some more logs. Since this one wasn’t especially useful I tried that one, you can open this log file which is in the following location: /storage/log/vmware/cloudvm/domain_data_export.log

Well OK now it makes sense. It looks like we have an invalid STS certificate registration on our vCenter Server, which is causing the issue on this environment.

The Resolution

Now that we know that the vCenter STS certificate registration is invalid, we can replace it. At first I tried the following VMware KB, more on this in a later blogpost, because I knew about this from previous activities. Since this was a brand new vCenter, I did not get my hopes up though. Once I used the scripts to check and replace my STS certificates (eventhough they were not expired) the result was still the same while executing the ‘cmsso-util’ command.

Next up I remembered my previous blogpost, on misplaced sslTrust Anchors in the vCenter lookup service, and figured that I would try and check if these anchors were still good. Especially since I just replaced all the certificates on the vCenter Server. A complete guide on how to do this can be found in the following VMware KB, but I will also guide you throug this process. Sometimes it happens that when updating the certificates on a vCenter Server some solutions don’t get properly updated, which means that the sslTrust anchors are broken. These need to be restored and to do this you would need to executing the following list of actions:

  1. Create a new folder called “certificates” somewhere on the vCenter Server.
  2. Validate the sslTrust Anchors for the vCenter lookupservice against what is used in the solution vs the vCenter Server.
    1. Login to the vCenter Server through SSH and enter shell.set --enabled true.
    2. Run the following command to receive the current sslTrust anchor: /usr/lib/vmidentity/tools/scripts/lstool.py list --url https://localhost/lookupservice/sdk --no-check-cert --ep-type com.vmware.cis.cs.identity.sso 2>/dev/null
    3. This will give you something like the code block below. Please note I have shortened the SSLtrust string:
  1. Now that we have this, we can run the following command to look up the current installed vCenter Server certificate: echo | openssl s_client -connect localhost:443.
  2. Compare the sslTrust strings from step 2 and step 3. If they are the same, this blogpost is not for you. If they are not the same, like on my environment, continue to step 5.
  3. Now we need to retrieve the old STS certificate so that we can create a SHA1 thumbprint from it. Login to the vCenter MOB service on the following example URL: https://vc_with_embedded_psc.example.com/lookupservice/mob?moid=ServiceRegistration&method=List
    1. In the “filterCriteria” field you need to delete everything so that it displays the following tags <filterCriteria></filterCriteria> and press “Invoke”:
    2. Lookup the value array where the Name = URL, Type = anyURI and Value = https://vc_with_embedded_psc.example.com:443/sdk
    3. Note down the Name = sslTrust, Type = ArrayofString and Value = Certificate SSL trust value. We need this value in the next step. This is usually (as to what I’ve seen in the field), for the “com.vmware.cis.cs.identity.sso” registration.
    4. Copy this value inside a .crt file on the vCenter Server in the folder we previously created and append -----BEGIN CERTIFICATE----- before the string, and -----END CERTIFICATE----- after the string. Also make sure you enter a carriage return after the 64th character. I found it easier just to copy the SSLtrust string from step 3, which is already encoded correctly. Just make sure you append the BEGIN/END certificate piece. Save the file as a .crt file.
    5. Now that we have the old certificate, we can extract a SHA1 thumbprint from it, enter the following command: openssl x509 -in old_machine.crt -noout -sha1 -fingerprint
  4. Now we need the new certificate that is currently connected to the vCenter Server. We can get this by executing the following command: /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT
  5. Export the certificate with the following command: /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /root/certificates/new_machine_cert.crt

Next up is the last step in this process. Make sure you have the following information present:

  • The SHA1 thumbprint from the old certificate (Step 1 through 5 above).
  • The currently used (new) certificate on the vCenter Server (Step 6 and 7 above).
  • The vCenter Server lookupservice address (Should be https://vc.example.com/lookupservice/sdk).
  • The administrator@vsphere.local password.

Please note that the following part will replace the STS certificate with the new certificate throughout your entire vCenter Server and all 34 services. So please make sure you are executing this on the correct vCenter Server.

  1. For this step we need to use the ls_update_certs script which is located at /usr/lib/vmidentity/tools/scripts/. Use the following command syntax to replace the STS certificate on all registered solutions.

The script will run for a couple of minutes, and it will end with the following piece of information:

Now run step 1 through 4 again and validate if the two sslTrust Anchors are the same. They should be. Once I did this I tried to use the ‘cmsso-util’ command again and this time it worked flawlessly. Once it works you should receive the following output:

This was all for today. I hope this helped you guys. Once we found out that the STS certificate registration was invalid, it was a pretty straight forward approach to fix this. This will also be the last blogpost for 2020!

Leave a Reply

Your email address will not be published. Required fields are marked *