Fixing an invalid vCenter STS certificate while using ‘cmsso-util’

During an issue I had with my lab environment while using the ‘cmsso-util’ command to join two of my lab vCenters to the same SSO domain, I also had another issue which I will talk about in this blog post. Once I fixed the previous issue, which was due to the fact that my vCenter was rolled out in another timezone which caused some chicken and egg problems in regards to certificates, I encountered the following issue:

Starting License pre-check                                                      ... Done
Starting Tagging Data export                                                   ... Failed
Conflict data, if any, can be found under /storage/domain-data/Conflict*.json
Pre-checks failed.

This was the output after using the ‘cmsso-util’ command below:

cmsso-util domain-repoint -m pre-check --src-emb-admin "administrator" --replication-partner-fqdn "vc01" --replication-partner-admin administrator --dest-domain-name "vsphere.local"

Just as in the last post I talked about earlier, we can find logs for the ‘cmsso-util’ command in the following location: /var/log/vmware/cloudvm/domain_consolidator.log. If we go back and open this up we could read the following log snippet:

root@vc02 [ /storage ]# cat /var/log/vmware/cloudvm/domain_consolidator.log
2020-12-07T11:07:13.939Z INFO domain_consolidator Tagging service script running in pre-check mode.
2020-12-07T11:07:13.939Z INFO domain_consolidator Starting Tagging service export phase...
2020-12-07T11:07:13.940Z INFO domain_consolidator Starting Tagging Data export                                                    ...
2020-12-07T11:07:13.940Z INFO domain_consolidator Starting required services...
2020-12-07T11:07:13.940Z INFO domain_consolidator Executing command ['/bin/service-control', '--start', 'vmafdd', 'vmware-rhttpproxy', 'vmware-vpxd-svcs', 'vmware-vpostgres']
2020-12-07T11:07:14.245Z INFO domain_consolidator Started required services.
2020-12-07T11:07:15.762Z INFO domain_consolidator RC = 1
Stderr = Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M
-Dcom.sun.org.apache.xml.internal.security.ignoreLineBreaks=true
-Dorg.apache.xml.security.ignoreLineBreaks=true
Exception in thread "main" java.lang.Exception: QueryClient creation failed for VC:vc02 Check 'domain_data_export.log'
        at com.vmware.vim.dataservices.ExportTaggingData.main(ExportTaggingData.java:228)

2020-12-07T11:07:15.762Z INFO domain_consolidator Export of Tagging Data failed. Exception {
    "detail": [
        {
            "translatable": "An error occurred while invoking external command : '%(0)s'",
            "localized": "An error occurred while invoking external command : 'Stderr: Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M\n-Dcom.sun.org.apache.xml.internal.security.ignoreLineBreaks=true\n-Dorg.apache.xml.security.ignoreLineBreaks=true\nException in thread \"main\" java.lang.Exception: QueryClient creation failed for VC:vc02. Check 'domain_data_export.log'\n\tat com.vmware.vim.dataservices.ExportTaggingData.main(ExportTaggingData.java:228)\n'",
            "id": "install.ciscommon.command.errinvoke",
            "args": [
                "Stderr: Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M\n-Dcom.sun.org.apache.xml.internal.security.ignoreLineBreaks=true\n-Dorg.apache.xml.security.ignoreLineBreaks=true\nException in thread \"main\" java.lang.Exception: QueryClient creation failed for VC:vc02. Check 'domain_data_export.log'\n\tat com.vmware.vim.dataservices.ExportTaggingData.main(ExportTaggingData.java:228)\n"
            ]
        }
    ],
    "resolution": null,
    "componentKey": null,
    "problemId": null
}
2020-12-07T11:07:15.762Z INFO domain_consolidator  Failed
2020-12-07T11:07:15.762Z INFO domain_consolidator Export of Tagging data failed
2020-12-07T11:07:15.822Z INFO domain_consolidator Failed to execute script /usr/lib/repoint/taggingservice_component_script.py
2020-12-07T11:07:15.823Z INFO domain_consolidator Conflict data, if any, can be found under /storage/domain-data/Conflict*.json
2020-12-07T11:07:15.823Z INFO domain_consolidator Failed executing <cis.component_data.DcComponentsPreCheck object at 0x7f22704a7630>
2020-12-07T11:07:15.823Z ERROR domain_consolidator Failed to run pre-checks for domain consolidation.
2020-12-07T11:07:15.823Z INFO domain_consolidator Cleaning up the temp directories
2020-12-07T11:07:15.823Z INFO domain_consolidator Successfully cleaned the storage directory
2020-12-07T11:07:15.823Z INFO domain_consolidator Pre-checks failed.

After looking at the log snippet I saw that it was referencing another log file for some more logs. Since this one wasn’t especially useful I tried that one, you can open this log file which is in the following location: /storage/log/vmware/cloudvm/domain_data_export.log

root@vc02 [ /storage ]# cat /storage/log/vmware/cloudvm/domain_data_export.log
2020-12-07T12:07:15.752+01:00 [main  DEBUG com.vmware.vim.sso.client.impl.ssl.StsSslTrustManager  opId=] The SSL certificate of STS service cannot be verified against the list of client-trusted certificates
sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:450)
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:317)
        at sun.security.validator.Validator.validate(Validator.java:262)
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330)
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:235)
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:113)
        at com.vmware.vim.sso.client.impl.ssl.StsSslTrustManager.checkServerTrusted(StsSslTrustManager.java:112)
        at sun.security.ssl.AbstractTrustManagerWrapper.checkServerTrusted(SSLContextImpl.java:1099)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1671)
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:226)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1082)
        at sun.security.ssl.Handshaker.process_record(Handshaker.java:1010)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1079)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1388)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1416)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1400)
        at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1340)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1315)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:264)
        at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:104)
        at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:208)
        at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:130)
        at com.sun.xml.internal.ws.transport.DeferredTransportPipe.processRequest(DeferredTransportPipe.java:124)
        at com.sun.xml.internal.ws.api.pipe.Fiber.__doRun(Fiber.java:1121)
        at com.sun.xml.internal.ws.api.pipe.Fiber._doRun(Fiber.java:1035)
        at com.sun.xml.internal.ws.api.pipe.Fiber.doRun(Fiber.java:1004)
        at com.sun.xml.internal.ws.api.pipe.Fiber.runSync(Fiber.java:862)
        at com.sun.xml.internal.ws.client.Stub.process(Stub.java:448)
        at com.sun.xml.internal.ws.client.dispatch.DispatchImpl.doInvoke(DispatchImpl.java:250)
        at com.sun.xml.internal.ws.client.dispatch.DispatchImpl.invoke(DispatchImpl.java:289)
        at com.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage(SoapBindingImpl.java:161)
        at com.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage(SoapBindingImpl.java:114)
        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.java:927)
        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.executeRoundtrip(SecurityTokenServiceImpl.java:856)
        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl.acquireToken(SecurityTokenServiceImpl.java:144)
        at com.vmware.vim.dataservices.ExportImportUtils.getBearerToken(ExportImportUtils.java:136)
        at com.vmware.vim.dataservices.ExportImportUtils.getQueryClientFromLS(ExportImportUtils.java:850)
        at com.vmware.vim.dataservices.ExportImportUtils.createClient(ExportImportUtils.java:266)
        at com.vmware.vim.dataservices.ExportTaggingData.main(ExportTaggingData.java:222)
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
        at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
        at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:445)
        ... 40 more

Well OK now it makes sense. It looks like we have an invalid STS certificate registration on our vCenter Server, which is causing the issue on this environment.

The Resolution

Now that we know that the vCenter STS certificate registration is invalid, we can replace it. At first I tried the following VMware KB, more on this in a later blogpost, because I knew about this from previous activities. Since this was a brand new vCenter, I did not get my hopes up though. Once I used the scripts to check and replace my STS certificates (eventhough they were not expired) the result was still the same while executing the ‘cmsso-util’ command.

Next up I remembered my previous blogpost, on misplaced sslTrust Anchors in the vCenter lookup service, and figured that I would try and check if these anchors were still good. Especially since I just replaced all the certificates on the vCenter Server. A complete guide on how to do this can be found in the following VMware KB, but I will also guide you throug this process. Sometimes it happens that when updating the certificates on a vCenter Server some solutions don’t get properly updated, which means that the sslTrust anchors are broken. These need to be restored and to do this you would need to executing the following list of actions:

1. Create a new folder called “certificates” somewhere on the vCenter Server.
2. Validate the sslTrust Anchors for the vCenter lookupservice against what is used in the solution vs the vCenter Server.
   1. Login to the vCenter Server through SSH and enter shell.set --enabled true.
   2. Run the following command to receive the current sslTrust anchor: /usr/lib/vmidentity/tools/scripts/lstool.py list --url https://localhost/lookupservice/sdk --no-check-cert --ep-type com.vmware.cis.cs.identity.sso 2>/dev/null
   3. This will give you something like the code block below. Please note I have shortened the SSLtrust string:

root@vc02 [ ~ ]# /usr/lib/vmidentity/tools/scripts/lstool.py list --url https://localhost/lookupservice/sdk --no-check-cert --ep-type com.vmware.cis.cs.identity.sso 2>/dev/null
        Service Product: com.vmware.cis
        Service Type: cs.identity
        Service ID: 6e0a387e-5eb0-440e-881f-0fb0072af7da
        Site ID: default-site
        Owner ID: vc02@vsphere.local
        Version: 2.0
        Endpoints:
                Type: com.vmware.cis.cs.identity.sso
                Protocol: wsTrust
                URL: https://vc02./sts/STSService/vsphere.local
                SSL trust: MIID8zCCAtugAwIBAgIJANF05NDek

3. Now that we have this, we can run the following command to look up the current installed vCenter Server certificate: echo | openssl s_client -connect localhost:443.
4. Compare the sslTrust strings from step 2 and step 3. If they are the same, this blogpost is not for you. If they are not the same, like on my environment, continue to step 5.
5. Now we need to retrieve the old STS certificate so that we can create a SHA1 thumbprint from it. Login to the vCenter MOB service on the following example URL: https://vc_with_embedded_psc.example.com/lookupservice/mob?moid=ServiceRegistration&method=List
   1. In the “filterCriteria” field you need to delete everything so that it displays the following tags <filterCriteria></filterCriteria> and press “Invoke”:
   2. Lookup the value array where the Name = URL, Type = anyURI and Value = https://vc_with_embedded_psc.example.com:443/sdk
   3. Note down the Name = sslTrust, Type = ArrayofString and Value = Certificate SSL trust value. We need this value in the next step. This is usually (as to what I’ve seen in the field), for the “com.vmware.cis.cs.identity.sso” registration.
   4. Copy this value inside a .crt file on the vCenter Server in the folder we previously created and append -----BEGIN CERTIFICATE----- before the string, and -----END CERTIFICATE----- after the string. Also make sure you enter a carriage return after the 64th character. I found it easier just to copy the SSLtrust string from step 3, which is already encoded correctly. Just make sure you append the BEGIN/END certificate piece. Save the file as a .crt file.
   5. Now that we have the old certificate, we can extract a SHA1 thumbprint from it, enter the following command: openssl x509 -in old_machine.crt -noout -sha1 -fingerprint
6. Now we need the new certificate that is currently connected to the vCenter Server. We can get this by executing the following command: /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT
7. Export the certificate with the following command: /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /root/certificates/new_machine_cert.crt

Next up is the last step in this process. Make sure you have the following information present:

• The SHA1 thumbprint from the old certificate (Step 1 through 5 above).
• The currently used (new) certificate on the vCenter Server (Step 6 and 7 above).
• The vCenter Server lookupservice address (Should be https://vc.example.com/lookupservice/sdk).
• The administrator@vsphere.local password.

Please note that the following part will replace the STS certificate with the new certificate throughout your entire vCenter Server and all 34 services. So please make sure you are executing this on the correct vCenter Server.

8. For this step we need to use the ls_update_certs script which is located at /usr/lib/vmidentity/tools/scripts/. Use the following command syntax to replace the STS certificate on all registered solutions.

python ls_update_certs.py --url https://vc02/lookupservice/sdk --fingerprint SHA1_FINGERPRINT_FROM_STEP_5.E --certfile NEW_MACHINE_SSL_FROM_STEP_7 --user Administrator@vsphere.local --password 'password'

The script will run for a couple of minutes, and it will end with the following piece of information:

2020-12-07 13:25:37,692 INFO  com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider - Site affinity is disabled
2020-12-07 13:25:37,733 WARN  com.vmware.vim.vmomi.client.http.impl.HttpConfigurationCompilerBase$ConnectionMonitorThreadBase - Shutting down the connection monitor.
2020-12-07 13:25:37,733 WARN  com.vmware.vim.vmomi.client.http.impl.HttpConfigurationCompilerBase$ConnectionMonitorThreadBase - Interrupted, no more connection pool cleanups will be performed.
Updated 34 service(s)

Now run step 1 through 4 again and validate if the two sslTrust Anchors are the same. They should be. Once I did this I tried to use the ‘cmsso-util’ command again and this time it worked flawlessly. Once it works you should receive the following output:

Starting License pre-check                                                      ... Done
Starting Tagging Data export                                                   ... Done
Starting Authz Data export                                                      ... Done
Conflict data, if any, can be found under /storage/domain-data/Conflict*.json
Pre-checks successful.

This was all for today. I hope this helped you guys. Once we found out that the STS certificate registration was invalid, it was a pretty straight forward approach to fix this. This will also be the last blogpost for 2020!