Unverifiable vCenter STS certificate while adding NSX-V Manager

Introduction

Today we have yet another certificate blogpost. This is the fourth blogpost on this in the last couple of months. I’ve recently created a new environment and added custom CA signed vCenter Server certificates through the vSphere Client (UI). In a recent blogpost I had some issues with this since I couldn’t find my private key that I needed with the UI generated .CSR to replace the certificate. Now, it seems there is another issue when using the vSphere Client to replace your MACHINE_CERT certificate on the vCenter Server.

It seems that the vCenter VMware Security Token Service (STS) Certificates don’t get replaced when we use this feature in the vSphere Client. I came across this when I tried to add a fresh NSX Manager to the vCenter Server. I was surprised to receive the below error message:

NSX Management Service operation failed. ( Initialization of Admin Registration Service Provider failed. Root Cause: The SSL certificate of STS service cannot be verified ).

Fortunately I’ve done this procedure about 15 times now since I’ve uploaded one of my previous blogposts. I do find it troublesome that this is occuring so many times in the last couple of months. I am not sure if this is because or our environment, or because there is a bug in the vCenter Server. We are running the latest vCenter Server 6.7 release for this environment.

The reason for this message is that the internal vCenter Server SslTrusts are not correctly configured for the certificates that are being used at this point. More precisely, the STS Service certificate isn’t properly using the new SSL certificate. We will have to update the sslTrust Anchors to get this working again.

Resolution

You can have a look at my previous blog post for the entire procedure, explained step by step. Or just follow the below list of actions:

1. Create a new folder called “certificates” somewhere on the vCenter Server.
2. Validate the sslTrust Anchors for the vCenter lookupservice against what is used in the solution vs the vCenter Server.
   1. Login to the vCenter Server through SSH and enter shell.set --enabled true.
   2. Run the following command to receive the current sslTrust anchor: /usr/lib/vmidentity/tools/scripts/lstool.py list --url https://localhost/lookupservice/sdk --no-check-cert --ep-type com.vmware.cis.cs.identity.sso 2>/dev/null
3. Now that we have this, we can run the following command to look up the current installed vCenter Server certificate: echo | openssl s_client -connect localhost:443.
4. Compare the sslTrust strings from step 2 and step 3. If they are the same, this blogpost is not for you. If they are not the same, like on my environment, continue to step 5.
5. Copy the SSLTrust string from step 3 and copy it inside a .crt file on the vCenter Server in the folder we previously created and append -----BEGIN CERTIFICATE----- before the string, and -----END CERTIFICATE----- after the string. (Make sure the string is BASE64 encoded, just add a space after each 64th character).
6. Now that we have the old certificate, we can extract a SHA1 thumbprint from it, enter the following command: openssl x509 -in old_machine.crt -noout -sha1 -fingerprint
7. Export the current certificate with the following command: /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /root/certificates/new_machine_cert.crt
8. For this step we need to use the ls_update_certs script which is located at /usr/lib/vmidentity/tools/scripts/. Use the following command syntax to replace the STS certificate on all registered solutions.

python ls_update_certs.py --url https://vc02/lookupservice/sdk --fingerprint SHA1_FINGERPRINT_FROM_STEP_5.E --certfile NEW_MACHINE_SSL_FROM_STEP_7 --user Administrator@vsphere.local --password 'password'

Once this is done, the sslTrust Anchors for the STS Service have been replaced and you should be able to add the NSX Manager to the vCenter Server.