Chicken and Egg problem with “cmsso-util domain-repoint” on vCenter

Today I came across a funny little issue on my lab environment. I was trying to use the “cmsso-util” tooling to play around and joining my second lab vCenter to the first vCenter Servers SSO Domain. You can read the following VMware piece of documentation if you want to know more about this command HERE. Fortunately I’ve done this multiple times before so the exact command syntax wasn’t hard to create. Once I did this I encountered an issue that resulted in the following:

root@vc02 [ ~ ]# date
Fri Nov 20 16:20:52 CET 2020
root@vc02 [ ~ ]# cmsso-util domain-repoint -m pre-check --src-emb-admin "administrator" --replication-partner-fqdn "vc01" --replication-partner-admin administrator --dest-domain-name "vsphere.local"
Enter Source embedded vCenter Server Admin Password :
Enter Replication partner Platform Services Controller Admin Password :

Source Platform Services Controller details provided are incorrect. Check the credentials

After explicitly triple testing and checking the command’s and passwords that I have I was sure something else was going on. If you are ever pursuing a similar situation it will be good for you to know that you can find the log files for this check on the following location: cat /var/log/vmware/cloudvm/domain_consolidator.log. Once you open this up I noticed that there were notifications regarding a out of time sync situation. This means that the Source Embedded PSC and the Destination Embedded PSC were not on the same time syncing schedule. This checked out to be true, so I changed it and I tried it again.

Once again I received the same issues, eventhough the time was explicitly correct this time around. I was still receiving the “Incorrect User Credentials” error. Going back to the logfile I mentioned before I found the following log snippet:

root@vc02 [ ~ ]# cat /var/log/vmware/cloudvm/domain_consolidator.log
2020-11-20T15:22:54.152Z INFO domain_consolidator Fetched thumbprint [74:0A:CE:DD:5A:4E:66:01:F8:3A:1B:B5:E3:E6:9D] for host vc01
2020-11-20T15:22:54.152Z INFO domain_consolidator Using thumbprint to make GET request to https://vc01:443/rest/vcenter/system-config/deployment-type
2020-11-20T15:22:54.167Z INFO domain_consolidator Remote server vc01 is of type VCSA_EMBEDDED
2020-11-20T15:23:01.764Z INFO domain_consolidator Fetched thumbprint [74:0A:CE:DD:5A:4E:66:01:F8:3A:1B:B5:E3:E6:9D] for host vc01
2020-11-20T15:23:01.785Z INFO domain_consolidator Fetched thumbprint [74:0A:CE:DD:5A:4E:66:01:F8:3A:1B:B5:E3:E6:9D] for host vc01
2020-11-20T15:23:01.786Z INFO domain_consolidator validating saml token
2020-11-20T15:23:01.866Z ERROR domain_consolidator Failed to validate sso credentials. Error SoapException:
faultcode: ns0:InvalidTimeRange
faultstring: The token authority rejected an issue request for TimePeriod [startTime=Fri Nov 20 16:23:01 CET 2020, endTime=Fri Nov 20 16:33:01 CET 2020] :: Signing certificate is not valid at Fri Nov 20 16:23:01 CET 2020, cert validity: TimePeriod [startTime=Fri Nov 20 20:27:31 CET 2020, endTime=Fri Nov 15 20:37:14 CET 2030]
faultxml: <?xml version='1.0' encoding='UTF-8'?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><S:Fault xmlns:ns4="http://www.w3.org/2003/05/soap-envelope"><faultcode xmlns:ns0="http://docs.oasis-open.org/ws-sx/ws-trust/200512">ns0:InvalidTimeRange</faultcode><faultstring>The token authority rejected an issue request for TimePeriod [startTime=Fri Nov 20 16:23:01 CET 2020, endTime=Fri Nov 20 16:33:01 CET 2020] :: Signing certificate is not valid at Fri Nov 20 16:23:01 CET 2020, cert validity: TimePeriod [startTime=Fri Nov 20 20:27:31 CET 2020, endTime=Fri Nov 15 20:37:14 CET 2030]</faultstring></S:Fault></S:Body></S:Envelope>
2020-11-20T15:23:01.866Z INFO domain_consolidator
Source Platform Services Controller details provided are incorrect. Check the credentials
2020-11-20T15:23:01.867Z INFO domain_consolidator Invalid user input entered.

If you look closely at the above log snippet you can actually already see the issue that is happening here. The vCenter I deployed got deployed on a different timezone (ahead in time). Once I changed the Time Sync settings on the second vCenter, we went “back in time”. The vCenter was deployed with a self-signed certificate that was valid starting from “Fri Nov 20 20:27:31 CET 2020”. I tried to join the vCenter to my other SSO Domain on “Fri Nov 20 16:20:52 CET 2020”. You see the issue here? The self-signed certificate isn’t valid yet, so that is why the ‘cmsso-util’ command is failing. It would be nice if the command actually displayed more of the issue itself instead of just telling me the credentials are invalid, which they are not.

The Resolution

There are two things that we can do here. We can either wait a couple more hours until the certificate is valid, or we can re-create the VCSA self-signed certificate. The first resolution is easy, just grab a couple of beers and wait :). The second on is also easy. Just do the following (Note: this procedure will introduce downtime on the vCenter server):

1. Login to the VCSA server through a SSH connection.
2. Go to: cd /usr/lib/vmware-vmca/bin/
3. Run the Certificate Manager: ./certificate-manager (See the following link to find out everything about the certificate-manager tooling HERE.
4. Choose option number 4 (Regenerate a new VMCA Root Certificate and replace all certificates) or 8 (Reset all Certificates) they perform the same.

5. Answer with a “Y” on the question: Do you wish to generate all certificates using configuration file : Option[Y/N] ? :
6. Enter the credentials, just press Enter if you want to use “administrator@vsphere.local”. Enter username [Administrator@vsphere.local]:
7. Please configure certool.cfg with proper values before proceeding to next step.
Press Enter key to skip optional parameters or use Default value.
8. Enter the required information in the certool.cfg configuration, you can press enter to use default values for everything except IP Address, Hostname and VMCA Name.:

Press Enter key to skip optional parameters or use Previous value.
Enter proper value for 'Country' [Previous value : US] :
Enter proper value for 'Name' [Previous value : CA] :
Enter proper value for 'Organization' [Previous value : VMware] :
Enter proper value for 'OrgUnit' [Previous value : VMware Engineering] :
Enter proper value for 'State' [Previous value : California] :
Enter proper value for 'Locality' [Previous value : Palo Alto] :
Enter proper value for 'IPAddress' (Provide comma separated values for multiple IP addresses) [optional] : XXX.XXX.X.XX
Enter proper value for 'Email' [Previous value : email@acme.com] :
Enter proper value for 'Hostname' (Provide comma separated values for multiple Hostname entries) [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : vc02
Enter proper value for VMCA 'Name' :vc02

Continue operation : Option[Y/N] ? : Y
You are going to reset by regenerating Root Certificate and replace all certificates using VMCA
Continue operation : Option[Y/N] ? : Y

9. Once this is done, all the services get re-signed with the fresh certificate.
10. All services will automatically restart on the vCenter Server.
11. You can now check the SSL certificate validity with the following command:

root@vc02 [ ~ ]# echo | openssl s_client -servername localhost -connect localhost:443 2>/dev/null | openssl x509 -noout -dates
notBefore=Nov 20 16:50:34 2020 GMT
notAfter=Nov 20 16:50:34 2022 GMT

This looks better. Now we can retry the “cmsso-util domain-repoint” command again to see if this helped for our issue. And look here, we fixed the issues we had, the “cmsso-util” command now works again:

The “cmsso-util” pre-checks now work out without any issues.

Starting License pre-check                                                      ... Done
Starting Tagging Data export                                                   ... Done
Starting Authz Data export                                                      ... Done
Conflict data, if any, can be found under /storage/domain-data/Conflict*.json
Pre-checks successful.

The only thing left for us to do is to check (if any) the conflicts that the pre-check command provided. You can find the “conflict*.json” files on the location mentioned in the above snippet. Once this is completed, you can go ahead and change the “cmsso-util” command and use the “execute” flag instead of the “pre-check” flag. Once you execute that, you will join the vCenter Server into the SSO domain that is being used on the other vCenter Server.

This was all for today. I hope this helped. Eventhough the error message was vague in the end, always check the log files to find clues! Another blogpost is coming up on issue #2 with this procedure I did.