Today I came across a funny little issue on my lab environment. I was trying to use the “cmsso-util” tooling to play around and joining my second lab vCenter to the first vCenter Servers SSO Domain. You can read the following VMware piece of documentation if you want to know more about this command HERE. Fortunately I’ve done this multiple times before so the exact command syntax wasn’t hard to create. Once I did this I encountered an issue that resulted in the following:

cmsso-util domain-repoint command failure
cmsso-util domain-repoint command failure

After explicitly triple testing and checking the command’s and passwords that I have I was sure something else was going on. If you are ever pursuing a similar situation it will be good for you to know that you can find the log files for this check on the following location: cat /var/log/vmware/cloudvm/domain_consolidator.log. Once you open this up I noticed that there were notifications regarding a out of time sync situation. This means that the Source Embedded PSC and the Destination Embedded PSC were not on the same time syncing schedule. This checked out to be true, so I changed it and I tried it again.

Once again I received the same issues, eventhough the time was explicitly correct this time around. I was still receiving the “Incorrect User Credentials” error. Going back to the logfile I mentioned before I found the following log snippet:

If you look closely at the above log snippet you can actually already see the issue that is happening here. The vCenter I deployed got deployed on a different timezone (ahead in time). Once I changed the Time Sync settings on the second vCenter, we went “back in time”. The vCenter was deployed with a self-signed certificate that was valid starting from “Fri Nov 20 20:27:31 CET 2020”. I tried to join the vCenter to my other SSO Domain on “Fri Nov 20 16:20:52 CET 2020”. You see the issue here? The self-signed certificate isn’t valid yet, so that is why the ‘cmsso-util’ command is failing. It would be nice if the command actually displayed more of the issue itself instead of just telling me the credentials are invalid, which they are not.

The Resolution

There are two things that we can do here. We can either wait a couple more hours until the certificate is valid, or we can re-create the VCSA self-signed certificate. The first resolution is easy, just grab a couple of beers and wait :). The second on is also easy. Just do the following (Note: this procedure will introduce downtime on the vCenter server):

  1. Login to the VCSA server through a SSH connection.
  2. Go to: cd /usr/lib/vmware-vmca/bin/
  3. Run the Certificate Manager: ./certificate-manager (See the following link to find out everything about the certificate-manager tooling HERE.
  4. Choose option number 4 (Regenerate a new VMCA Root Certificate and replace all certificates) or 8 (Reset all Certificates) they perform the same.
Certificate-Manager utility in the vCenter Server
Certificate-Manager utility in the vCenter Server
  1. Answer with a “Y” on the question: Do you wish to generate all certificates using configuration file : Option[Y/N] ? :
  2. Enter the credentials, just press Enter if you want to use “administrator@vsphere.local”. Enter username [Administrator@vsphere.local]:
  3. Please configure certool.cfg with proper values before proceeding to next step.
    Press Enter key to skip optional parameters or use Default value.
  4. Enter the required information in the certool.cfg configuration, you can press enter to use default values for everything except IP Address, Hostname and VMCA Name.:
  1. Once this is done, all the services get re-signed with the fresh certificate.
  2. All services will automatically restart on the vCenter Server.
  3. You can now check the SSL certificate validity with the following command:

This looks better. Now we can retry the “cmsso-util domain-repoint” command again to see if this helped for our issue. And look here, we fixed the issues we had, the “cmsso-util” command now works again:

Working "cmsso-util" command after replacing certificates
Working “cmsso-util” command after replacing certificates

The “cmsso-util” pre-checks now work out without any issues.

The only thing left for us to do is to check (if any) the conflicts that the pre-check command provided. You can find the “conflict*.json” files on the location mentioned in the above snippet. Once this is completed, you can go ahead and change the “cmsso-util” command and use the “execute” flag instead of the “pre-check” flag. Once you execute that, you will join the vCenter Server into the SSO domain that is being used on the other vCenter Server.

This was all for today. I hope this helped. Eventhough the error message was vague in the end, always check the log files to find clues! Another blogpost is coming up on issue #2 with this procedure I did.

Leave a Reply

Your email address will not be published. Required fields are marked *