How to resolve a VMware vCenter Server PSC convergence failure

The other day I had to converge a couple of vCenters to an embedded Platform Services Controller (PSC) configuration, because we updated all the vCenter’s and since VMware is moving away from the External PSC configuration. In the release we are running (most current 6.7 U3 release) VMware has a nice build-in PSC Convergence tool right within the vSphere Client. I honestly never had the opportunity to use the vcsa-util cli tool together with a json file back in the day, but it seemed pretty straightforward to me, especially since it’s now all in the vSphere Client.

PSC Convergence Prerequisites

Before trying to converge your external PSC configuration to an embedded one you should (at least) meet the following prerequisites to be sure you are good to go:

• Check your NTP server configurations.
• Check all of the A and PTR DNS records.
• Make sure you have a backup for all the PSC’s and connected vCenters in the SSO domain.
• If your vCenter Server has no internet access, you should download the vCenter Server ISO beforehand. Make sure you download the exact same ISO for the vCenter Server version you are running.
• Disable vCenter HA before you start to converge the vCenter Server.

The issue

Like I said earlier, everything can now be done from the vSphere Client and you don’t need a command line utility anymore. So I started ahead and went to the Administration -> System Configuration part in the vSphere Client, selected the desired vCenter to converge and hit “Converge to Embedded”. When you hit this screen up you acknowledge that this operation will introduce downtime for the vCenter services, just enter the current vSphere SSO Domain administrator user and password, select a replication partner (if required), decide whether or not you want to join AD while you are at it, and acknowledge that you have a backup.

This was all good until I clicked “Converge” I got hit pretty instantly with the following error message:

The vCenter Server appliance build number does not match with the vCenter Server appliance configured in the update repository.

Ok great. Well at least it’s a pretty good error message which indicates what’s happening, or not happening if we’re accurate. So as a good engineer let’s start troubleshooting. The vCenter Server can update itself through an ISO, or through a connected online repository. Since I did not change anything on this environment, I checked the repository settings in the vCenter VAMI. This can easily be done by logging into the VAMI, going to Update and clicking on Settings:

Alright, this seems legit. The vCenter Server is using a 6.7 repository on the version it is running. But why is it telling me it’s not correct? Let’s check the settings on the CLI. Just login through SSH on your vCenter Server and issue the following command update.get, before you enter the Bash Shell.

Alright now we are getting somewhere. It seems that the vCenter Server is actually still using the 6.5 repository, which makes sense when looking at the error message we were receiving. Why this is happening, I’m not sure. We can also see this from the converge.log logfile on the vCenter Server. This file is located in “/var/log/vmware/converge/converge.log“.

root@vc02 [ ~ ]# tail -f /var/log/vmware/converge/converge.log
2020-10-17T08:37:21.522Z INFO converge Getting value 'password' from '/etc/applmgmt/appliance/update.conf'
2020-10-17T08:37:21.523Z INFO converge https://vapp-updates.vmware.com/vai-catalog/valm/vmw/8d167796-34d5-4899-be0a-6daade4005a3/6.5.0.15000/manifest/manifest-latest.xml
2020-10-17T08:37:21.523Z INFO converge Fetching substring-after(/update/fullVersion/text(), "Build ")
2020-10-17T08:37:21.763Z INFO converge Couldn't get https://vapp-updates.vmware.com/vai-catalog/valm/vmw/8d167796-34d5-4899-be0a-6daade4005a3/6.5.0.15000/manifest/manifest-latest.xml: HTTP Error 404: Not Found
2020-10-17T08:37:21.764Z INFO converge Couldn't open https://vapp-updates.vmware.com/vai-catalog/valm/vmw/8d167796-34d5-4899-be0a-6daade4005a3/6.5.0.15000/manifest/manifest-latest.xml
2020-10-17T08:37:21.764Z INFO converge Returning None
2020-10-17T08:37:21.764Z INFO converge The vCenter Server appliance build number does not match with the vCenter Server appliance configured in the update repository.
2020-10-17T08:37:21.767Z ERROR converge Pre-checks failed for converge.
2020-10-17T08:37:21.770Z INFO converge Unsetting the converge running flag
2020-10-17T08:37:22.428Z INFO converge Cleanup successful with partial flag = True.

But this can be fixed rather easily. There are two methods to fix this, I actually converged a couple of environments this week, and I have encountered that I had to use both. The first one is the easiest. You just login to the vCenter VAMI -> Update -> Settings -> copy the Default Repository URL -> Paste this into the “Specified” repository setting and hit Save. This triggers the vCenter to re-apply the default repository settings, which should fix the issue in most cases.

If this however does not fix the issue, and you are still getting the message, we have to login to the vCenter through SSH again. Once you did, just enter update.set --currentURL "default" this will convert the “Currenturl” setting back to the default repository. After you did this everything should be good to go like the below screenshot:

Now the converge process should start and you’re vCenter Server should be converged within a couple of minutes. You can watch the process in the previously mentioned converge.log file on the vCenter Server.

root@vc02 [ ~ ]# tail -f /var/log/vmware/converge/converge.log
2020-10-24T08:14:05.818Z INFO converge Granting permission to the user eam
2020-10-24T08:14:05.836Z INFO converge Granting permission to the user mbcs
2020-10-24T08:14:05.854Z INFO converge Granting permission to the user deploy
2020-10-24T08:14:05.872Z INFO converge Granting permission to the user vpxd
2020-10-24T08:14:05.890Z INFO converge Granting permission to the user vpxd
2020-10-24T08:14:05.909Z INFO converge Granting permission to the user vpxd
2020-10-24T08:14:05.929Z INFO converge Running Firstboots for : license_firstboot,pschealth-firstboot
2020-10-24T08:15:00.741Z INFO converge Unsetting the converge running flag
2020-10-24T08:15:01.166Z INFO converge Cleanup successful with partial flag = True.
2020-10-24T08:15:01.169Z INFO converge Starting all the services ...
2020-10-24T08:17:28.368Z INFO converge Started all the services.
2020-10-24T08:17:28.385Z INFO converge Unsetting the converge running flag
2020-10-24T08:17:28.836Z INFO converge Cleanup successful with partial flag = False.
2020-10-24T08:17:28.836Z DEBUG converge Updating the DCUI console.
2020-10-24T08:17:28.841Z INFO converge Converged the external PSC to embedded PSC Successfully.
2020-10-24T08:17:28.841Z INFO converge Join AD domain if required.
2020-10-24T08:17:28.841Z INFO converge Skipping AD Domain joining.
2020-10-24T08:17:28.854Z INFO converge Collected end_time:16048.8 as a part of Telemetry
2020-10-24T08:17:28.854Z INFO converge Collected status:True as a part of Telemetry
2020-10-24T08:17:28.854Z INFO converge Collected @type:convergence_telemetry as a part of Telemetry
2020-10-24T08:17:28.857Z DEBUG converge Starting new HTTP connection (1): localhost
2020-10-24T08:17:28.979Z DEBUG converge http://localhost:15080 "POST /analytics/telemetry/ph/api/hyper/send?_c=vcebc.1_0 HTTP/1.1" 201 0
2020-10-24T08:17:28.982Z DEBUG converge Pushed the Collected Data to VMWare Analytics Service as a part of  Telemetry

Once you see the last line, you know the process is finished. It might still be (re)starting some services, but within a couple of minutes you should be able to login to the vCenter and see that your vCenter is now using an embedded Platform Services Controller.

I did try to find out why the vCenter Server is still using an older repository setting, but unfortunately I couldn’t find anything related in any KB. But there you have it guys, another simple fix, once you know it!

See you guys in the next post.