How to fix VCF Usage Meter v9 connectivity issues after upgrade

Introduction

Starting last month Broadcom announced the general availability for a new version of VMware Usage Meter. This time we were required to upgrade from v4.8 to v9 (inline with the upcoming VCF 9 release).

This upgrade is mandatory for all Pinnacle Partners and their respective White Label Partners within 90 days. We upgraded a couple of our own Usage Meters and found several issues for which this blog will be the first issue that you might encounter after the upgrade itself.

Troubleshooting and Solution

After the upgrade itself which fine on this appliance, we encountered an error that resulted in the Usage Meter not being able to connect to the Broadcom online environment. So that means it could not upload Usage data. The following error was found in the UI:

VCF Business Services console Test FAiled due to "Http failure response for https://um.fqdn/api/v1/run_uploader: 500 OK". Please open a ticket with Broadcom support at https://support.broadcom.com.

In the logs we found the following (root@um [ /var/log/vmware-um-platform ]# vi dss_error.log):

2025-06-12 06:33:03.109 ERROR --- [pool-2-thread-1] c.v.um.common.platform.JournalClient     : Hostname um not verified:
    certificate: sha256/54VckSkreIEG1yA9ANKp/TuK+TF48Ax4/3pZq8iK6o4=
    DN: CN=mgmt-um,O=um,ST=Some-State,C=US
    subjectAltNames: []. Check the appliance logs for more details.
com.vmware.um.common.err.UmException: Hostname um not verified:
    certificate: sha256/54VckSkreIEG1yA9ANKp/TuK+TF48Ax4/3pZq8iK6o4=
    DN: CN=um,O=um,ST=Some-State,C=US
    subjectAltNames: []. Check the appliance logs for more details.
        at com.vmware.um.common.platform.UmPlatformClient.accessToken(UmPlatformClient.java:199)
        at com.vmware.um.common.platform.UmPlatformClient.callPlatformService(UmPlatformClient.java:215)
        at com.vmware.um.common.platform.JournalClient.createJournalEntry(JournalClient.java:151)
        at com.vmware.um.common.platform.JournalClient.lambda$createJournalEntryAsync$1(JournalClient.java:176)
        at com.vmware.um.common.platform.JournalClient$JournalWorkerThread.run(JournalClient.java:89)
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: javax.net.ssl.SSLPeerUnverifiedException: Hostname um not verified:
    certificate: sha256/54VckSkreIEG1yA9ANKp/TuK+TF48Ax4/3pZq8iK6o4=
    DN: CN=um,O=um,ST=Some-State,C=US
    subjectAltNames: []
        at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.kt:389)
        at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.kt:337)
        at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:209)
        at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226)
        at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106)
        at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74)
        at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255)
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at com.vmware.um.common.http.LoggingInterceptor.intercept(LoggingInterceptor.java:25)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at com.vmware.um.common.http.RetryInterceptor.intercept(RetryInterceptor.java:72)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
        at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
        at com.vmware.um.common.platform.UmPlatformClient.oauth2Flow(UmPlatformClient.java:174)
        at com.vmware.um.common.platform.UmPlatformClient.login(UmPlatformClient.java:153)
        at com.vmware.um.common.platform.UmPlatformClient.accessToken(UmPlatformClient.java:197)
        ... 8 common frames omitted

Turns out there is a KB for that. It seems the old Usage Meter v4.8 had less strict requirements for the certificates placed on the Usage Meters. So we had to replace them. The following actions can be followed if you wish to replace the certificates with a new Default Self Signed certificate. If you wish to follow other implementations i.e. with a CA Signed certificate you can follow the others steps mentioned here in this KB.

1. stop.sh All as the ‘usagemeter’ user.
2. Export the environment variables.
   export $(grep -v '^#' "/opt/vmware/cloudusagemetering/platform/conf/env.properties" | xargs)
3. Back up the existing VCF usage meter appliance certificate.
   • Backup the existing keystore
     • mv /opt/vmware/cloudusagemetering/platform/security/keystore /opt/vmware/cloudusagemetering/platform/security/keystore.backup
   • Move the existing keystore entry from the specifed alias to a new alias that is under the destalias parameter.
     • keytool -changealias -alias "usage-meter-platform" -destalias "usage-meter-platform-backup" -keystore /opt/vmware/cloudusagemetering/platform/security/cacerts -storepass "${TRUST_STORE_PASSWORD}"

The second command we entered gave us an error which we could not place:

keytool error: java.security.KeyStoreException: Unrecognized keystore format. Please load it with a specified type

This was due to us having FIPS Security mode enabled. Strange enough, we never enabled this and it looks like it was auto enabled after the VCF 9 upgrade (not sure though). Because we already stopped the services and the keystore was already moved the UI was not working so we were not able to double check this. However, there is a CLI command to check if your UM UI is using FIPS:

fips-util.sh -s | jq

usagemeter@um [ /opt/vmware/cloudusagemetering/platform/security ]$ fips-util.sh -s | jq
{
  "platform": {
    "fips_enabled": true,
    "key_store_type": "BCFKS",
    "trust_store_type": "BCFKS",
    "sign_key_store_type": "BCFKS"
  },
  "os": {
    "kernel": true,
    "openssl": "fips=yes",
    "openssh": {
      "ciphers": "aes256-cbc,aes192-cbc,aes128-cbc,3des-cbc,aes128-ctr,aes192-ctr,aes256-ctr,aes256-gcm@openssh.com,aes128-gcm@openssh.com",
      "macs": "hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-sha1",
      "kex": "diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521"
    }
  },
  "nginx": {
    "nginx_ciphers": "TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:!aNULL;",
    "tls_protocols": "TLSv1.2 TLSv1.3;"
  },
  "files": {
    "key_store_type": "BCFKS",
    "trust_store_type": "Failed to find key store /opt/vmware/cloudusagemetering/platform/security/keystore.",
    "sign_key_store_type": "BCFKS"
  }
}

3. So if you have FIPS enabled, use this as a second command:
   • keytool -changealias -alias "usage-meter-platform" -destalias "usage-meter-platform-backup" -keystore /opt/vmware/cloudusagemetering/platform/security/cacerts -storetype BCFKS -providerclass org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath /opt/vmware/cloudusagemetering/platform/lib/bc-fips-*.jar -storepass "${TRUST_STORE_PASSWORD}"
4. Create a temporary directory and set the directory path to the NGINX_FOLDER environment variable.
5. Create a keystore for the new self-signed certificate. Make sure that you are in the /opt/vmware/cloudusagemetering directory.
   • ./platform/bin/create-keystore.sh
6. Remove all temporary and backup folders, and delete the old VCF usage meter certificate.
   • rm -rf $NGINX_FOLDER
   • rm /opt/vmware/cloudusagemetering/platform/security/keystore.backup
   • keytool -delete -alias "usage-meter-platform-backup" -keystore /opt/vmware/cloudusagemetering/platform/security/cacerts -storepass "${TRUST_STORE_PASSWORD}"
   • Or with FIPS mode enabled: keytool -delete -alias "usage-meter-platform-backup" -keystore /opt/vmware/cloudusagemetering/platform/security/cacerts -storetype BCFKS -providerclass org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath /opt/vmware/cloudusagemetering/platform/lib/bc-fips-*.jar -storepass "${TRUST_STORE_PASSWORD}"
7. Configure the permissions for the keystore.
   • chmod 0640 /opt/vmware/cloudusagemetering/platform/security/keystore
8. Reboot the appliance sudo reboot.

Once that is done, you should have a working environment again! To to Settings -> Cloud Connectivity and try the Test Connection again.