Where is my private key when using the vSphere UI?

Today a very quick post for anybody that is using the vSphere Client to generate a new .CSR to create a new certificate for the vCenter Server. I usually use the certificate-manager tool on the commandline, but I figured, let’s diversify my daily actions and let us use the vSphere Client. This has been around for a while now. The below blog post was made while using the 6.7 U3 vSphere Client.

You can easily do this by logging into the vSphere Client -> Administration -> Certificates -> Certificate Management. Press on Actions -> Generate CSR on the __MACHINE_CERT, like below (procedure can also be found here):

Next you can enter the certificate fields like you usually do on the command line:

And once this is done you get a window that displays the .CSR you just created. You can copy this .CSR and use your favorite CA to create the new certificate for the vCenter Server. Once you received the certificate file with the Chain, go back to the previous window in the vSphere Client, and instead of pressing Actions -> Generate CSR, press Replace. In this window upload the certificate file, and the Private Key file.

Wait what? I never received the private key file. The procedure also does not mention anything about the private key file. Ok so where is this private key file located? I did a quick recursive search on the command line with locate*.key but didn’t really find anything that looked like it was my private key:

root@vcsa02 [ ~ ]# locate *.key
/etc/vmware-rbd/ssl/rbd-ca.key
/etc/vmware-rbd/ssl/waiter.key
/etc/vmware-rhttpproxy/ssl/rui.key
/etc/vmware-vpx/ssl/data-encipherment.key
/etc/vmware-vpx/ssl/rui.key
/etc/vmware-vpx/ssl/vcsoluser.key
/etc/vmware/vmware-vmafd/machine-ssl.key
/opt/vmware/var/lib/vami/update/data/signature.key
/storage/db/vpostgres_ssl/server.key
/usr/lib/vmware-updatemgr/bin/ssl/jetty.key
/usr/lib/vmware-updatemgr/bin/ssl/rui.key
/var/lib/vmware/vmcam/ssl/rui.key

After carefully looking around and checking the below log file, I noticed the second to last line in the /var/log/vmware/certificatemanagement/certificatemanagement-svcs.log log file.

2020-12-11T12:06:08.274Z [tomcat-exec-9  INFO  com.vmware.certificatemanagement.vapi.impl.setup.AuthzPermissionValidator  opId=] User VSPHERE.LOCAL\Administrator has required privileges [CertificateManagement.Manage] to invoke API com.vmware.vcenter.certificate_management.vcenter.tls_csr.create
2020-12-11T12:06:08.376Z [tomcat-exec-9  INFO  com.vmware.certificatemanagement.impl.SSLCertificate  opId=] Generating a new certifcate signing request for user provided input specifications
2020-12-11T12:06:08.377Z [tomcat-exec-9  INFO  com.vmware.certificatemanagement.impl.SSLCertificate  opId=] Input Spec -
2020-12-11T12:06:08.377Z [tomcat-exec-9  INFO  com.vmware.certificatemanagement.impl.SSLCertificate  opId=] key size: 4096
2020-12-11T12:06:08.377Z [tomcat-exec-9  INFO  com.vmware.certificatemanagement.impl.SSLCertificate  opId=] Common Name: vc02.bvecloud.nl
2020-12-11T12:06:08.377Z [tomcat-exec-9  INFO  com.vmware.certificatemanagement.impl.SSLCertificate  opId=] Organization: BveCloud
2020-12-11T12:06:08.378Z [tomcat-exec-9  INFO  com.vmware.certificatemanagement.impl.SSLCertificate  opId=] Organization Unit Name: Operations
2020-12-11T12:06:08.378Z [tomcat-exec-9  INFO  com.vmware.certificatemanagement.impl.SSLCertificate  opId=] Email: bvecloud@bvecloud.nl
2020-12-11T12:06:08.378Z [tomcat-exec-9  INFO  com.vmware.certificatemanagement.impl.SSLCertificate  opId=] Locality: Amsterdam
2020-12-11T12:06:08.378Z [tomcat-exec-9  INFO  com.vmware.certificatemanagement.impl.SSLCertificate  opId=] State: Noord-Holland
2020-12-11T12:06:08.378Z [tomcat-exec-9  INFO  com.vmware.certificatemanagement.impl.SSLCertificate  opId=] Country: NL
2020-12-11T12:06:08.378Z [tomcat-exec-9  INFO  com.vmware.certificatemanagement.impl.SSLCertificate  opId=] Hostnames: 10.0.0.1,vc02.bvecloud.nl
2020-12-11T12:06:08.378Z [tomcat-exec-9  INFO  com.vmware.certificatemanagement.impl.SSLCertificate  opId=] IP Address: 10.0.0.1
2020-12-11T12:06:08.378Z [tomcat-exec-9  INFO  com.vmware.certificatemanagement.impl.SSLCertificate  opId=] User provided key size: 4096
2020-12-11T12:06:08.822Z [tomcat-exec-9  INFO  com.vmware.certificatemanagement.impl.SSLCertificate  opId=] Successfully generated private key
2020-12-11T12:06:08.836Z [tomcat-exec-9  INFO  com.vmware.certificatemanagement.impl.SSLCertificate  opId=] Successfully generated certificate signing request
2020-12-11T12:06:08.986Z [tomcat-exec-9  INFO  com.vmware.certificatemanagement.impl.SSLCertificate  opId=] Added new entry __MACHINE_CSR to VECS store MACHINE_SSL_CERT
2020-12-11T12:06:08.987Z [tomcat-exec-9  INFO  com.vmware.certificatemanagement.impl.SSLCertificate  opId=] Successfully stored the private key to VECS store MACHINE_SSL_CERT, alias __MACHINE_CSR
2020-12-11T12:06:08.987Z [tomcat-exec-9  INFO  com.vmware.certificatemanagement.impl.SSLCertificate  opId=] CSR generation API executed successfully

It looks like the .CSR got created successfully and the private key is stored inside the VECS certificate store under the __MACHINE_CSR alias. Aah ok this makes sense. I just wish that VMware said this on their documentation, or just provided the user with a notification in the vSphere Client when a .CSR is generated. You can dig up the private key with the following command:

root@vc02 [ /tmp ]# /usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store MACHINE_SSL_CERT --alias __MACHINE_CSR 

Copy this private key to a notepad file and save it as private.key. Now this file can be used while importing the validated certificate file into the vSphere Client.

Now all we need to do is restart the following services on the command line, please be aware that this will introduce some downtime on the vCenter Server:

service-control --stop --all
service-control --start --all

Once this is done the certificate will be replaced. Fortunately this has been updated in vSphere 7.0. It will now automatically use the private key that is stored in the VECS store.

This was all for today. It took me a bit of time to find this out, so I hope I can prevent you from looking around to long!