This post is about how to update/renew custom certificates on External PSC, vCenter appliance and ESXi hosts.
So, i am writing this post so that you can easily understand how to update the custom certificates in a distributed environment.
Let me give you a brief overview about my distributed environment which is as follows.
2 X PSC appliances 6.5 U3 version (External PSC behind F5 load balancer)
1 X vCenter appliance 6.5 U3
3 X ESXi hosts 6.5 U3
Alright, Lets start the certificate update/renew in below order only.
1. Update/Renew certificates on External PSC appliances.
2. Update/Renew certificates on vCenter appliance.
3. Update/Renew certificates on ESXI hosts.
First, please take snapshots on all PSC and VCSA appliances.
Here, we are going to update/renew the Machine SSL certificates for PSC, VCSA and ESXi host.
=====================================================================
External PSC Certificate Update/Renew
For PSC certificates update/renew, please follow below VMware KB.
Above VMware KB article explains, how to generate .cfg file then .csr and .key files.
Once the .csr file generated then send it to your Internal Microsoft CA to generate the certificates and also ask them to share the Root CA certificate and Intermediate CAs certificates as well.
In many organisation, there might be 2 Intermediate CAs then take the certificates of both Intermediate CAs as well.
Once you have server, Root CA and Intermediate CAs certificates ready then Copy these certificates files on first PSC appliances under /certs directory and follow the above KB article section starting from this subject (Generating a certificate from an external certificate authority)
You need to update/renew certificates on all PSCs appliances one by one and reboot them as well. Please make sure all services are UP on both the PSC appliances with this command (service-control --status --all) after reboot.
Post PSCs certificate update, please STOP (service-control --stop --all) and START (service-control --start --all) all services on VCSA appliance or you can reboot the VCSA appliance as well if you want.
Please make sure all VCSA services are UP post reboot and you are able to connect to vCenter via GUI as well.
that's all for PSCs certificate update/renew part.
================================================================
VCSA Certificate Update/Renew
Let's start the certificate update on VCSA appliance. We will update the Machine SSL certificate on VCSA.
I hope you took snapshot on VCSA appliance already if not please do it now.
Please follow the below KB article for VCSA appliance certificate update/renew.
Above KB article will tell you how to generate .cfg file and then create a .csr and .key file. Once you have .csr file ready, sent it to Internal Microsoft CA to generate the server certificate and also get the relevant Root CA and Intermediate CAs certificate as we did above during PSC certificate task.
One thing, i want to highlight here that .cer and .crt extension of certificate are same, so don't confuse about these.
Once you have server certificate, ROOT CA and Intermediate CAs certificate ready then you have to create below certificates file now.
1. Machine_SSL.cer : This is a complete chain of server certificate + intermediate CAs(if applicable) + Root CA
2. Root64.cer: This is a chain of intermediate CAs(if applicable) + Root CA
3. SSL.key : this is .key file which we created earlier during .csr file creation.
Now, follow the below steps to update/renew the certificate on VCSA server.
Login on VCSA via SSH and copy the above certificates files under directory /certs via WinSCP or SCP.
- Launch the vSphere 6.x Certificate Manager:
/usr/lib/vmware-vmca/bin/certificate-manager
2. Select Option 1 (Replace Machine SSL certificate with Custom Certificate).
3. Provide the administrator@vsphere.local password when prompted.
4. Select Option 1 (Continue to importing Custom certificate(s) and key(s) for Machine SSL certificate).
5. It will ask the PSC IP address now, please provide the PSC load balancer IP or PSC appliance IP whatever is applicable here. In my case, my PSCs are behind the F5 load balancer so i gave Load Balancer IP.
6. It will ask for below files location one by one like below.
Please provide valid custom certificate for Machine SSL.
File : /certs/machine_ssl.cer
Please provide valid custom key for Machine SSL.
File : /certs/ssl.key
Please provide the signing certificate of the Machine SSL certificate.
File : /certs/Root64.cer
Now, certificate update task will start and it will take some time to complete. If any issues then this certificate task will rollback the VCSA on previous certificates. If any issues occurs the you can check the below logs for more information.
/var/log/vmware/certificate-manager.log
Once the certificate task completed successfully then reboot your VCSA appliance once and make sure that all services should be UP.
That's for all for VCSA certificate update/renew part.
=================================================================
ESXI Certificate Update/Renew
Let's start the ESXi custom certificate update/renew process.
ESXi don't requires custom certificate as by default they always get the VMCA certificate when added to the vCenter server.
But in some cases specially any organisation security policy requires custom internal organisation certificates on ESXi hosts as well. So, in this case we need to apply the custom certificates on all ESXi as well.
It is not possible that we are using VMCA certificates on VCSA and but want to apply custom certificates on ESXi hosts. VCSA also needs to have custom certificates as well before update the custom certificate on ESXi hosts otherwise you cannot add the ESXi hosts to vCenter server because of different certificate providers.
Before apply the ESXi certificates, we need to change one advance setting (Certificate Management mode for ESXi hosts) on vCenter Server which is VMCA by default but we need to change it to "custom".
Below is the steps how to change this setting on vCenter Server.
Procedure:
1. Select the vCenter Server that manages the hosts and click Settings.
2. Click Advanced Settings, and click Edit.
3. In the Filter box, enter certmgmt to display only certificate management keys.
4. Change the value of vpxd.certmgmt.mode to "custom" if you intend to manage your own certificates.
5. Restart the vCenter Server service.
Now, lets generate the ESXi certificates from Internal Microsoft CA as per below process.
Procedure:
1. Create a .cfg file and then .csr and .key files. (Process is same as we did in PSC and VCSA case)
2. Send them to Internal Microsoft CA to generate the certificates.
3. Put the ESXi host into maintenance mode.
4. Log in to the ESXi Shell, either directly from the DCUI or from an SSH client, as a user with administrator privileges.
5. In the directory on ESXi host /etc/vmware/ssl, rename the existing certificates using the following commands.
mv rui.crt orig.rui.crt
mv rui.key orig.rui.key
6. Copy the certificates that you want to use to /etc/vmware/ssl.
7. Rename the new certificate and key to rui.crt and rui.key.
8. Restart the host after you install the new certificate.
Alternatively, you can put the host into maintenance mode, install the new certificate, use the Direct Console User Interface (DCUI) to restart the management agents, and set the host to exit maintenance mode.
9. Check the certificate status on ESXI, click on ESXi on vCenter server, go to Configure Tab, select Certificate option under Systems, you will see the new updated custom certificate there.
I saw one weird thing as well, if you still see the VMCA certificate on ESXi host then you need to disconnect and reconnect the ESXi host.
Also, when you will copy the certificate and key files on ESXi directory under /etc/vmware/ssl then edit these files with vi editor and remove the ^M character from all lines as shown in below screenshot.
That's all for this certificate update/renew post.
I know this post seems to be very long but i hope it will be informative to you as well.
Cheers.............
