Custom Certificate update/renew on External PSC, VCSA appliances and ESXi hosts on 6.5 U3 version.

Hello Everyone,

This post is about how to update/renew custom certificates on External PSC, vCenter appliance and ESXi hosts.

So, i am writing this post so that you can easily understand how to update the custom certificates in a distributed environment.

Let me give you a brief overview about my distributed environment which is as follows.

2 X PSC appliances 6.5 U3 version (External PSC behind F5 load balancer)

1 X vCenter appliance 6.5 U3 

3 X ESXi hosts 6.5 U3

Alright, Lets start the certificate update/renew in below order only.

1. Update/Renew certificates on External PSC appliances.

2. Update/Renew certificates on vCenter appliance.

3. Update/Renew certificates on ESXI hosts.

First, please take snapshots on all PSC and VCSA appliances.

Here, we are going to update/renew the Machine SSL certificates for PSC, VCSA and ESXi host. 

=====================================================================

                                     External PSC Certificate Update/Renew

For PSC certificates update/renew, please follow below VMware KB.

https://kb.vmware.com/s/article/2147627

Above VMware KB article explains, how to generate .cfg file then .csr and .key files.

Once the .csr file generated then send it to your Internal Microsoft CA to generate the certificates and also ask them to share the Root CA certificate and Intermediate CAs certificates as well.

In many organisation, there might be 2 Intermediate CAs then take the certificates of both Intermediate CAs as well.

Once you have server, Root CA and Intermediate CAs certificates ready then Copy these certificates files on first PSC appliances under /certs directory and follow the above KB article section starting from this subject (Generating a certificate from an external certificate authority)

You need to update/renew certificates on all PSCs appliances one by one and reboot them as well. Please make sure all services are UP on both the PSC appliances with this command (service-control --status --all) after reboot.

Post PSCs certificate update, please STOP (service-control --stop --all) and START (service-control --start --all) all services on VCSA appliance or you can reboot the VCSA appliance as well if you want. 

Please make sure all VCSA services are UP post reboot and you are able to connect to vCenter via GUI as well.

that's all for PSCs certificate update/renew part.

================================================================

                                        VCSA Certificate Update/Renew

Let's start the certificate update on VCSA appliance. We will update the Machine SSL certificate on VCSA.

I hope you took snapshot on VCSA appliance already if not please do it now.

Please follow the below KB article for VCSA appliance certificate update/renew.

https://kb.vmware.com/s/article/2112277

Above KB article will tell you how to generate .cfg file and then create a .csr and .key file. Once you have .csr file ready, sent it to Internal Microsoft CA to generate the server certificate and also get the relevant Root CA and Intermediate CAs certificate as we did above during PSC certificate task.

One thing, i want to highlight here that .cer and .crt extension of certificate are same, so don't confuse about these.

Once you have server certificate, ROOT CA and Intermediate CAs certificate ready then you have to create below certificates file now.

1. Machine_SSL.cer : This is a complete chain of server certificate + intermediate CAs(if applicable) + Root CA

2. Root64.cer: This is a chain of intermediate CAs(if applicable) + Root CA

3. SSL.key : this is .key file which we created earlier during .csr file creation.

Now, follow the below steps to update/renew the certificate on VCSA server. 

Login on VCSA via SSH and copy the above certificates files under directory /certs via WinSCP or SCP.

1. Launch the vSphere 6.x Certificate Manager:
   
   /usr/lib/vmware-vmca/bin/certificate-manager

     2. Select Option 1 (Replace Machine SSL certificate with Custom Certificate).

     3. Provide the administrator@vsphere.local password when prompted.

      4. Select Option 1 (Continue to importing Custom certificate(s) and key(s) for Machine SSL certificate).

    5. It will ask the PSC IP address now, please provide the PSC load balancer IP or PSC                appliance IP whatever is applicable here. In my case, my PSCs are behind the F5 load            balancer so i gave Load Balancer IP.

    6. It will ask for below files location one by one like below.

Please provide valid custom certificate for Machine SSL.

File : /certs/machine_ssl.cer
 

Please provide valid custom key for Machine SSL.

File : /certs/ssl.key

 

Please provide the signing certificate of the Machine SSL certificate.

File : /certs/Root64.cer

Now, certificate update task will start and it will take some time to complete. If any issues then this certificate task will rollback the VCSA on previous certificates. If any issues occurs the you can check the below logs for more information.

/var/log/vmware/certificate-manager.log

Once the certificate task completed successfully then reboot your VCSA appliance once and make sure that all services should be UP.

That's for all for VCSA certificate update/renew part.

=================================================================

                                   ESXI Certificate Update/Renew

Let's start the ESXi custom certificate update/renew process.

ESXi don't requires custom certificate as by default they always get the VMCA certificate when added to the vCenter server. 

But in some cases specially any organisation security policy requires custom internal organisation certificates on ESXi hosts as well. So, in this case we need to apply the custom certificates on all ESXi as well.

It is not possible that we are using VMCA certificates on VCSA and but want to apply custom certificates on ESXi hosts. VCSA also needs to have custom certificates as well before update the custom certificate on ESXi hosts otherwise you cannot add the ESXi hosts to vCenter server because of different certificate providers.

Before apply the ESXi certificates, we need to change one advance setting (Certificate Management mode for ESXi hosts) on vCenter Server which is VMCA by default but we need to change it to "custom".

Below is the steps how to change this setting on vCenter Server.

Procedure:

1. Select the vCenter Server that manages the hosts and click Settings.

2. Click Advanced Settings, and click Edit.

3. In the Filter box, enter certmgmt to display only certificate management keys.

4. Change the value of vpxd.certmgmt.mode to "custom" if you intend to                    manage your own certificates.

5. Restart the vCenter Server service.

Now, lets generate the ESXi certificates from Internal Microsoft CA as per below process.

Procedure:

1. Create a .cfg file and then .csr and .key files. (Process is same as we did in PSC and VCSA case)

2. Send them to Internal Microsoft CA to generate the certificates.

3. Put the ESXi host into maintenance mode.

4. Log in to the ESXi Shell, either directly from the DCUI or from an SSH client, as a user with administrator privileges.

5. In the directory on ESXi host /etc/vmware/ssl, rename the existing certificates using the following commands.

        mv rui.crt orig.rui.crt

        mv rui.key orig.rui.key

6. Copy the certificates that you want to use to /etc/vmware/ssl.

7. Rename the new certificate and key to rui.crt and rui.key.

8. Restart the host after you install the new certificate.

Alternatively, you can put the host into maintenance mode, install the new certificate, use the Direct Console User Interface (DCUI) to restart the management agents, and set the host to exit maintenance mode.

        9. Check the certificate status on ESXI, click on ESXi on vCenter server, go to                      Configure Tab, select Certificate option under Systems, you will see the new                    updated custom certificate there. 

I saw one weird thing as well, if you still see the VMCA certificate on ESXi host then you need to disconnect and reconnect the ESXi host.

Also, when you will copy the certificate and key files on ESXi directory under /etc/vmware/ssl then edit these files with vi editor and remove the ^M character from all lines as shown in below screenshot.

That's all for this certificate update/renew post. 

I know this post seems to be very long but i hope it will be informative to you as well.

Cheers.............

Imported Virtual machine in vRA not showing IP address allocation in network profile

 

Imported Virtual machine in vRA not showing IP address allocation in network profile

Hello Everyone,

Today, I am writing this post for the issue which I encountered after bulk import of Virtual machine into vRealize Automation 7.6 version.

I saw the behaviour that imported virtual machine IP address is not showing allocated in Network profile section.

Due to this IP address allocation not updated in network profile, vRA allocates that same IP to another VM during the VM deployment via vRA. This is default behaviour of vRA because as per the vRA inventory that particular IP is not allocated to any VM.

But when I raised a VM provision request via vRA, it got stucked on customise the virtual machine because it detects that the IP is already in use on the network so my VM request got failed after 2 hrs. time out period.

Now, below are the steps to update the IP address for the registered VMs into vRA via IAAS SQL database so that IP address allocated in network profile and vRA come know about that IP address as well.

 

1.      Login to your IAAS SQL server.

2.      Open the SQL server management studio with SA account or any other account which should have full access on vRA database.

3.      Run below search query to find out the details about that IP address.

                  select * from StaticIPv4Address where IPv4Address ='172.16.10.90'

4.      You will get below details about this IP address.

 

 

5.      So in above pictures, you can see Virtual Machine ID is NULL , and StaticIPv4AddressState value is 1, this means IP is not allocated to any VM by vRA.

6.      Now, we need to update Virtual Machine ID, IP address and StaticIPv4AddressState to “0” via IAAS SQL database update command as shown below.

 

UPDATE StaticIPv4Address SET VirtualMachineID  = '962982F5-06DD-4EF8-B03B-F173742D1028', StaticIPv4AddressState = '0' WHERE IPv4Address = '172.16.10.90'

 

7.      Now, VM ID and IPv4 address details updated as shown below

 

 

8.      So, if you check in vRA under Network profile section, you will see that above IP has been allocated to the respective VM.

 

That’s all for this post, hope this will informative to everyone, Cheers…..!!!!

 

vRealize Automation 7.6 cluster replica nodes AD joining issue

Hi There,

This article is for how to resolve the vRA replica nodes domain joining issues.

Let me tell you the background of this issue, I have upgraded my vRA environment 7.3 to 7.6 version successfully. After the upgrade, I introduced 2 replica nodes into vRA cluster to make it distributed deployment.

Once, i have added the replica nodes into vRA cluster successfully. I logged into my vRA portal default tenant and try to add these 2 nodes( which are basically called as connectors into vRA portal) to my Active Directory domain so that i can leverage the benefit of Load balancer and user authentication can be made via any nodes into the vRA cluster.

On vRA portal, go to Administration Tab, under Directories Management, click on Connectors tab

When i click on Join domain tab on my second Replica node, i got below error.

Connector communication failed with response: Certificate for <VRA002.cloudarena.com> doesn't match common name of the certificate subject: VRA001.cloudarena.com for the connector VRA002.cloudarena.com

This error is same for third Replica node as well.

Connector communication failed with response: Certificate for <VRA003.cloudarena.com> doesn't match common name of the certificate subject: VRA001.cloudarena.com for the connector VRA003.cloudarena.com


I checked my vRA certificate status and all my certificates are valid until Sep 2020. 

I rebooted the whole stack of my vRA environment but still issue is same.

Then, we have a look on certificate part on replica nodes and figure out that 

• As per the error, the newly deployed vRA replica appliances were using the internal horizon certificate for the original appliance VRA001.cloudarena.com
• This certificate is unrelated to the one we would have generated and applied to vRA.
• We tried to run a command to update the certificates locally on the 2 new replicas
• Run this command >>
• /usr/local/horizon/scripts/installExternalCertificate.hzn --ca /usr/local/horizon/conf/root_ca.pem --cert /usr/local/horizon/conf/VRA002.cloudarena.com_cert.pem --key /usr/local/horizon/conf/VRA002.cloudarena.com_key.pem --alias "horizoninternalcert"
  /usr/local/horizon/conf/root_ca.pem --cert /usr/local/horizon/conf/VRA002.cloudarena.com_cert.pem --key /usr/local/horizon/conf/VRA002.cloudarena.com_key.pem --alias "horizoninternalcert" 
•  
• This failed with the following error;
• keytool error: java.io.IOException: DER length more than 4 bytes: 109
• Run steps to workaround this on both vRA replicas nodes;
• mkdir /root/tmp-bkp
• mv /usr/local/horizon/conf/flags/fips* /root/tmp-bkp
• /usr/local/horizon/scripts/secure/wizardssl.hzn
• mv /root/tmp-bkp/fips* /usr/local/horizon/conf/flags
• service horizon-workspace restart
• We now successfully joined the replica to AD domain.

The above steps needs to be run on both the replica nodes to update the correct certificates.

Now, we are able to join both the replica nodes to AD successfully without any errors.

Then we could add the load-balancer address to the directory config, to allow log in through the LB address.

Hope this was informative to you..Cheers..

LDAPs configuration for vCenter Server.

Hello there !!

I am covering this post for the configuration of LDAPS authentication for vCenter server or Platform Service controller in case of external PSC deployment.

Before doing this, let me give you an important update that Microsoft gave advisory that everyone needs to enable the LDAP binding and signing at Active Directory domain controllers for secure LDAP authentication. Below are some KB articles for how to enable LDAP binding and signing at AD level.

·         KB4520412: 2020 LDAP channel binding and LDAP signing requirement for Windows

·         KB4034879: LDAP channel binding

·         KB935834: LDAP signing

This LDAP change will affect all applications which are using LDAP authentication with AD. 

So. Lets begin to change the LDAP configuration on vCenter Server 6.7 in my case here.

Step 1

First check which domain controller is preferable to this vCenter server as per site. May be someone has different DCs as per sites basis.

Run below command to get the list of all domain controllers in your environment.

nltest /dclist:yourDomainName

Step 2

Select one of the Domain controller from above list which you want to use for authentication with vCenter server and that is configured as LDAP identity source. Login to vCenter appliance using SSH session and run below command to get LDAP certificate from that domain controller.

openssl s_client -connect dc1.cloudarena.com:636 -showcerts

This command will show the certificate of that domain controller, copy the top most certificate which is always DC certificate. Copy that starting from ---BEGIN CERTIFICATE---- until ----END CERTIFICATE----.

Make sure there is no other characters and space within this selection.

-----BEGIN CERTIFICATE-----

MwMDAwMDBaMBkxFzAVBgNVBAMTDmFkLmdzc2xhYnMub3JnMIIBIjANBgkqhkiG

9w0BAQEFAAOCAQ8AMIIBCgKCAQEA7JFQshqvAH+bsej+FE6IYf3LA38EpMmnsCJV

nvvX1RXoHs5tr8iwbm6fMggRHZA8jHY3Z/wnLkh1Ct+8MylrGVRL4MB1bXeSH7MT

TTCMCI/ikokCO6vkVlG1RP/YcMOIUCLERsgJiZ8qCEZYLdw8ioZuA1kaGQkiJRy8

KZI5lz4nqV9owks1e4TW5TtCTDqorYxBz2x2PsZLTih/fgLf9kRr0QUHc/f8TMuI

3LWdGdodxUKKAP7cHU5awhsOdiDjqWEuYA4gioog0Dd9sE111JvPP0opSPMgnMpf

CWOc04z8dqkR15BChG36Gvgqqbnf77vknDe1RgkFhyK6GjKGTQIDAQABoyQwIjAL

BgNVHQ8EBAMCBDAwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDQYJKoZIhvcNAQEFBQAD

ggEBAC8sNBB5e5WffE9VjU5zcDqvOQqE24XD1bdFeKW/ud6aYwmF5YV4wFpEGkA9

bez2zI5TKVnm2XE4mpwyZHSbbiXzh2SbAQI1QTde9slTFTkib0HsMZYxBE5Xsgdq

RXUX6xvU2sMbHevj13zkGfoF71T72ddq78LTCbrX3EU0jYbHhrKTqRc6qHAv9fz4

2z8xKysVs+CCx8g+qEm+igMxb9/XdA2HUOA8l+NDlH/qS78e9ty0XNayl8ZC/7bZ

-----END CERTIFICATE-----

Save the above certificate content into Notepad file and save file with as .cer extension (e.g. ldap_dc.cer).

Step 3

Now, open the browser and login into the VCSA 6.7 . you need to login with local administrator account i.e administrator@vsphere.local then password.

Click on Menu and choose Administration section

Then click on Configuration option under SSO

Click on identity Sources and then click on ADD IDENTITY SOURCE

Choose identity source type is Active Directory over LDAP and then fill all details of Domain like below and also upload the certificate under section SSL Certificates which we had saved in Step2 

Click ADD and if all details are correct then your identity source will add and show under identity Source Tab.


That's all for this LDAPs configuration. Hope, this will informative to all... Cheers..!!

Upgrading vRealize Automation from 7.3 to 7.6 version.

 Hello !!

Today, I am writing this post for the upgrade of vRealize Automation platform from 7.3 to 7.6 version.

Let me give you a look at my current vRA infrastructure components.

We have vRA 7.3 platform in distributed deployment  mode which have following components/Nodes.

3 X vRA Appliances.

2 X IAAS Web server.

2 X IAAS Manager server 

2 X DEM server for ABC vCenter server Endpoint

2X DEM server for DEF vCenter server Endpoint

2X DEM server for XYZ vCenter server Endpoint

2 X SQL 2016 server

All the IAAS/Manager/DEM/SQL servers are running on Windows 2012 R2 STD edition.

First of all, we need to prepare for some pre-requisites for the upgrade. I have upgraded my 2 vRA distributed platform and basis on that experience, i can say it is quite difficult and you can face many issues during the upgrade.

The Reason of issues is that the mixture of appliances and Windows servers on this vRealize product.

So, let's don't worry about the result, we should prepare our self properly and we can achieve our goal.

Below are things, we need to prepare for that before upgrade.

1. Please ensure that you have taken backups of all vRA and IAAS/Manager/DEM servers.

2. Check and reset the "root" password on all vRA appliances if expired.

3. Disable load balancers for all components if any.

4. Unregister vRealize Business from vRA if you have vRB in your environment.

5. Plz make ensure that no user is able to raise any VM request during the activity.

6. Make a backup of below directories on all vRA appliances.

•  /ect/vcac/
•  /etc/vco/
•  /etc/apache2/
• /etc/rabbitmq/

7. These nodes must have at least 5 GB of free disk space: 

• Primary IaaS Website 
• Microsoft SQL database 
• Model Manager  

8. on vRA appliance nodes, below free space should be available.

• At least 15 GB on the root partition
• 5 GB on the /storage/db partition for the master vRealize Automation appliance 
• 15 GB on the root partition for each replica virtual appliance 

9. Complete these steps if you are upgrading a distributed environment configured with an embedded PostgreSQL database.

•   Examine the files in the pgdata directory on the master host before you upgrade the replica hosts. 
•  Navigate to the PostgreSQL data folder on the master host at /var/vmware/vpostgres/current/pgdata/. 
• Close and remove any .swp files in the pgdata directory. Files with a .swp suffix require you to   close the VI session and delete the file. 
• Verify that all files in this directory have the correct owner name: postgres:<owner-group>

10. In Distributed vRA environment, we need to set the PostgreSQL replication mode to Asynchronous.

11. If you have antivirus solution on IAAS/Manager/DEM servers, plz disable the antivirus or remove if possible.

12. At this point, take a snapshot on all IAAS/Manager/DEM servers.

13. On all IAAS/Manager/DEM/SQL servers, we need to install the JAVA SE Runtime Environment 8, 64 bits, update 181 or later. After you install Java, you must set the environment variable, JAVA_HOME, to the new version on each server node.

14. At this point, take a snapshot on all vRA appliances nodes.

15. At this point, we need to clean up postgres database.

• ·        Make sure taking a snapshot of the master virtual appliance.
• ·        From the vRA VAMI, switch replication from sync to async if not done             previously.
• ·        Run below commands to vacuum the database and remove lob entries.
• ·        su - postgres -c "/opt/vmware/vpostgres/current/bin/vacuumlo -v -p 5432       vcac"
• ·        su - postgres -c "/opt/vmware/vpostgres/current/bin/vacuumdb -f -p 5432        -t pg_largeobject -t pg_largeobject_metadata vcac"
• ·        To reclaim database space, use the vacuum full commands.
• ·        ./psql -U postgres -d vcac
• ·        vacuum full
• ·        vacuum analyze

16. Download the vRealize update repo iso file from VMware portal (VMware-vR-Appliance-7.6.0.317-13027280-updaterepo.iso).

17. At this point, we need to restart the all vRA and Windows nodes because some time if any reboot pending on any servers due to patches or other reason it will fail the upgrade activity.

18. Once all the vRA and Windows nodes came back after reboot and all vRA services registered on VAMI page. we also make sure that all nodes are connected well with each other. you can check this on VAMI page under Cluster tab and Last connected time should be less than 30 seconds for IAAS nodes and 10 mins for vRA nodes.

Alright, this is our pre-requisites and readiness checks before the upgrade starts.

Now, open the master vRA appliance Console and attached the downloaded ISO file via CDROM.

Next, login to the VAMI page, go to "update" tab and click on "Settings" tab under it.

Choose "Use CDROM updates" option and Save settings.

Next, click on "Status" tab and then click on "Check Updates". It will check updates from attached ISO file in CDROM and then gives option for "Install Updates".

Next, click on "Install Updates" and upgrade task will start now.

To Monitor the update progress, you can ssh vRA master nodes and run below commands to see the progress.

# tail -f /opt/vmware/var/log/vami/updatecli.log

# tail -f /opt/vmware/var/log/vami/vami.log

When the vRA appliance update finishes, we hav to reboot the master appliance from VAMI.

In distributed environment, all successful upgraded replica vRA appliances nodes reboot when we reboot the master vRA appliance.

After the vRA appliances reboot, please wait for vRA nodes comes back and all VA services to start and registered as well.

Go to Update tab, Now you will see IAAS nodes updates will start and it will take 30-40 mins time.

Once IAAS updates finish, you can check the status under "Cluster" tab and here it will show that all vRA as well IAAS nodes upgraded to 7.6 version.

                                Post Upgrade tasks

1. Enable all load balancers and check LBs health.

2. Set the PostgreSQL database replication mode to Synchronous.

3. Check the VMs deployment from vRA portal.

4. Register back vRealize business with vRA.

5. Remove all snapshots from vRA appliances as well as IAAS nodes.

That's all.. Cheers !!!