vRealize Automation 7.6 cluster replica nodes AD joining issue

Hi There,

This article is for how to resolve the vRA replica nodes domain joining issues.

Let me tell you the background of this issue, I have upgraded my vRA environment 7.3 to 7.6 version successfully. After the upgrade, I introduced 2 replica nodes into vRA cluster to make it distributed deployment.

Once, i have added the replica nodes into vRA cluster successfully. I logged into my vRA portal default tenant and try to add these 2 nodes( which are basically called as connectors into vRA portal) to my Active Directory domain so that i can leverage the benefit of Load balancer and user authentication can be made via any nodes into the vRA cluster.

On vRA portal, go to Administration Tab, under Directories Management, click on Connectors tab

When i click on Join domain tab on my second Replica node, i got below error.

Connector communication failed with response: Certificate for <VRA002.cloudarena.com> doesn't match common name of the certificate subject: VRA001.cloudarena.com for the connector VRA002.cloudarena.com

This error is same for third Replica node as well.

Connector communication failed with response: Certificate for <VRA003.cloudarena.com> doesn't match common name of the certificate subject: VRA001.cloudarena.com for the connector VRA003.cloudarena.com


I checked my vRA certificate status and all my certificates are valid until Sep 2020. 

I rebooted the whole stack of my vRA environment but still issue is same.

Then, we have a look on certificate part on replica nodes and figure out that 

• As per the error, the newly deployed vRA replica appliances were using the internal horizon certificate for the original appliance VRA001.cloudarena.com
• This certificate is unrelated to the one we would have generated and applied to vRA.
• We tried to run a command to update the certificates locally on the 2 new replicas
• Run this command >>
• /usr/local/horizon/scripts/installExternalCertificate.hzn --ca /usr/local/horizon/conf/root_ca.pem --cert /usr/local/horizon/conf/VRA002.cloudarena.com_cert.pem --key /usr/local/horizon/conf/VRA002.cloudarena.com_key.pem --alias "horizoninternalcert"
  /usr/local/horizon/conf/root_ca.pem --cert /usr/local/horizon/conf/VRA002.cloudarena.com_cert.pem --key /usr/local/horizon/conf/VRA002.cloudarena.com_key.pem --alias "horizoninternalcert" 
•  
• This failed with the following error;
• keytool error: java.io.IOException: DER length more than 4 bytes: 109
• Run steps to workaround this on both vRA replicas nodes;
• mkdir /root/tmp-bkp
• mv /usr/local/horizon/conf/flags/fips* /root/tmp-bkp
• /usr/local/horizon/scripts/secure/wizardssl.hzn
• mv /root/tmp-bkp/fips* /usr/local/horizon/conf/flags
• service horizon-workspace restart
• We now successfully joined the replica to AD domain.

The above steps needs to be run on both the replica nodes to update the correct certificates.

Now, we are able to join both the replica nodes to AD successfully without any errors.

Then we could add the load-balancer address to the directory config, to allow log in through the LB address.

Hope this was informative to you..Cheers..

LDAPs configuration for vCenter Server.

Hello there !!

I am covering this post for the configuration of LDAPS authentication for vCenter server or Platform Service controller in case of external PSC deployment.

Before doing this, let me give you an important update that Microsoft gave advisory that everyone needs to enable the LDAP binding and signing at Active Directory domain controllers for secure LDAP authentication. Below are some KB articles for how to enable LDAP binding and signing at AD level.

·         KB4520412: 2020 LDAP channel binding and LDAP signing requirement for Windows

·         KB4034879: LDAP channel binding

·         KB935834: LDAP signing

This LDAP change will affect all applications which are using LDAP authentication with AD. 

So. Lets begin to change the LDAP configuration on vCenter Server 6.7 in my case here.

Step 1

First check which domain controller is preferable to this vCenter server as per site. May be someone has different DCs as per sites basis.

Run below command to get the list of all domain controllers in your environment.

nltest /dclist:yourDomainName

Step 2

Select one of the Domain controller from above list which you want to use for authentication with vCenter server and that is configured as LDAP identity source. Login to vCenter appliance using SSH session and run below command to get LDAP certificate from that domain controller.

openssl s_client -connect dc1.cloudarena.com:636 -showcerts

This command will show the certificate of that domain controller, copy the top most certificate which is always DC certificate. Copy that starting from ---BEGIN CERTIFICATE---- until ----END CERTIFICATE----.

Make sure there is no other characters and space within this selection.

-----BEGIN CERTIFICATE-----

MwMDAwMDBaMBkxFzAVBgNVBAMTDmFkLmdzc2xhYnMub3JnMIIBIjANBgkqhkiG

9w0BAQEFAAOCAQ8AMIIBCgKCAQEA7JFQshqvAH+bsej+FE6IYf3LA38EpMmnsCJV

nvvX1RXoHs5tr8iwbm6fMggRHZA8jHY3Z/wnLkh1Ct+8MylrGVRL4MB1bXeSH7MT

TTCMCI/ikokCO6vkVlG1RP/YcMOIUCLERsgJiZ8qCEZYLdw8ioZuA1kaGQkiJRy8

KZI5lz4nqV9owks1e4TW5TtCTDqorYxBz2x2PsZLTih/fgLf9kRr0QUHc/f8TMuI

3LWdGdodxUKKAP7cHU5awhsOdiDjqWEuYA4gioog0Dd9sE111JvPP0opSPMgnMpf

CWOc04z8dqkR15BChG36Gvgqqbnf77vknDe1RgkFhyK6GjKGTQIDAQABoyQwIjAL

BgNVHQ8EBAMCBDAwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDQYJKoZIhvcNAQEFBQAD

ggEBAC8sNBB5e5WffE9VjU5zcDqvOQqE24XD1bdFeKW/ud6aYwmF5YV4wFpEGkA9

bez2zI5TKVnm2XE4mpwyZHSbbiXzh2SbAQI1QTde9slTFTkib0HsMZYxBE5Xsgdq

RXUX6xvU2sMbHevj13zkGfoF71T72ddq78LTCbrX3EU0jYbHhrKTqRc6qHAv9fz4

2z8xKysVs+CCx8g+qEm+igMxb9/XdA2HUOA8l+NDlH/qS78e9ty0XNayl8ZC/7bZ

-----END CERTIFICATE-----

Save the above certificate content into Notepad file and save file with as .cer extension (e.g. ldap_dc.cer).

Step 3

Now, open the browser and login into the VCSA 6.7 . you need to login with local administrator account i.e administrator@vsphere.local then password.

Click on Menu and choose Administration section

Then click on Configuration option under SSO

Click on identity Sources and then click on ADD IDENTITY SOURCE

Choose identity source type is Active Directory over LDAP and then fill all details of Domain like below and also upload the certificate under section SSL Certificates which we had saved in Step2 

Click ADD and if all details are correct then your identity source will add and show under identity Source Tab.


That's all for this LDAPs configuration. Hope, this will informative to all... Cheers..!!

Upgrading vRealize Automation from 7.3 to 7.6 version.

 Hello !!

Today, I am writing this post for the upgrade of vRealize Automation platform from 7.3 to 7.6 version.

Let me give you a look at my current vRA infrastructure components.

We have vRA 7.3 platform in distributed deployment  mode which have following components/Nodes.

3 X vRA Appliances.

2 X IAAS Web server.

2 X IAAS Manager server 

2 X DEM server for ABC vCenter server Endpoint

2X DEM server for DEF vCenter server Endpoint

2X DEM server for XYZ vCenter server Endpoint

2 X SQL 2016 server

All the IAAS/Manager/DEM/SQL servers are running on Windows 2012 R2 STD edition.

First of all, we need to prepare for some pre-requisites for the upgrade. I have upgraded my 2 vRA distributed platform and basis on that experience, i can say it is quite difficult and you can face many issues during the upgrade.

The Reason of issues is that the mixture of appliances and Windows servers on this vRealize product.

So, let's don't worry about the result, we should prepare our self properly and we can achieve our goal.

Below are things, we need to prepare for that before upgrade.

1. Please ensure that you have taken backups of all vRA and IAAS/Manager/DEM servers.

2. Check and reset the "root" password on all vRA appliances if expired.

3. Disable load balancers for all components if any.

4. Unregister vRealize Business from vRA if you have vRB in your environment.

5. Plz make ensure that no user is able to raise any VM request during the activity.

6. Make a backup of below directories on all vRA appliances.

•  /ect/vcac/
•  /etc/vco/
•  /etc/apache2/
• /etc/rabbitmq/

7. These nodes must have at least 5 GB of free disk space: 

• Primary IaaS Website 
• Microsoft SQL database 
• Model Manager  

8. on vRA appliance nodes, below free space should be available.

• At least 15 GB on the root partition
• 5 GB on the /storage/db partition for the master vRealize Automation appliance 
• 15 GB on the root partition for each replica virtual appliance 

9. Complete these steps if you are upgrading a distributed environment configured with an embedded PostgreSQL database.

•   Examine the files in the pgdata directory on the master host before you upgrade the replica hosts. 
•  Navigate to the PostgreSQL data folder on the master host at /var/vmware/vpostgres/current/pgdata/. 
• Close and remove any .swp files in the pgdata directory. Files with a .swp suffix require you to   close the VI session and delete the file. 
• Verify that all files in this directory have the correct owner name: postgres:<owner-group>

10. In Distributed vRA environment, we need to set the PostgreSQL replication mode to Asynchronous.

11. If you have antivirus solution on IAAS/Manager/DEM servers, plz disable the antivirus or remove if possible.

12. At this point, take a snapshot on all IAAS/Manager/DEM servers.

13. On all IAAS/Manager/DEM/SQL servers, we need to install the JAVA SE Runtime Environment 8, 64 bits, update 181 or later. After you install Java, you must set the environment variable, JAVA_HOME, to the new version on each server node.

14. At this point, take a snapshot on all vRA appliances nodes.

15. At this point, we need to clean up postgres database.

• ·        Make sure taking a snapshot of the master virtual appliance.
• ·        From the vRA VAMI, switch replication from sync to async if not done             previously.
• ·        Run below commands to vacuum the database and remove lob entries.
• ·        su - postgres -c "/opt/vmware/vpostgres/current/bin/vacuumlo -v -p 5432       vcac"
• ·        su - postgres -c "/opt/vmware/vpostgres/current/bin/vacuumdb -f -p 5432        -t pg_largeobject -t pg_largeobject_metadata vcac"
• ·        To reclaim database space, use the vacuum full commands.
• ·        ./psql -U postgres -d vcac
• ·        vacuum full
• ·        vacuum analyze

16. Download the vRealize update repo iso file from VMware portal (VMware-vR-Appliance-7.6.0.317-13027280-updaterepo.iso).

17. At this point, we need to restart the all vRA and Windows nodes because some time if any reboot pending on any servers due to patches or other reason it will fail the upgrade activity.

18. Once all the vRA and Windows nodes came back after reboot and all vRA services registered on VAMI page. we also make sure that all nodes are connected well with each other. you can check this on VAMI page under Cluster tab and Last connected time should be less than 30 seconds for IAAS nodes and 10 mins for vRA nodes.

Alright, this is our pre-requisites and readiness checks before the upgrade starts.

Now, open the master vRA appliance Console and attached the downloaded ISO file via CDROM.

Next, login to the VAMI page, go to "update" tab and click on "Settings" tab under it.

Choose "Use CDROM updates" option and Save settings.

Next, click on "Status" tab and then click on "Check Updates". It will check updates from attached ISO file in CDROM and then gives option for "Install Updates".

Next, click on "Install Updates" and upgrade task will start now.

To Monitor the update progress, you can ssh vRA master nodes and run below commands to see the progress.

# tail -f /opt/vmware/var/log/vami/updatecli.log

# tail -f /opt/vmware/var/log/vami/vami.log

When the vRA appliance update finishes, we hav to reboot the master appliance from VAMI.

In distributed environment, all successful upgraded replica vRA appliances nodes reboot when we reboot the master vRA appliance.

After the vRA appliances reboot, please wait for vRA nodes comes back and all VA services to start and registered as well.

Go to Update tab, Now you will see IAAS nodes updates will start and it will take 30-40 mins time.

Once IAAS updates finish, you can check the status under "Cluster" tab and here it will show that all vRA as well IAAS nodes upgraded to 7.6 version.

                                Post Upgrade tasks

1. Enable all load balancers and check LBs health.

2. Set the PostgreSQL database replication mode to Synchronous.

3. Check the VMs deployment from vRA portal.

4. Register back vRealize business with vRA.

5. Remove all snapshots from vRA appliances as well as IAAS nodes.

That's all.. Cheers !!!

vCenter 6.5 file based Backup and Restore process

                         vCenter 6.5 file based Backup and Restore process

Hello Everyone,

This article is for how to take VCSA 6.5 file based Backup and Restore process in case of vCenter server appliance got failed due to any O/S or application issues.

VCSA 6.5 has in-built feature available with management interface of vCenter server appliance 6.5 (VAMI) page. With this native Backup and Restore VCSA 6.5 features removes the dependency on third-party native backup tools to backup and restore vCenter server appliance. Native Backup feature of VCSA 6.5 supports the backup and restore of vCenter server appliance and also can take backup and restore of platform services controller.

This native backup and restore feature of VCSA 6.5 helps you to protect vCenter data and helps us to reduce the downtime required for to restore the vCenter management stack. Backup of VCSA 6.5 process collects the important files into tar bundle and compress the bundle to reduce the load on the network. You have option to encrypt the compressed file before transmission the backup files to the backup storage location as well. You can specify the password during encryption and specify the same password to decrypt the file during the restore process.

First of all, we will start with the backup of VCSA 6.5 appliance through VAMI console of vCenter appliance.

In the Summary tab, Click on “Backup” to start the backup of the vCenter Sever appliance.

VCSA backup can be backed and sent up to different location using different protocols:

·         FTP

·         FTPS

·         HTTP

·         HTTPS

·         SCP

Here, I choose SCP option for backup. Choose SCP from drop down and fill all details.

CLICK NEXT

During VCSA backup, Minimum set of data needed to restore the appliance will be backed up by default. This includes the data such as OS, VC services, vCenter Server database,  inventory and configuration. In addition to that, you can also choose to backup additional things such as Historical data( Stats, Events, alarms and tasks ) in the vCenter Server database.

CLICK NEXT

Review the backup option selection and Click on Finish.

CLICK FINISH

Backup Task finished successfully.

So, backup task completed successfully. Now, we look at the backup folder which files are stored there. See below snap-in.

Restore vCenter Server Appliance 6.5

We are done with the backup of the vCenter server appliance 6.5. Restore the VCSA 6.5 from the backup file if it is required. Restore process is the straight forward process. Restore needs to be performed from the VCSA 6.5 installation media.  

There is a important point here to make a note that VCSA 6.5 installer versions should be the same as the current VCSA 6.5 version then only we can restore the VCSA otherwise Restore process can not be initiated.

For Example- If your current VCSA 6.5 version is 15259038 then you need to run the same version installer for restore process.


Launch the VCSA 6.5 installer, Click on Restore to restore VCSA from a previously created vCenter Server appliance backup.

Specify the backup location details and credentials to retrieve the backup metadata information. 

Restores option will restore vCenter server appliance to a brand new appliance with the existing backup data.

In this Restore process, it will first deploy new VCSA appliance and then copy the backup data to this new brand VCSA appliance.

That's all, stay tuned, Cheers...