SDDC manager backup task failed with error "Could not start SDDC Manager backup Backup failed : Unexpected error encountered when processing SDDC Manager Backup"

-
Today, I am writing this post related to SDDC manager backup task failure with error message "Could not start SDDC Manager backup Backup failed : Unexpected error encountered when processing SDDC Manager Backup". SDDC manager is running with VCF 4.5.1 version. We have configured the SDDC backups on a external SFTP server. Below is the screenshot of backup task failure.
After checking the operations logs under
 /var/log/vmware/vcf/operationsmanager/operationsmanager.logs

we have found below entries related to backup failures that indicating issue with SOS client service and Too many files open in backup task.

 2023-09-25T11:22:34.655+0000 ERROR [vcf_om,99cc8bcd8fbb453b,d881] [c.v.v.b.helper.SosBackupApiClient,http-nio-127.0.0.1-7300-exec-5] An exception 500 INTERNAL SERVER ERROR: "{"arguments":[],"causes":[{"message":"[Errno 24] Too many open files: '/var/log/vmware/vcf/sddc-support/vcf-sos.log'","type":null}],"context":null,"errorCode":"BACKUP_OPERATION_FAILED","message":"Unexpected error encountered when processing SDDC Manager Backup","referenceToken":"","remediationMessage":null}" occurred while making a call to SOS 2023-09-25T11:22:34.655+0000 ERROR [vcf_om,99cc8bcd8fbb453b,d881] [c.v.v.b.helper.SosBackupApiClient,http-nio-127.0.0.1-7300-exec-5] Response body as string {"arguments":[],"causes":[{"message":"[Errno 24] Too many open files: '/var/log/vmware/vcf/sddc-support/vcf-sos.log'","type":null}],"context":null,"errorCode":"BACKUP_OPERATION_FAILED","message":"Unexpected error encountered when processing SDDC Manager Backup","referenceToken":"","remediationMessage":null} 2023-09-25T11:22:34.655+0000 ERROR [vcf_om,99cc8bcd8fbb453b,d881] [c.v.v.b.helper.SosBackupApiClient,http-nio-127.0.0.1-7300-exec-5] Sos client error message Unexpected error encountered when processing SDDC Manager Backup 2023-09-25T11:22:34.655+0000 DEBUG [vcf_om,99cc8bcd8fbb453b,d881] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7300-exec-5] Processing localizable exception Backup failed : Unexpected error encountered when processing SDDC Manager Backup 2023-09-25T11:22:34.659+0000 ERROR [vcf_om,99cc8bcd8fbb453b,d881] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7300-exec-5] [CUJHPT] BACKUP_FAILED Backup failed : Unexpected error encountered when processing SDDC Manager Backup >

WORKAROUND:

1. To resolve this issue, we can REBOOT the SDDC manager once.
                                     
                                               OR

2. As this error is related to SOS service then we can execute below command to restart the SOS REST service on SDDC manager.

systemctl restart sosrest

Once, above steps done, trigger the backup again and it will be successful then.

....Cheers !!

Comments

Post a Comment

Popular posts from this blog

How to migrate the N-VDS as the host switch to VDS 7.0 in NSX-T 3.x

-
Image
  Hello There, In this article, i am covering how to migrate the ESXi host switch from N-VDS to VDS 7.0 switch in NSX-T 3.2.x version. When using N-VDS as the host switch,  NSX-T  is represented as an opaque network in  vCenter Server . N-VDS owns one or more of the physical interfaces (pNICs) on the transport node, and port configuration is performed from  NSX-T Data Center . You can migrate your host switch to  vSphere  Distributed Switch (VDS) 7.0 for optimal pNIC usage and manage the networking for  NSX-T  hosts from  vCenter Server . When running  NSX-T  on a VDS switch, a segment is represented as an  NSX-T  Distributed Virtual Port Groups. Any changes to the segments on the  NSX-T  network are synchronized in  vCenter Server. We have an NSX-T environment running with NSX-T 3.2.2.1 version. this environment was designed and implemented with NSX-T 2.x version and that time we used the N-VDS as host switch configuration on the transport nodes bcuz VDS was not supported that time. 
Read more

vROPS appliances password remediation tasks failed from SDDC manager

-
Image
Issue details:-   Password remediation tasks on SDDC manager getting failed with below error. However, we are able to connect SSH with root password on the vROPS appliances, no issues with credentials. I checked Operations logs on SDDC manager and found below logs, indicating the SSH connectivity issues from SDDC to vROPS appliances. Tried to do SSH vROPS appliance from SDDC manager, getting below error... Seems some issue with ECDSA key.. Resolution:- SSH to vROPS appliance and retrieve the ECDSA ssh keys as below. Now, we have 2 options to update the correct SSH keys for vROPS appliance on SDDC manager known_Hosts files located at below location.   /root/.ssh/known_hosts   /etc/vmware/vcf/commonsvcs/known_hosts   /home/vcf/.ssh/known_hosts   /opt/vmware/vcf/commonsvcs/defaults/hosts/known_hosts First option is manually copy and paste the SSH key on all the known_hosts files and restart the SDDC manager services and then try to remediate the password again. it will be successful this
Read more

How to Import/Register a VM into vRA portal

-
Image
You can import an unmanaged virtual machine to a vRealize Automation environment. An unmanaged virtual machine exists in a vCenter but is not managed in a vRealize Automation environment and cannot be viewed in the vRA portal . Once you import an unmanaged virtual machine, the virtual machine is managed by using the vRealize Automation management interface. you can see the virtual machine on the Managed Machines tab or the Items tab. Please follow below steps to Import/Register VM into a vRA portal. Log in to vRealize Automation as a fabric administrator and as a business group manager. If you are importing virtual machines that use static IP addresses, prepare a properly configured address pool.  If you use bulk import to import a virtual machine with a static IP address that is allocated to another virtual machine, the import fails. Create a Blueprint and should not have even network attached as well. See below snapin. Create Network profile that shoul
Read more