SDDC manager backup task failed with error "Could not start SDDC Manager backup Backup failed : Unexpected error encountered when processing SDDC Manager Backup"

Today, I am writing this post related to SDDC manager backup task failure with error message "Could not start SDDC Manager backup Backup failed : Unexpected error encountered when processing SDDC Manager Backup". SDDC manager is running with VCF 4.5.1 version. We have configured the SDDC backups on a external SFTP server. Below is the screenshot of backup task failure.

After checking the operations logs under

 /var/log/vmware/vcf/operationsmanager/operationsmanager.logs

we have found below entries related to backup failures that indicating issue with SOS client service and Too many files open in backup task.

 2023-09-25T11:22:34.655+0000 ERROR [vcf_om,99cc8bcd8fbb453b,d881] [c.v.v.b.helper.SosBackupApiClient,http-nio-127.0.0.1-7300-exec-5] An exception 500 INTERNAL SERVER ERROR: "{"arguments":[],"causes":[{"message":"[Errno 24] Too many open files: '/var/log/vmware/vcf/sddc-support/vcf-sos.log'","type":null}],"context":null,"errorCode":"BACKUP_OPERATION_FAILED","message":"Unexpected error encountered when processing SDDC Manager Backup","referenceToken":"","remediationMessage":null}" occurred while making a call to SOS 2023-09-25T11:22:34.655+0000 ERROR [vcf_om,99cc8bcd8fbb453b,d881] [c.v.v.b.helper.SosBackupApiClient,http-nio-127.0.0.1-7300-exec-5] Response body as string {"arguments":[],"causes":[{"message":"[Errno 24] Too many open files: '/var/log/vmware/vcf/sddc-support/vcf-sos.log'","type":null}],"context":null,"errorCode":"BACKUP_OPERATION_FAILED","message":"Unexpected error encountered when processing SDDC Manager Backup","referenceToken":"","remediationMessage":null} 2023-09-25T11:22:34.655+0000 ERROR [vcf_om,99cc8bcd8fbb453b,d881] [c.v.v.b.helper.SosBackupApiClient,http-nio-127.0.0.1-7300-exec-5] Sos client error message Unexpected error encountered when processing SDDC Manager Backup 2023-09-25T11:22:34.655+0000 DEBUG [vcf_om,99cc8bcd8fbb453b,d881] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7300-exec-5] Processing localizable exception Backup failed : Unexpected error encountered when processing SDDC Manager Backup 2023-09-25T11:22:34.659+0000 ERROR [vcf_om,99cc8bcd8fbb453b,d881] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7300-exec-5] [CUJHPT] BACKUP_FAILED Backup failed : Unexpected error encountered when processing SDDC Manager Backup >

WORKAROUND:

1. To resolve this issue, we can REBOOT the SDDC manager once.

                                     

                                               OR

2. As this error is related to SOS service then we can execute below command to restart the SOS REST service on SDDC manager.

systemctl restart sosrest

Once, above steps done, trigger the backup again and it will be successful then.

....Cheers !!