Today Post we will talk about backup failure on the SDDC Manager. Backup configuration is successfully completed but when we click on button Backup Now it fails with the error. Below is the screen shot of the error .
Upon checking the logs /var/log/vmware/vcf/operationmanaer/operationmanaer.log , i found error with Too many open files on /var/log/vmware/vcf/sddc-support/backup
2023-03-17T13:37:47.118+0000 ERROR [vcf_om,2b34c0312ed54e95,8b45] [c.v.v.b.helper.SosBackupApiClient,http-nio-127.0.0.1-7300-exec-10] Response body as string {"arguments":[],"causes":[{"message":"[Errno 24] Too many open files: '/var/log/vmware/vcf/sddc-support/backup-2023-03-17-13-37-47-2206/sos.log'","type":null}],"context":null,"errorCode":"BACKUP_OPERATION_FAILED","message":"Unexpected error encountered when processing SDDC Manager Backup","referenceToken":"","remediationMessage":null}
2023-03-17T13:37:47.118+0000 ERROR [vcf_om,2b34c0312ed54e95,8b45] [c.v.v.b.helper.SosBackupApiClient,http-nio-127.0.0.1-7300-exec-10] Sos client error message Unexpected error encountered when processing SDDC Manager Backup
2023-03-17T13:37:47.118+0000 DEBUG [vcf_om,2b34c0312ed54e95,8b45] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7300-exec-10] Processing localizable exception Backup failed : Unexpected error encountered when processing SDDC Manager Backup
2023-03-17T13:37:47.119+0000 ERROR [vcf_om,2b34c0312ed54e95,8b45] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7300-exec-10] [35HG00] BACKUP_FAILED Backup failed : Unexpected error encountered when processing SDDC Manager Backup
com.vmware.vcf.backuprestore.errors.BackupRestoreException: Backup failed : Unexpected error encountered when processing SDDC Manager Backup
checked the sos log /var/log/vmware/vcf/sddc-support/backup-2023-03-17-13-37-47
2023-03-17T13:37:47.112Z [INFO backupapi.py::backup::81::MainThread] Initiate SDDC Manager Backup operation
2023-03-17T13:37:47.115Z [INFO backupapi.py::backup::111::MainThread] Backup retention policy: {'retainNumOfMostRecent': 2, 'retainHourlyForDays': 2, 'retainDailyForDays': 2}
2023-03-17T13:37:47.116Z [ERROR backupapi.py::backup::173::MainThread] Unexpected error encountered when processing SDDC Manager Backup
2023-03-17T13:37:47.116Z [ERROR backupapi.py::backup::174::MainThread] Traceback (most recent call last):
File "sosapi/controllers/backupapi.py", line 114, in backup
File "./sos.py", line 90, in __init__
File "utils/log.py", line 144, in __init__
File "utils/log.py", line 170, in create_file_handlers
File "/usr/lib/python3.7/logging/__init__.py", line 1087, in __init__
File "/usr/lib/python3.7/logging/__init__.py", line 1116, in _open
OSError: [Errno 24] Too many open files: '/var/log/vmware/vcf/sddc-support/backup-2023-03-17-13-37-47-2206/sos.log'
WORK AROUND :
Reboot SDDC Manager , which cleared the files .
After which click on button BACKUP NOW , it started taking the backup .
Comments