Today Blog we will talk about VCF 4.3.1 to 4.4.1 SDDC Drift Configuration failure with error Failed to Run Clean VUM DB . The configuration drift bundle applies configuration changes required for 2nd party software components in the VMware Cloud Foundation Bill of Materials for the target release i.e. VCF 4.4.1 . This issue I have seen it for the 1st time , the Error msg says about Failed to run clean vum db , however that was not the issue . Lets check out the below steps how we find the issue and fix it.
Check the logs on /var/log/vmware/vcf/lcm/thirdparty/upgrades/ , found the error msg as
2022-09-14T03:02:12.886+0000 INFO [vcf_migration,0000000000000000,0000] [c.v.e.s.c.v.ExecutionContextToValidationConverter,main] Collecting processing task errors: FAILED_TO_RUN_CLEAN_VUM_DB for validation aggregation
2022-09-14T03:02:12.886+0000 INFO [vcf_migration,0000000000000000,0000] [c.v.e.s.c.v.ExecutionContextToValidationConverter,main] Collecting processing task errors: FAILED_TO_RUN_CLEAN_VUM_DB for validation aggregation
2022-09-14T03:02:12.889+0000 DEBUG [vcf_migration,0000000000000000,0000] [c.v.v.m.ContractRecipeExecutor,main] Validation output Optional[com.vmware.vcf.migration.model.WorkflowOutput@621500b3]
2022-09-14T03:02:12.889+0000 DEBUG [vcf_migration,0000000000000000,0000] [c.v.v.m.ContractRecipeExecutor,main] Validation output Optional[com.vmware.vcf.migration.model.WorkflowOutput@621500b3]
2022-09-14T03:02:12.898+0000 INFO [vcf_migration,0000000000000000,0000] [c.v.v.m.ContractRecipeExecutor,main] Execution ID: cb156fe7-fa79-463f-bfde-689b4eba1482, Status: COMPLETED_WITH_FAILURE
2022-09-14T03:02:12.898+0000 INFO [vcf_migration,0000000000000000,0000] [c.v.v.m.ContractRecipeExecutor,main] Execution ID: cb156fe7-fa79-463f-bfde-689b4eba1482, Status: COMPLETED_WITH_FAILURE
2022-09-14T03:02:12.905+0000 INFO [vcf_migration,0000000000000000,0000] [c.v.v.migration.SddcManagerMigration,main] Output Values: {}
2022-09-14T03:02:12.905+0000 INFO [vcf_migration,0000000000000000,0000] [c.v.v.migration.SddcManagerMigration,main] Output Values: {}
2022-09-14T03:02:12.905+0000 INFO [vcf_migration,0000000000000000,0000] [com.vmware.vcf.migration.Application,main] SDDC Manager migration completed
2022-09-14T03:02:12.905+0000 INFO [vcf_migration,0000000000000000,0000] [com.vmware.vcf.migration.Application,main] SDDC Manager migration completed
2022-09-14T03:01:24.713+0000 ERROR [vcf_migration,0000000000000000,0000]
[c.v.e.s.o.model.error.ErrorFactory,pool-5-thread-15] [6469SC] FAILED_TO_RUN_CLEAN_VUM_DB Failed to run clean VUM DB.
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to run clean VUM DB.
at com.vmware.vcf.migration.actions.workarounds.CleanVumDBAction.execute(CleanVumDBAction.java:315)
at com.vmware.vcf.migration.actions.workarounds.CleanVumDBAction.execute(CleanVumDBAction.java:40)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:62)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:159)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:144)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:400)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.processTask(ProcessingTaskSubscriber.java:520)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:124)
at sun.reflect.GeneratedMethodAccessor598.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:87)
at com.google.common.eventbus.Subscriber$1.run(Subscriber.java:72)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.vmware.vapi.std.errors.ServiceUnavailable: ServiceUnavailable (com.vmware.vapi.std.errors.service_unavailable) => {
messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
id = com.vmware.vapi.endpoint.cis.ServiceUnavailable,
defaultMessage = Service unavailable.,
args = [],
params = <null>,
localized = <null>
}],
data = <null>,
errorType = SERVICE_UNAVAILABLE
Tried to follow the VUM DB Reset KB 2147284 , but didnt work. the upgrade failed again .
On vCenter Appliance checked the /var/log/vmware/applmgmt: Failed to get issuers certificates
2022-09-14T05:31:16.0 [2661]DEBUG:vmware.appliance.extensions.authorization.authorization_sso:User=vsphere.local\vpxd-extension-d93b7f11-f22a-4436-a03e-eb9e72c68489, groups=
set()
2022-09-14T05:31:16.0 [2661]DEBUG:root:Validated user privileges in localstore or SSO
2022-09-14T05:31:16.0 [2661]DEBUG:vmware.appliance.extensions.authorization.authorization_sso:Required privileges = ['ModifyLocalConf']
2022-09-14T05:31:16.4 [2661]ERROR:vmware.appliance.extensions.authorization.authorization_sso:FindAllParentGroups Failed {(vmodl.fault.SystemError) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
msg = 'Internal server error',
faultCause = <unset>,
faultMessage = (vmodl.LocalizableMessage) [],
reason = "Failed to get issuers certificates: [Can't contact LDAP server]"
}}
Traceback (most recent call last):
File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authorization/authorization_sso.py", line 285, in get_groups
user_id)
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 583, in <lambda>
self.f(*(self.args + (obj,) + args), **kwargs)
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 373, in _InvokeMethod
return self._stub.InvokeMethod(self, info, args)
File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1802, in InvokeMethod
raise obj
pyVmomi.VmomiSupport.vmodl.fault.SystemError: (vmodl.fault.SystemError) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
msg = 'Internal server error',
faultCause = <unset>,
faultMessage = (vmodl.LocalizableMessage) [],
reason = "Failed to get issuers certificates: [Can't contact LDAP server]"
}
signing_chain = self.validate_certificate()
File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 685, in validate_certificate
'One or more certificates cannot be verified.')
vmware.appliance.extensions.authentication.authentication_sso.AuthenticationError: One or more certificates cannot be verified.
Fix :
1. Reset the STS certificate on the Management vCenter (https://kb.vmware.com/s/article/76719) 2. Reset the solution user certificates on the Management vCenter using Option 6 of the certificate manager (https://kb.vmware.com/s/article/2112283) That should restart all services on the vCenter as well. 3. Restart services on all remaining vCenter . 4. Attempt the Configuration Drift Bundle update again from the SDDC Manager UI.
Comentários