top of page

VCF 4.2.1 SDDC Going down due to root partition issue

  • viquarmca
  • Jul 4, 2022
  • 3 min read

In this blog we will take about my experience with VCF 4.2.1 GUI down issue, it was very wired that the GUI was not loading . After a lot of troubleshooting found the issue . This issue has been fixed in VCF 4.4



  • When check the SDDC VM , we found the root /var/log has Audit.log 12GB and Auth.log 12GB

  • df -h

Filesystem Size Used Avail Use% Mounted on

devtmpfs 7.9G 0 7.9G 0% /dev

tmpfs 7.9G 4.0K 7.9G 1% /dev/shm

tmpfs 7.9G 298M 7.6G 4% /run

tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup

/dev/sda4 28G 28G 0 100% /

tmpfs 7.9G 64K 7.9G 1% /tmp

/dev/sda2 120M 27M 85M 24% /boot

/dev/mapper/data_vg-data 7.9G 185M 7.3G 3% /data

/dev/mapper/lcmmount_vg-nfs--mount 503G 81G 397G 17% /nfs/vmware/vcf/nfs-mount

/dev/mapper/data_vg-vmware 63G 3.6G 56G 7% /var/log/vmware

/dev/mapper/data_vg-vcf 181G 1.7G 170G 1% /opt/vmware/vcf

tmpfs1.6G01.6G0% /run/user/1000


  • cd /var/log/

  • df -h .

Filesystem Size Used Avail Use% Mounted on

/dev/sda4 28G 28G 0 100% /


root@vcfsddc[ /var/log ]# ls -l

total 12610320

drwx------ 2 root root 4096 Feb 3 2021 audit

-rw------- 1 root root 12884303872 Aug 12 12:23 auth.log

-rw-r--r-- 1 root root 833 May 7 12:19 boot.log

-rw-rw---- 1 root utmp 11520 Aug 10 15:58 btmp

-rw------- 1 root root 4624384 Aug 12 12:20 cron

drwx------ 2 root root 4096 Aug 9 04:01 devicelist

lrwxrwxrwx 1 root root 30 Jan 21 2021 dracut.log -> /var/opt/dracut/log/dracut.log

-rw-r--r-- 1 root root 0 Feb 3 2021 installer-kickstart.log

-rw-r--r-- 1 root root 0 Feb 3 2021 installer.log

drwxr-sr-x 3 root systemd-journal 4096 Jun 16 08:20 journal

-rw-rw-r-- 1 root utmp 292292 Aug 12 12:23 lastlog

drwxr-x--- 2 root root 4096 Aug 11 18:10 loginsight-agent

-rw-r--r-- 1 root root 2519040 Aug 12 12:20 maillog

-rw-r----- 1 root root 1478656 Aug 12 12:26 messages

drwxr-xr-x 2 root root 4096 Jun 24 03:15 nginx

drwx------ 2 postgres users 12288 Aug 10 00:00 postgres

drwx------ 2 root root 4096 Feb 3 2021 private

drwx------ 2 root root 4096 Aug 9 04:01 rpmcheck

-rw-r--r-- 1 root root 19582976 Aug 12 12:20 sa

drwx------ 2 root root 4096 Aug 9 04:01 sgidlist

-rw------- 1 root root 11297 Jun 16 08:31 stigreport.log

drwx------ 2 root root 4096 Aug 9 04:02 suidlist

-rw------- 1 root root 64128 Aug 12 12:26 tallylog

-rw-r--r-- 1 root root 0 Feb 3 2021 tdnf.log

drwxr-xr-x 6 root root 4096 Jul 22 12:35 vmware

-rw------- 1 root root 719 Aug 10 16:00 vmware-network.1.log

-rw------- 1 root root 741 Aug 10 15:53 vmware-network.2.log

-rw------- 1 root root 741 Jul 20 08:20 vmware-network.3.log

-rw------- 1 root root 719 Jul 19 12:41 vmware-network.4.log

-rw------- 1 root root 741 Jul 19 12:29 vmware-network.5.log

-rw------- 1 root root 719 Jul 19 08:36 vmware-network.6.log

-rw------- 1 root root 741 Jul 19 08:20 vmware-network.7.log

-rw------- 1 root root 719 Jul 15 11:36 vmware-network.8.log

-rw------- 1 root root 719 Jun 11 15:09 vmware-network.9.log

-rw------- 1 root root 719 Aug 10 19:45 vmware-network.log

-rw------- 1 root root 31554 Aug 10 19:40 vmware-vgauthsvc.log.0

-rw------- 1 root root 45668 Aug 10 19:45 vmware-vmsvc-root.log

-rw------- 1 root root 3480 Aug 10 19:45 vmware-vmtoolsd-root.log

-rw-rw-r-- 1 root utmp 184320 Aug 12 12:23 wtmp


Cause : It occurs due to log rotation policy for auth.log and audit.log.


Fix :

  • Take the snapshots of the SDDC Manager.

  • Deleted the auth and audit logs both

root@vcfsddc[ /var/log ]# uptime

12:29:27 up 1 day, 16:49, 2 users, load average: 0.00, 0.02, 0.00

root@vcfsddc[ /var/log ]# rm auth.log

root@vcfsddc[ /var/log ]# cd audit

root@vcfsddc[ /var/log/audit ]# ls

audit.log

root@vcfsddc[ /var/log/audit ]# ls -l

total 12425896

-rw------- 1 root root 12724088832 Aug 12 12:31 audit.log

  • Reboot the sddc manager , then the space was back to normal and the GUI was able to login.

  • After few weeks the partition got again full , later vmware gss gave an workaround which is mention in the KB 86401

Comments


bottom of page