In this blog we will take about my experience with VCF 4.2.1 GUI down issue, it was very wired that the GUI was not loading . After a lot of troubleshooting found the issue . This issue has been fixed in VCF 4.4
When check the SDDC VM , we found the root /var/log has Audit.log 12GB and Auth.log 12GB
df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 7.9G 0 7.9G 0% /dev
tmpfs 7.9G 4.0K 7.9G 1% /dev/shm
tmpfs 7.9G 298M 7.6G 4% /run
tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup
/dev/sda4 28G 28G 0 100% /
tmpfs 7.9G 64K 7.9G 1% /tmp
/dev/sda2 120M 27M 85M 24% /boot
/dev/mapper/data_vg-data 7.9G 185M 7.3G 3% /data
/dev/mapper/lcmmount_vg-nfs--mount 503G 81G 397G 17% /nfs/vmware/vcf/nfs-mount
/dev/mapper/data_vg-vmware 63G 3.6G 56G 7% /var/log/vmware
/dev/mapper/data_vg-vcf 181G 1.7G 170G 1% /opt/vmware/vcf
tmpfs1.6G01.6G0% /run/user/1000
cd /var/log/
df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/sda4 28G 28G 0 100% /
root@vcfsddc[ /var/log ]# ls -l
total 12610320
drwx------ 2 root root 4096 Feb 3 2021 audit
-rw------- 1 root root 12884303872 Aug 12 12:23 auth.log
-rw-r--r-- 1 root root 833 May 7 12:19 boot.log
-rw-rw---- 1 root utmp 11520 Aug 10 15:58 btmp
-rw------- 1 root root 4624384 Aug 12 12:20 cron
drwx------ 2 root root 4096 Aug 9 04:01 devicelist
lrwxrwxrwx 1 root root 30 Jan 21 2021 dracut.log -> /var/opt/dracut/log/dracut.log
-rw-r--r-- 1 root root 0 Feb 3 2021 installer-kickstart.log
-rw-r--r-- 1 root root 0 Feb 3 2021 installer.log
drwxr-sr-x 3 root systemd-journal 4096 Jun 16 08:20 journal
-rw-rw-r-- 1 root utmp 292292 Aug 12 12:23 lastlog
drwxr-x--- 2 root root 4096 Aug 11 18:10 loginsight-agent
-rw-r--r-- 1 root root 2519040 Aug 12 12:20 maillog
-rw-r----- 1 root root 1478656 Aug 12 12:26 messages
drwxr-xr-x 2 root root 4096 Jun 24 03:15 nginx
drwx------ 2 postgres users 12288 Aug 10 00:00 postgres
drwx------ 2 root root 4096 Feb 3 2021 private
drwx------ 2 root root 4096 Aug 9 04:01 rpmcheck
-rw-r--r-- 1 root root 19582976 Aug 12 12:20 sa
drwx------ 2 root root 4096 Aug 9 04:01 sgidlist
-rw------- 1 root root 11297 Jun 16 08:31 stigreport.log
drwx------ 2 root root 4096 Aug 9 04:02 suidlist
-rw------- 1 root root 64128 Aug 12 12:26 tallylog
-rw-r--r-- 1 root root 0 Feb 3 2021 tdnf.log
drwxr-xr-x 6 root root 4096 Jul 22 12:35 vmware
-rw------- 1 root root 719 Aug 10 16:00 vmware-network.1.log
-rw------- 1 root root 741 Aug 10 15:53 vmware-network.2.log
-rw------- 1 root root 741 Jul 20 08:20 vmware-network.3.log
-rw------- 1 root root 719 Jul 19 12:41 vmware-network.4.log
-rw------- 1 root root 741 Jul 19 12:29 vmware-network.5.log
-rw------- 1 root root 719 Jul 19 08:36 vmware-network.6.log
-rw------- 1 root root 741 Jul 19 08:20 vmware-network.7.log
-rw------- 1 root root 719 Jul 15 11:36 vmware-network.8.log
-rw------- 1 root root 719 Jun 11 15:09 vmware-network.9.log
-rw------- 1 root root 719 Aug 10 19:45 vmware-network.log
-rw------- 1 root root 31554 Aug 10 19:40 vmware-vgauthsvc.log.0
-rw------- 1 root root 45668 Aug 10 19:45 vmware-vmsvc-root.log
-rw------- 1 root root 3480 Aug 10 19:45 vmware-vmtoolsd-root.log
-rw-rw-r-- 1 root utmp 184320 Aug 12 12:23 wtmp
Cause : It occurs due to log rotation policy for auth.log and audit.log.
Fix :
Take the snapshots of the SDDC Manager.
Deleted the auth and audit logs both
root@vcfsddc[ /var/log ]# uptime
12:29:27 up 1 day, 16:49, 2 users, load average: 0.00, 0.02, 0.00
root@vcfsddc[ /var/log ]# rm auth.log
root@vcfsddc[ /var/log ]# cd audit
root@vcfsddc[ /var/log/audit ]# ls
audit.log
root@vcfsddc[ /var/log/audit ]# ls -l
total 12425896
-rw------- 1 root root 12724088832 Aug 12 12:31 audit.log
Reboot the sddc manager , then the space was back to normal and the GUI was able to login.
After few weeks the partition got again full , later vmware gss gave an workaround which is mention in the KB 86401
Comments