Overview

In the Monitoring Tool Orange Alerts are seen for the dPCC/PCRF Server. Upon checking the alarms it is noticed that the Alarms were related to Disk Space on the dPCC/PCRF Server. This article covers the initial steps to check and clear some old data from the Servers.

Solution

• Check disk space utilization using the command df -kh
• Check the particular partition consuming maximum disk space using the command du-sh * 
  
• NOTE: The central Support team will only provide the steps to clear the logs. The execution should be done by the Customers 
• In most cases clearing the dPCC Logs should clear the disk space issues(namely Netvertex server logs & Netvertex Std Logs)

• rm -rvf netvertex-_*.log

  • Here * is the date in yyyy-mm-dd format to be removed from the log folder (Server logs).
    
  • The Server logs are usually located in a directory such as - /var/lib/kubelet/pods/fcaacce6-c6cf-4393-94e9-4e6a62520c81/volumes/kubernetes.io~empty-dir/data-logs/logs

• > netvertex-std_*.log

  • Here * is the date in yyyy-mm-dd format to be removed from the standard log folder (GUI Portal Logs).
  • In case of the customers with dockerized containers, standard logs are usually located in a directory such as - /var/lib/kubelet/pods/fcaacce6-c6cf-4393-94e9-4e6a62520c81/volumes/kubernetes.io~empty-dir/data-logs/logs
    • Here the pods and the container ID associated can be listed using the docker ps command
• Additional Note - In most deployments the Catalina.out(Tomcat System Logs) will be set to DEBUG level. So there is no need to change the Tomcat log levels.
• However in case the catalina.out file is growing bigger they can be truncated using

  • > catalina.out

Verification

Disk space optimized below 50%

Verify that the node health is moved back to Green: