Overview

In the Monitoring Tool Orange Alerts are seen for the AAA Server. Upon checking the alarms it is noticed that the Alarms were related to Disk Space on the AAA Server. This article covers the initial steps to check and clear some old data from the Servers.

Solution

Login to the reporting node. After that as the "root" user, follow the below steps to clear the logs and free up disk space on the node. Note: You might have to switch to the "root" account using sudo su -

Step 1: Removing Server Logs

• The server logs are named in the format netvertex-_<Timestamp>.log.
• Here <Timestamp> is in yyyy-mm-dd format.
• To identify the specific directotories that are facing issues do the following:
  • Run du -sh from the root directory("/"). Example:
  • 
  • Then identify the top-level directory and then recurse to identify the specific directory that has the issues.
  • 
  • We generally recommend only maintaining the logs for 15-30 days.
  • Then one by one remove the files with older timestamps using the below command:

    • rm -rvf netvertex-_<Timestamp>.log

 

Step 2: Truncating the GUI Portal Logs:

• The GUI Portal Logs are in the format netvertex-std_<Timestamp>.log.
• Here <Timestamp> is in yyyy-mm-dd format.
• They are located in the directory /var/lib/kubelet/pods/<pod-detail>/volumes/kubernetes.io~empty-dir/data-logs/logs (in case of containerized deployments) or similar directory. Use du -sh to recurse and identify the specific path.
  • We generally recommend maintaining the GUI Portal logs for 2-3 months.

     

• Truncate the older logs using the command

  • > netvertex-std_*.log

An example Solution document is attached for your reference.

 

Verification

Ask the customer to validate the Disk Space using df 

After completing the above steps ask the customer to check the monitoring tool and verify that there are no more alarms reported and that the node status appears in Green.