24 August 2014

Linux System Activity Report SAR Graphs

One of the extremely useful tools to get an overview of what system is doing is a SAR reports,

We can see SAR stats by simply issuing the command:

# sar

Linux 3.0.74-0.6.8-default (hostname)      10/09/13        _x86_64_

00:00:01 CPU %user %nice %system %iowait %steal %idle
00:10:01 all  0.25  0.00  0.18    0.02     0.00 99.56

In most cases it is run as daemon in system and create the stats every 10 minutes,

We can also see current system statistics by issuing following command:

# sar 2 5

which create report every 2 seconds for 5 times,

Cron jobs can be find in sysstat file under cron.d directory:

# cat /etc/cron.d/sysstat
# Run system activity accounting tool every 10 minutes
*/10 * * * * root /usr/lib64/sa/sa1 1 1
# 0 * * * * root /usr/lib64/sa/sa1 600 6 &
# Generate a daily summary of process accounting at 23:53
53 23 * * * root /usr/lib64/sa/sa2 -A

The first cron sa1 gather systems stats every 10min by default to non human readable raw sa** files. (** represnt a day of the month in sa and sar files)

Then sa2 process sa** files once a day at 23:53 evening.

You can process the sa** raw data and generate reports under sar** manually by issuing the sa2 command:

/usr/lib64/sa/sa2 -A

You can create graphs from SAR reports to have a better view of system performance, to achieve this we can use a third party tool KSar which is avalible from it's site: http://sourceforge.net/projects/ksar/

In order to make graph we need sar files located in /var/log/sa directory:

# ls /var/log/sa

sa11  sa12  sa13  sa14  sa15  sa16  sa17  sa18  sar11  sar12  sar13  sar14  sar15  sar16  sar17  sar18

You can transfer sar files over to your system via WinSCP. Java Runtime JRE needs to be installed in your system as well.

Then open a command prompt and change directory to Ksar folder and issue the following command:

> cd C:\KSAR

> java -jar kSar.jar -input "C:\KSAR\sar10" -outputPDF "C:\Ksar_graphs\SAR_report.pdf"

for -input option you need to give the path of your sar log where you downloaded it, and -outputPDF is a path where graph will be created in PDF format.

And there you go. You have the SAR reports nicely graphed for you.

6 August 2014

Install SNMP and open its port on iptables firewall

In order to get your system monitored by a centralized monitoring system, most probably you need to enable snmp in your Linux,

Here is the way snmpd daemon can be installed and enabled:

1. Installation via yum or rpm:

# yum install net-snmp

# rpm -ivh net-snmp-libs-5.5-44.0.1.el6.x86_64.rpm net-snmp-5.5-44.0.1.el6.x86_64.rpm lnet-snmp-utils-5.5-44.0.1.el6.x86_64.rpm

2. SNMP configuration can be edited in snmpd.conf.

# cat /etc/snmp/snmpd.conf |grep -i public
# By default, the agent responds to the "public" community for read
# First, map the community name "public" into a "security name"
com2sec notConfigUser  default       public

3. Start the snmpd service:

# /etc/init.d/snmpd start
Starting snmpd:                                                         [OK]

# /etc/init.d/snmpd status
snmpd (pid  19508) is running...

# ps -ef|grep snmp
root     19508     1  0 Jul08 ?        00:00:18 /usr/sbin/snmpd -LS0-6d -Lf /dev/null -p /var/run/snmpd.pid

# netstat -an|grep 161
udp        0      0       *

4. Enable snmpd to start at boot:

# chkconfig snmpd on

# chkconfig --list snmpd
snmpd           0:off   1:off   2:on    3:on    4:on    5:on    6:off

5. Open the SNMP ports on the firewall: 

# iptables -I INPUT -p udp -m udp --dport 161 -j ACCEPT 

# iptables-save > /etc/sysconfig/iptables

6. Now you should be able to walk through your snmp MIBs via snmpwalk from a local or remote server:

# snmpwalk -mALL -v1 -c public <IP> system

RFC1213-MIB::sysDescr.0 = STRING: "Linux hostname 2.6.32-431.el6.x86_64 #1 SMP Sun Nov 10 22:19:54 EST 2013 x86_64"
RFC1213-MIB::sysObjectID.0 = OID: NET-SNMP-TC::linux
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (7152525) 19:52:05.25
RFC1213-MIB::sysContact.0 = STRING: "Root <root@localhost> (configure /etc/snmp/snmp.local.conf)"
RFC1213-MIB::sysName.0 = STRING: "hostname"
RFC1213-MIB::sysLocation.0 = STRING: "Unknown (edit /etc/snmp/snmpd.conf)"
SNMPv2-MIB::sysORLastChange.0 = Timeticks: (1) 0:00:00.01
SNMPv2-MIB::sysORID.1 = OID: SNMP-MPD-MIB::snmpMPDMIBObjects.3.1.1
SNMPv2-MIB::sysORID.3 = OID: SNMP-FRAMEWORK-MIB::snmpFrameworkMIBCompliance
SNMPv2-MIB::sysORID.4 = OID: SNMPv2-MIB::snmpMIB
SNMPv2-MIB::sysORID.6 = OID: RFC1213-MIB::ip
SNMPv2-MIB::sysORDescr.1 = STRING: The MIB for Message Processing and Dispatching.
SNMPv2-MIB::sysORDescr.2 = STRING: The MIB for Message Processing and Dispatching.
SNMPv2-MIB::sysORDescr.3 = STRING: The SNMP Management Architecture MIB.
SNMPv2-MIB::sysORDescr.4 = STRING: The MIB module for SNMPv2 entities
SNMPv2-MIB::sysORDescr.5 = STRING: The MIB module for managing TCP implementations
SNMPv2-MIB::sysORDescr.6 = STRING: The MIB module for managing IP and ICMP implementations
SNMPv2-MIB::sysORDescr.7 = STRING: The MIB module for managing UDP implementations
SNMPv2-MIB::sysORDescr.8 = STRING: View-based Access Control Model for SNMP.
SNMPv2-MIB::sysORUpTime.1 = Timeticks: (1) 0:00:00.01
SNMPv2-MIB::sysORUpTime.2 = Timeticks: (1) 0:00:00.01
SNMPv2-MIB::sysORUpTime.3 = Timeticks: (1) 0:00:00.01
SNMPv2-MIB::sysORUpTime.4 = Timeticks: (1) 0:00:00.01
SNMPv2-MIB::sysORUpTime.5 = Timeticks: (1) 0:00:00.01
SNMPv2-MIB::sysORUpTime.6 = Timeticks: (1) 0:00:00.01
SNMPv2-MIB::sysORUpTime.7 = Timeticks: (1) 0:00:00.01
SNMPv2-MIB::sysORUpTime.8 = Timeticks: (1) 0:00:00.01