HowTo Handle Mass Output - Conclusion

This is the conclusion article of the tutorial series...

In this article, I will show you guys how to combine the usage of these commands (grep, awk, sed, sort, uniq, seq) to run one single command line and generate tons of output, yet formatted into a very verbose but informative human readable format.

Assumptions

- We have a job server farm consisting of server names job7201 to job7955 (continuous consecutive)
- These job servers run identical processes in production and serve one common purpose
- There is an administrative server called "adm4125" from which we run all commands
- Passwordless SSH has been setup so that you can do passwordless SSH from adm4125 to ALL job servers

Objectives

# Current Objective
- Generate an inventory listing of all hardware, Serial Numbers, OS, CPU/RAM

# Future updates to this article will include
# - Perform remote health checks on the servers and their applications
# - Find out which server(s) participated in a certain job using a jobID

Preparations

Prepare the server names list 
[blogger@adm4125:~] $ seq 7201 7955
7201
7202
7203
....
7954
7955

Prepare Commands to Extract Data
We are using server job7201 as a test to extract sample data and observing from which, we can decide how to formulate our commands create the report

[blogger@adm4125:~] $ ssh job7201 "sudo /usr/sbin/dmidecode | grep -A6 'System Information'" # Get the hardware information
    System Information
        Manufacturer: HP
        Product Name: ProLiant DL385 G2
        Version: Not Specified
        Serial Number: ************     
        UUID: 34313431-3039-5553-4537-31364E324248
        Wake-up Type: Power Switch

[blogger@adm4125:~] $ ssh job7201 "uname -a" # Get the kernel version
Linux job7201.sentiblue.com 2.6.18-128.el5 #1 SMP Wed Dec 17 11:42:39 EST 2008 i686 i686 i386 GNU/Linux[blogger@adm4125:~] $ ssh job7201 "cat /etc/redhat-release" # Get the OS version
Red Hat Enterprise Linux Server release 5.3 (Tikanga)

[blogger@adm4125:~] $ MEM=`ssh job7201 "head -1 /proc/meminfo" | awk '{ print $2 }'` # Get the memory amount (in KB)
[blogger@adm4125:~] $ MEM=$(($MEM/1024/1024)) # Calculate out to GB
[blogger@adm4125:~] $ echo ${MEM}GB
125GB
[blogger@adm4125:~] $ ssh job7201 "grep -c Processor /proc/cpuinfo" # Count CPU Cores
48

Generate Hardware/OS/Resource Inventory Report

$ for SEQ in `seq 7201 7955 | xargs`; do
> SERVER="job$SEQ"
> echo -n "$SERVER: "
> ssh $SERVER "sudo /usr/sbin/dmidecode | grep -A6 'System Info' 
> | grep -e Manufacturer -e 'Product Name' -e Serial"
> done

job7201:         Manufacturer: HP
        Product Name: ProLiant DL385 G2
        Serial Number: SF349827     
job7202:         Manufacturer: HP
        Product Name: ProLiant DL360 G5
        Serial Number: F39ERIU2     
...
...
job7955:         Manufacturer: HP
        Product Name: ProLiant DL585 G7
        Serial Number: 2DF20D8R     

That doesn't look pretty! Let's fine tune the command some more;

- We want to get rid of the headers like "Manufacturer: ", "Product Name: ", "Serial Number: "
- We also want all data pertaining to each server to show in one single line

The colors of each fine tune will match below between objective and syntax.

$ for SEQ in `seq 7201 7955 | xargs`; do
> SERVER="job$SEQ"
> echo -n "$SERVER: "
> ssh $SERVER "sudo /usr/sbin/dmidecode | grep -A6 'System Info' 
> | grep -e Manufacturer -e 'Product Name' -e Serial"
> | sed 's/     Manufacturer: //g'
> | sed 's/     Product Name: //g'
> | sed 's/     Serial Number: //g' 
> | xargs
> done

job7201: HP ProLiant DL385 G2 SF349827
job7202: HP ProLiant DL360 G5 F39ERIU2
...
...
job7955: HP ProLiant DL585 G7 2DF20D8R

MUCH MUCH MUCH BETTER!

But so far we've only gathered Manufacturer, Model and Serial Numbers. Let's get some additional information; like the OS version and kernel architecture.

$ for SEQ in `seq 7201 7955 | xargs`; do
> SERVER="job$SEQ"
> echo -n "$SERVER: "
> ssh $SERVER "sudo /usr/sbin/dmidecode | grep -A6 'System Info'
> | grep -e Manufacturer -e 'Product Name' -e Serial"
> | sed 's/     Manufacturer: //g'
> | sed 's/     Product Name: //g'
> | sed 's/     Serial Number: //g' 
> | xargs
> ( cat /etc/redhat-release | awk '{ print \$NF }'; uname -i ) | xargs 
> done

job7201: HP ProLiant DL385 G2 SF349827 (Tikanga) x86_64
job7202: HP ProLiant DL360 G5 F39ERIU2 (Tikanga) x86_64
...
...
job7955: HP ProLiant DL585 G7 2DF20D8R (Tikanga) x86_64

There was more to the Original Objectives. For now I will end the article here leaving Objective #1 hanging a little loose.

I'd like those who read this article to fine tune the above command to include the CPU/RAM information into the report so that it looks like this

job7201: HP ProLiant DL385 G2 SF349827 (Tikanga) x86_64  24 Cores  196GB
job7202: HP ProLiant DL360 G5 F39ERIU2 (Tikanga) x86_64  48 Cores  64GB
...
...
job7955: HP ProLiant DL585 G7 2DF20D8R (Tikanga) x86_64  96 Cores  128GB

Don't let me down!!

This article will be revised in the future to meet Objectives #2 and #3
- Perform remote health checks on the servers and their applications
- Find out which server(s) participated in a certain job using a jobID