HowTo Handle Mass Output - The awk Command

This is part of a tutorial series. The main thread begins here:
http://blog.sentiblue.net/2013/08/howto-handle-massive-standard-output.html

Hi guys,

Sorry for going silent past couple months! My daughter underwent surgery and she's fine now. So I'm back writing tutorials on my blog.

The awk command allows us to extract information by separation per column or row.

Let's say you have a file named "ipchange.info" with content like below

Hostname      Old IP          New IP
========      =============   ============
WDF23487      192.168.1.243   172.16.32.77
FEIW3847      192.168.1.223   172.16.14.32
DFW23489      192.168.2.44    172.32.12.35

You want to look into this list and display only the hostnames plus the New IP Addresses.

$ cat ipchange.info | awk '{ print $1, $3 }'
Hostname        New IP
========        ============
WDF23487        172.16.32.77
FEIW3847        172.16.14.32
DFW23489        172.32.12.35

The awk command by default uses white spaces as delimiter. Here in this example, $1 and $3 represent first and third columns.  The columns are separated by white spaces, so they got printed correctly.

Specifying Delimiter

In other cases where the delimiter is not a white space, for example the /etc/passwd file. The delimiter there is the colon:

root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
news:x:9:13:news:/etc/news:
uucp:x:10:14:uucp:/var/spool/uucp:/sbin/nologin

If you want to print the username and the userID only, they would be the 1st and 3rd column, but here the syntax needs to be a bit different.

$ cat /etc/passwd | awk -f : '{ print $1, $3 }'
root 0
bin 1
daemon 2
adm 3
lp 4
sync 5
shutdown 6
halt 7
mail 8
news 9
uucp 10

This syntax allows us to parse data files and extract wanted information regardless of what delimiter being used in the file.

Last Element

When you need to print the last element but don't know how many elements there are in the data, you can use $NF.

Using /etc/passwd again to see how $NF works

$ cat /etc/passwd | awk -f : '{ print $NF }'
/bin/bash
/sbin/nologin
/sbin/nologin 
/sbin/nologin 
/sbin/nologin
/bin/sync
/sbin/shutdown
/sbin/halt
/sbin/nologin
/sbin/nologin

More Options

awk has lots and lots of options that allow us to do much more with text strings. We stop here to use only the above basic options allowing us enough ability to extract data out of a large amount of output and present them in a much more human-readable format.

Back to the IP Change example text above; we would like to write a script that prepares a network address change for the 3 machines in the list, here's a skeleton of the script

Here's the configuration file
Hostname      Old IP          New IP
========      =============   ============
WDF23487      192.168.1.243   172.16.32.77
FEIW3847      192.168.1.223   172.16.14.32
DFW23489      192.168.2.44    172.32.12.35

We will drill into this list, get the hostname, old and new address, then download the current config file from the actual server, modify the address and upload it back to the server.

#!/bin/bash

HOSTS=`cat ipchage.info | awk '{ print $1 }' | grep -v Hostname | grep -v '===='`

for HOST in $HOSTS; do
   scp username@$HOST:/etc/sysconfig/network-scripts/ifcfg-eth0 ./backup/$HOST.ifcfg-eth0
   grep $HOST ipchange.info | read HOSTNAME OLD_IP NEW_IP
   cat ./backup/$HOST_ifcfg-eth0 | sed 's/$OLD_IP/$NEW_IP/g' > ./output/$HOST.ifcfg-eth0
   scp ./output/$HOST.ifcfg-eth0 username@$HOST:/etc/sysconfig/network-scripts/ifcfg-eth0
done


In the code example above, we use grep to extract rows related to certain hostnames that we need to act on and used awk to extract columns to get specific information related to a host that has already been processed by grep.

You noticed that there was also a sed command? That's next

Articles in the Series

Part 1: The grep Command
Part 2: The awk Command 
Part 3: The sed Command
Part 4: Miscellaneous Commands