TDM 20100: Project 6 — Awk

To begin learning about awk, it is important to conceptualize the two fundamental units that awk operates on: records and fields.

You can think of a record as one row of data in a file.

A field can be thought of as a singular element of data (within a row of data).

Simple awk files that run on comma-separated data look this this:

but many awk programs do not have the beginning or ending section, i.e., they just look like this:

The action to run before processing the data only runs once, before awk ever touches the data. It is helpful, for example, if you want to print a header before running your output. (The BEGIN section is often omitted.)

The action to run after processing the data will print after the data is all processed. It is helpful if you want to print out some calculations that you ran on the data set. (The END section is often omitted.)

The main action of an awk program will run on each and every line of the data.

If the data is not comma separated, but (instead) is tab-separated, then we use -F\t instead of -F, (as an example). Or if the data has | between the pieces of data, then we use -F'|' instead. Or if the data has ; between the pieces of data, then we use -F';' instead.

The Sales values (given in Dollars) are available in field 22 of this data set:

head /anvil/projects/tdm/data/iowa_liquor_sales/iowa_liquor_sales_cleaner.txt | cut -d';' -f22

Use awk (without using cut!) to add the total Sales values from the entire file.

2a. Very similarly to question 1, use awk (without using cut!) to sum the total dollar amounts of the donations (altogether) given in column 15 of this file: /anvil/projects/tdm/data/election/itcont1980.txt

2b. Now sum the total dollar amounts of the donations (altogether) given in column 15 of all of the itcont*.txt files (altogether).

Consider the data in the file /anvil/projects/tdm/data/beer/reviews.csv

Notice that the number of columns on each line varies, because each line has a varied number of commas. Also note that the number of fields on each line is NF and therefore the last field on each line is $NF. Use this information to add all of the values in the score column, in a variable called totalscores. Also, at the same time, add the number of lines, in a variable called totallines. Finally, at the end, print the ratio of totalscores and totallines, so that we have the overall average score across the entire data set.

Consider the data in the file /anvil/projects/tdm/data/8451/The_Complete_Journey_2_Master/5000_transactions.csv

Use grep SOUTH and also awk (not cut) to sum the total amount of values in the SPEND column (corresponding to lines with SOUTH for the STORE_R value). Then do this again (in a separate bash pipeline) for the EAST stores, and then do it again (in a third bash pipeline) for the WEST stores, and finally in a fourth bash pipeline for the CENTRAL stores.

Consider the data in the file /anvil/projects/tdm/data/flights/subset/1990.csv

Use awk for formatted output, like this:

awk -F, '{print "flights from "$17" to "$18;}'

incorporated into a pipeline (with sort | uniq -c | sort -n | tail) from the previous projects, to find the 10 most popular flight paths in 1990 and the number of flights on those paths. Hint: The top two flight paths should be: