The Unix Shell

Loops

Loops are key to productivity improvements through automation as they allow us to execute commands repetitively. Similar to wildcards and tab completion, using loops also reduces the amount of typing (and typing mistakes). Suppose we have several hundred Landsat-derived NDVI raster files named 005_HARV_ndvi_crop.tif, 037_HARV_ndvi_crop.tif, and so on. In this example, we’ll use the NEON-DS-Landsat-NDVI/HARV/2011/NDVI directory which only has thirteen example files, but the principles can be applied to many many more files at once. We would like to modify these files, but also save a version of the original files, naming the copies original-005_HARV_ndvi_crop.tif and original-037_HARV_ndvi_crop.tif. We can’t use:

$ cp *.tif original-*.tif

because *.tif would expand to:

005_HARV_ndvi_crop.tif  133_HARV_ndvi_crop.tif  213_HARV_ndvi_crop.tif  261_HARV_ndvi_crop.tif 
037_HARV_ndvi_crop.tif  181_HARV_ndvi_crop.tif  229_HARV_ndvi_crop.tif  277_HARV_ndvi_crop.tif
085_HARV_ndvi_crop.tif  197_HARV_ndvi_crop.tif  245_HARV_ndvi_crop.tif  293_HARV_ndvi_crop.tif
309_HARV_ndvi_crop.tif

And you cannot copy several files into original-*.tif

This wouldn’t back up our files, instead we get an error:

cp: target `original-*.tif' is not a directory

This a problem arises when cp receives more than two inputs. When this happens, it expects the last input to be a directory where it can copy all the files it was passed. Since there is no directory named original-*.tif in the NEON-DS-Landsat-NDVI/HARV/2011/NDVI directory we get an error.

Instead, we can use a loop to do some operation once for each thing in a list. Here’s a simple example that displays the first three lines of each file in our previous InSitu_Data directory:

$ cd ~/Documents/data/NEON-DS-Landsat-NDVI/HARV/2011/NDVI
$ for filename in  005_HARV_ndvi_crop.tif 037_HARV_ndvi_crop.tif 085_HARV_ndvi_crop.tif
> do
>    echo "File to process: $filename"
> done

File to process: 005_HARV_ndvi_crop.tif
File to process: 037_HARV_ndvi_crop.tif
File to process: 085_HARV_ndvi_crop.tif

When the shell sees the keyword for, it knows it is supposed to repeat a command (or group of commands) once for each thing in a list. In this case, the list is the three filenames. Each time through the loop, the name of the thing currently being operated on is assigned to the variable called filename. Inside the loop, we get the variable’s value by putting $ in front of it: $filename is 005_HARV_ndvi_crop.tif the first time through the loop, 037_HARV_ndvi_crop.tif the second, and so on. In this case, the command prints (echoes) the filename to process but we could call a R program and run it over each filename.

By using the dollar sign we are telling the shell interpreter to treat filename as a variable name and substitute its value on its place, but not as some text or external command. When using variables it is also possible to put the names into curly braces to clearly delimit the variable name: $filename is equivalent to ${filename}, but is different from ${file}name. You may find this notation in other people’s programs.

We have called the variable in this loop filename in order to make its purpose clearer to human readers. The shell itself doesn’t care what the variable is called; if we wrote this loop as:

for x in 005_HARV_ndvi_crop.tif 037_HARV_ndvi_crop.tif 085_HARV_ndvi_crop.tif
do
    echo "File to process: $x"
done

or:

for temperature in 005_HARV_ndvi_crop.tif 037_HARV_ndvi_crop.tif 085_HARV_ndvi_crop.tif
do
    echo "File to process: $temperature"
done

it would work exactly the same way. Don’t do this. Programs are only useful if people can understand them, so meaningless names (like x) or misleading names (like temperature) increase the odds that the program won’t do what its readers think it does.

Here’s a slightly more complicated loop:

for filename in *.tif

do 
    echo "File to process: $filename"    
    gdalinfo $filename | tail -5
done

The shell starts by expanding *.tif to create the list of files it will process. The loop body then executes three commands for each of those files. The first, echo, just prints its command-line parameters to standard output. For example:

$ echo hello there

prints:

hello there

In this case, since the shell expands $filename to be the name of a file, echo "File to process: $filename" just prints the name of the file after “File to process:”. Note that we can’t write this as:

for filename in *.tif
do
    $filename
    gdalinfo $filename | tail -5
done

because then the first time through the loop, when $filename expanded to 005_HARV_ndvi_crop.tif, the shell would try to run 005_HARV_ndvi_crop.tif as a program. Finally, the gdalinfo command returns information about the raster data and tail combination selects last 5 lines from whatever file is being processed.

Going back to our original file copying problem, we can solve it using this loop:

for filename in *.tif
do
    cp "$filename" original-"$filename"
done

This loop runs the cp command once for each filename. The first time, when $filename expands to 005_HARV_ndvi_crop.tif, the shell executes:

cp 005_HARV_ndvi_crop.tif original-005_HARV_ndvi_crop.tif

The second time, the command is:

cp 037_HARV_ndvi_crop.tif original-037_HARV_ndvi_crop.tif

rh.grib    temperature.grib   vo.grib

for datafile in *.grib
do
    ls *.grib
done

for datafile in *.grib
do
  ls $datafile
done

for file in *.grib
do
    echo $file
    cat $file > bigfile.grib
done

rh.grib    temperature.grib   vo.grib   z.txt

for datafile in *.grib
do
    cat $datafile >> file.grib
done

for file in *.grib
do
  analyze $file > analyzed-$file
done

# Version 1
for file in *.grib
do
  echo analyze $file > analyzed-$file
done

# Version 2
for file in *.grib
do
  echo "analyze $file > analyzed-$file"
done

$ expr 3 + 5
8
$ expr 30 / 5 - 2
4

for left in 2 3
do
    for right in $left
    do
        expr $left + $right
    done
done