Bash Script 12: Includes

Including passwords and keys in your script is bad for many reasons, security being number 1. All of these should be stored in another file, preferably a hidden file, that will never make it into your source control. That is just a universal no-no. Luckily, you can include another file’s contents into your script.

Include Variables

Including a file with variables for passwords and keys is the way to go:

#!/bin/bash

source .env

echo "Password in .env file is $PASSWORD"

Here, it just pulls in the entire file ‘.env’. For this file, just create some bash variables and those will be available after the source command in your script. Here’s an example:

PASSWORD="mySuperSecretPassWord111"

You do need to escape normal characters here, just like defining a variable in your script.

Include Functions

Your include can also have functions defined in it.

# functions.sh include file

# Grab the local ip address
getLocalIp() {
  IP=`ifconfig | \
      grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*' | \
      grep -Eo '([0-9]*\.){3}[0-9]*' | \
      grep -v '127.0.0' | \
      head -n1`
  echo $IP
}

Shorthand way to include a file

There’s a quicker way to include a file, by replacing source with a period. Here’s the example above updated to use this method:

#!/bin/bash

. .env

echo "Password in .env file is $PASSWORD"

Bash Script 11: Internal Bash Builtins

Bash has a lot of builtins, or internal functions, you can call. While most of them are very similar to utility programs available, the distinction is that no additional process is spawned. It may not sound like much, but a linux operating system has a limit to the number of processes that can run. If a server happens to run out of processes, but you still have a terminal open, then you still have a way to work with the system.

I/O builtins

In previous bash script posts, I’ve used the echo function. Here are some examples of that and more:

# echo something to the screen
echo "something to the screen"

# read in from the keyboard, line-by-line
while read LINE;
do
  #Do some processing here
  echo "$LINE"
done

# read in file.txt, line-by-line
while read LINE < file.txt;
do
  #Do some processing here
  echo "$LINE"
done

# append to file.txt
echo "Appending a line to the end" >> file.txt

# echo a formatted line
COUNT=1
NAME="Barry"
printf "Hi %s! The count is %d" "$NAME" "$COUNT"

Filesystem builtins

Bash also provides many of the fs builtins used every day.

# Change directory
cd /dir/to/enter

# List dir contents
ls -l

# Print the current directory
pwd

There are many more bash builtins than I’ve listed here, but these are the most commonly used.

Bash Script 10: Common Network Utilities

There are a lot of utilities available to a bash script. Most will be installed by default, which can make scripting a breeze. However, if it’s not installed it can mean some unexpected behavior. First and foremost, you’ll need to check to see if the program is installed.

Request the user install a program

When a program is not available/installed, you want to notify the user and exit. Here’s an example:

which avconv
if [ $? -ne 0 ]
then
  echo 'avconv not found'
  echo 'This script requires avconv.'
  echo 'Please install it and try again'
  exit 99
fi

Here, we’re searching for ‘avconv’. You can swap ‘avconv’ with any utility. Make sure you do this for the utilities discussed here, as some may not be present.

scp – Copy of ssh

This is one of the easiest commands, because it uses the same format as regular ‘cp’.

#Copy file_to_copy.txt to the server's /path/for/file/
scp file_to_copy.txt user@example.com:/path/for/file/

#Copy file_to_copy.txt to the server's /path/to/file/ and name it remote_file_name.txt
scp file_to_copy.txt user@example.com:/path/for/file/remote_file_name.txt

#Copy file_to_copy.txt to the server's /path/to/file/, using port 22222 for the ssh connection
scp -P 22222 file_to_copy.txt user@example.com:/path/for/file/

As you can see, it’s pretty straightforward. The one trick here is specifying the port, which always comes right after ‘scp’ on the command line.

ssh

You might not think ssh is useful in a script, but it does allow you to run a command remotely.

#Run ls on the remote server
ssh user@example.com ls

#Run cp on the remote server
ssh user@example.com "cp file1.txt file2.txt"

Basically, any command you want to run on the server is run and ssh exits immediately.

nslookup

nslookup checks DNS for a hostname. This is great if you want to know if a domain name exists or not, just look at the error code returned ($?).

nslookup example.com
if [ $? -ne 0 ]
then
  echo 'Domain name not found, exiting'
  exit $?
fi

ping

Pinging a server can check to see if it’s up. Some servers don’t respond to pings, so it’s not 100%. If you control the server and have ping open on purpose, this works great!

ping example.com
if [ $? -ne 0 ]
then
  echo 'Server ping returned an error, exiting'
  exit $?
fi

curl and wget

curl and wget are very similar, so I’m going to cover them together. Basically they grab a webpage from a server. The main difference is where the page is placed by default. wget will save the page in a file and curl will output the page to stdout.

#Grab the html into a variable
HTML_CONTENTS=`curl http://example.com`

#Save the html to a file
wget http://example.com

My suggestion is to stick to curl if you want the contents in your script, because wget may append to the filename if there is a conflict.

rsync

Last, but not least, is rsync. This will sync the contents of a folder to another folder, possibly on a different computer. If files are the same, no transfer will take place, saving you on bandwidth. Here are some examples:

# Copy everything under a folder locally
rsync -av /folder/copying/from/ /folder/copying/to/

# Copy everything under a folder to a server
rsync -av /folder/copying/from/ user@example.com:/folder/copying/to/

# Copy to a server using 22222 as the ssh port
rsync -av -e "ssh -p 22222" /folder/copying/from/ user@example.com:/folder/copying/to/

Do Your Homework

These tools provide a massive amount of options you can pass in on the command line. Check out the man pages for each for more information.

Bash Script 9: Altering Files with sed

When writing bash scripts, sometimes you’ll want to do a search and replace on a file. If you already know regular expressions, this is super easy! If not, read on, because that’s what’s being covered in this post.

Regular Expressions

A search and replace expressions can be simple. The simplest is just changing out text directly. Here’s an example:

s/One/Two/g

The forward slash ‘/’ divides the expression into 4 parts.

The first part is just the ‘s’, which means we are going to search and replace.

The second part is the match expression. In this case, we want to match the text ‘One’.

The third part is the replace expression. Here, ‘Two’ is the replacement string.

The fourth part gives some extra flags. Here, it’s just ‘g’, which means perform this globally and replace all instances of ‘One’. Other flags include ‘i’ for case-insensitive searches and ‘m’ for a multi-line match.

Special Match Characters

In the match expression, there are some special characters that will match certain things.

Character Meaning
Period ‘.’ Matches any character
Carot ‘^’ Matches the start of a line
Dollar Sign ‘$’ Matches the end of a line
Question Mark ‘?’ Means the preceding character may be present or absent
Star ‘*’ Means the preceding character may be absent, present, or repeat over and over
Plus Sign ‘+’ Means the preceding character is present and may repeat over and over

Here is an example:

s/^MyKey=.*$/MyKey=NewValue/g

In this example, we are looking for ‘MyKey=’ at the beginning of a line in the file. The ‘.*$’ at the end of the match will match the rest of the line, so this will replace ‘MyKey=ASDF’ and ‘MyKey=Value’ with ‘MyKey=NewValue’.

If you want to search for that actual character, simply escape it by adding a backslash before it.

S/This is a sentence\./This sentence has no period now /g

Special Matches

There are some special escaped characters that we can use for matches:

Match String What is matches
\s Spaces and Tabs
\d Numbers (does not include period, dollar sign, percent, negative sign, etc.)
\w All alphabet characters, numbers and the underscore
\S Everything except spaces and tabs
\D Everything except numbers
\W Everything except alphabet characters, numbers, and the underscore

These can be used in combination with each other and the special characters above. Here are some examples:

s/^\w+/Replace the first word/g

s/^\d+/Numbers!/g

s/^\s\s\s\s/  /g

Using Regular Expressions with sed

Now that we’ve covered the basics, here is how to use them with sed.

sed -i -e 's/^MyLine.*$/YourLine/g' file_to_update.txt

This time, we’re replacing all lines that start with ‘MyLine’ to ‘YourLine’.

The -i parameter tells sed to do an inline replace. Removing this causes the output to go to your terminal.

The -e parameter tells sed that a regular expression is next.

It’s really that easy to use sed. The hard part is crafting the the replace expression.

Bash Script 8: Systemd Timer

In the previous post, the crontab was covered. Systemd gives another option, timers. This works basically the same way, it’s just a job scheduler.

Creating a timer

Creating a timer is simply creating a file. This will live in “~/.local/share/systemd/user”. Let’s name it example.timer.

[Unit]
Description=ExampleTimer which runs every 10 minutes

[Timer]
OnCalendar=*:0/10
Unit=example.service

The first parts are pretty basic, the description and the program to run (ExecStart). The OnCalendar= part is where you setup the schedule. It has this format:

OnCalendar=[DayOfWeek] [Year-Month-Day] [Hour:Minute:Second]
  • DayOfWeek – 3 character day of the week (i.e. Wed,Thu,Sun)
  • Year-Month-Day – Standard calendar dates
  • Hour:Minute:Second – Standard 24-hour time

Each value may contain a star to act as a wildcard and match any value. Each part is optional, but at least one piece is required.

Enabling the timer

You have to enable the timer via systemctl for it to be active:

systemctl --user enable example-timer

OnCalendar Examples

Here are some examples, with their crontab equivalent:

#Run every minute
#Cron: * * * * *
OnCalendar=*:*

#Run at the top of every hour
#Cron: 0 * * * *
OnCalendar=*:0

#Run at noon every day
#Cron: 0 12 * * *
OnCalendar=12:0

#Run at 4:25am, every Monday
#Cron: 25 4 * * 1
OnCalendar=Mon 4:25

#Run 2 hours before Christmas day
#Cron: 00 22 24 12 *
OnCalendar=*-12-24 22:00

#Runs Every Monday morning at midnight
#Cron: 0 0 * * 1
OnCalendar=Mon 00:00

Bash Script 7: Crontab

Want your script to run at regular intervals? Maybe irregular intervals is what you want? There’s a unix/linux/mac utility you can use called cron. Setup is easy, just type in “crontab -e” to edit your cron jobs.

Crontab fields

In the crontab, each job will have it’s own line. Each line starts with 5 special values, separated by at least 1 space, that will determine when the program is run. These values, in order, are:

  1. Minutes
  2. Hours
  3. Day of the month
  4. Month of the year
  5. Day of the week (0=Sunday)

Each of these can have different values:

Value Description
1 Runs only when the value is 1
30 Runs only when the value is 30
1,2,5 Runs only when the value is either 1, 2, or 5
1-5,8-10 Runs only when the value is either 1, 2, 3, 4, 5, 8, 9, or 10
*/5 Runs only when the value is a multiple of 5
* Runs at any value

The most used value is the star “*”. This is because most of the time, you will only want 1 or 2 values to be real numbers. Here are some examples:

* * * * * echo 'I run once a minute, every minute, every hour'
0 * * * * echo 'I run at the top of the hour, every hour'
0 12 * * * echo 'I run at 12:00 noon, every day'
25 4 * * 1 echo 'I run at 4:25am, every Monday'
00 22 24 12 * echo 'There is only 2 hours left until Christmas'

As you can see, the combinations are endless. You can call any command you’d like after the 5 crontab fields. I am just calling the echo command, which is pretty pointless because the cron is run in the background, so the output is never rendered to the screen.

Email cron output

If you do want the output to go somewhere, add MAILTO=”your_email@example.com” to the top of your crontab. If email is all setup on your system, it will email the output of all cron jobs to the email provided at the top of the file.

MAILTO="your_email@example.com"

Bash Script 6: Loops

Looping in bash is very easy. In the previous posts, there has been a while loop, which was used to process command line arguments. There are 2 other types of loops: for and until. This post will cover all 3 types.

While loops

The while loop will continue to loop until some condition becomes false, or zero. This condition is passed in as a test. Check out the previous post for information on bash tests.

Here’s a simple example:

COUNTER=0
while [ $COUNTER -lt 10 ]
do
  echo $COUNTER
  COUNTER=$((COUNTER + 1))
done

In this example, it loops 10 times, each time echoing the COUNTER then incrementing it. The test could be anything, even checking for the existence of files to process.

Until loops

The until loop is very similar to the while loop, but the condition check is reversed. The loop will continue as long as the condition is false. Here’s the same example as before, but using an until loop:

COUNTER=0
until [ $COUNTER -gt 9 ]
do
  echo $COUNTER
  COUNTER=$((COUNTER + 1))
done

For loops

The for loop iterates over a set of data. I’ve used this for file processing before, because it makes it super easy to do. Here’s an example that will loop through mkv video files in the current folder:

for FILE in ls *.mkv
do
  echo "$FILE";
  #Do Something with $FILE here, perhaps call conv.sh
done

Bash Script 5: Ifs/Tests

Almost every program contains some sort of logical test. This could be just checking to see if it’s a certain time, or if the user enter Y or N. Bash scripting offers a wide variety of tests. Let’s go over some of what’s possible here.

Basic If Structure

If we’re going to be doing a test, we want the program to respond one way if it’s true and another way if it’s false. This is accomplished with the if/then/else/fi bash control structure. Here’s a simple example:

if [ "1" -eq "1" ]
then
  echo "1 equals 1";
else
  echo "1 does not equal 1";
fi

Here we are checking to see if 1 equals 1. Since it does, the code between then and else will be executed. If the test were to be false, then the code between else and fi would be executed.

Number Tests

Let’s say you want to run some checks on 2 numbers. Here are the tests you can run:

a=1
b=2

# Test for equality
if [ "$a" -eq "$b" ]
then
  echo 'They are equal';
fi

# Test for a note qual to b
if [ "$a" -ne "$b" ]
then
  echo 'a is not equal to b';
fi

# Test for a greater than b
if [ "$a" -gt "$b" ]
then
  echo 'a is greater than b';
fi

# Test for a lesser than b
if [ "$a" -lt "$b" ]
then
  echo 'a is lesser than b';
fi

String Tests

Comparing 2 strings is very useful too.

a="asdf"
b="fdsa"

# Test for a equal to b
if [ "$a" = "$b" ]
then
  echo 'a and b are the same'
fi

# Test for a not equal to b
if [ "$a" != "$b" ]
then
  echo 'a and b are not the same'
fi

# Test if the contents of a comes before b, alphabetically
if [[ "$a" < "$b" ]]
then
  echo 'The contents of a come before b'
fi

# Test if a is empty
if [ -z "$a" ]
then
  echo 'a is empty'
fi

# Test if a is not empty
if [ -n "$a" ]
then
  echo 'a is not empty'
fi

Other Tests

There are other tests you can perform as well:

# Check for a file's existence
if [ -f my_file.mkv ]
then
  echo 'Found my_file.mkv';
else
  echo 'Cannot find my_file.mkv';
fi

# Check for non-zero filesize
if [ -s my_file.mkv ]
then
  echo 'File has contents'
fi

# Check if a filename is really a directory
if [ -d my_file.mkv ]
then
  echo 'my_file.mkv is really a directory'
fi

Bash Script 4: Man pages

For the last 3 posts, the conv.sh script has evolved to a stable place. Let’s look at creating a simple manual, or man page. This really is quite easy, once you’ve got the basic formatting down.

Creating a basic man page

Here is the basic page:

.\" Manpage for conv
.\" All of these lines here are comments.
.\" They will not be shown to the end user.
.\"
.\" Remember that certain characters need escaping, like the dash.
.TH man 1 "18 Oct 2018" "1.0" "conv man page"
.SH NAME
conv.sh \- convert a video file to x264/mp3/mkv format
.SH SYNOPSIS
conv.sh [-hv] [filename]
.SH DESCRIPTION
conv is a high level shell program for converting videos
.SH OPTIONS
.PP
\-h
.RS 4
This shows the help information
.RE
.PP
\-v
.RS 4
This increases verbosity
.RE
.SH SEE ALSO
avconv(1)
.SH BUGS
No known bugs.
.SH AUTHOR
Barry Gilbert (compguyaug.com)

I’m going to name this file “conv.1”.

Let’s go through the lines and their meaning:

Lines 1-5 are just comments. These can be scattered throughout the document and will not appear to the end user.
Line 6 is the header (TH). There should only be one of these in the file, and it should match this format. Just updated the date, version, and program name.
Lines 7, 9, 11, 13, 24, 26, and 28 are section headers (SH). There are several of these in the file and these are the standard ones, but others can be added as well.
Line 8, 10, 12, 25, 27, and 29 are the content to show under the respective sections.
Lines 14-23 are different, because they are the options sections, and I want things formatted in a specific way. Keep reading to learn more about the options

Options Formatting

For this, I wanted to follow the standard formatting for options, to keep the man pages looking the same. Basically, for each option there are 5 lines, 3 of which control how the content is rendered.

.PP – Starts a single option section. This should be immediately followed by the option tag itself.
.RS 4 – Starts the option description.
.RE – Ends the option description.

Manpage Levels

Before we install that manpage, let’s talk about levels. This man page has a level of 1, or User commands. There are others as well:

Level Description
1 User commands (Programs)
2 System calls
3 Library calls
4 Special files (devices)
5 File formats and configuration files
6 Games
7 Overview, conventions, and miscellaneous
8 System management commands

Installing the man page

In Ubuntu, the man pages live in the folder /usr/share/man/man1. Your system may be slightly different, but the utility manpath will show you the search path for man pages on your system. To install the man page, just run these 2 lines:

sudo install -g 0 -o 0 -m 0644 conv.1 /usr/share/man/man1
sudo gzip /usr/share/man/man1/conv.1

Bash Script 3: Error Checking

The last few posts, I’ve been talking about bash scripts. Today, I’m going to talk about error checking/handling. So far, the script has been fairly simple, but it does make some assumptions, such as having avconv installed, the video file being valid, and the output file not existing. Well, for that last one, the output file would be overwritten if it does exist, so we’d loose data and that’s always a bad thing. Without further ado, let’s get started.

Checking for an installed program

This is super simple. Just call the ‘which’ command. In our script, we make sure of avconv, so the test is this:

which avconv
if [ $? -ne 0 ]
then
  echo 'avconv not found'
  echo 'This script requires avconv.'
  echo 'Please install it and try again'
  exit 99
fi

Here, I’m calling which and checking the exit code, stored in $?. If it’s not equal to 0, the program was not found and the script informs the user and exits.

Checking for an existing output file

For this, it’s the same test covered in the last post, -f. To make things easier, let’s assign the output filename to a variable and then check it.

OUTPUT_FILE="$1.mkv"
if [ -f "$OUTPUT_FILE" ]
then
  echo 'Warning: The output file already exists: $OUTPUT_FILE'
  echo 'Please rename that file'
  exit 98
fi

Checking for a valid video file

This one is a bit tricky. Instead of verifying that the file is correct, let’s just check for the output of avconv, which will return a 0 if successful.

avconv -i "$1" $VERBOSITY_OPTION -c:v libx264 \
  -preset medium -crf 20 -c:a mp3 "$OUTPUT_FILE";
if [ $? -ne 0 ]
then
  echo 'There was a problem converting the video.'
  echo 'Your file may be corrupt.'
  echo 'Please verify the video file.'
  exit 97;
fi

Putting it all together

Here’s the full updated script:

#!/bin/bash

###############################
#                             #
# convert-video.sh            #
#                             #
# Written By: Barry Gilbert   #
# Written On: Oct 2018        #
#                             #
###############################

# Shows the end user the usage message
showUsage() {
  echo "" >&2
  echo "Usage: convert-video.sh [-h] [-v] [video-filename]" >&2
  echo "  -h  shows this help information" >&2
  echo "  -v  verbosity for avconv." >&2
  echo "      Multiple -v options increase the verbosity." >&2
  echo "      The maximum is 2." >&2
  echo "  video-filename video file to convert" >&2
  exit 99;
}

VERBOSITY_LEVEL=0

# Loop through the options passed in by the user
while getopts ":h" opt; do
  case $opt in
    h)
      showUsage
      ;;
    v)
      VERBOSITY_LEVEL=$((VERBOSITY_LEVEL+1))
      ;;
    \?)
      # Bad option, YELL AT USER
      echo 'Invalid option: -$OPTARG"
      showUsage
      ;;
  esac
done

# Reset the parameter, so we can grab the filename
shift $((OPTIND-1))

VERBOSITY_OPTION=""
if [ VERBOSITY_LEVEL -eq 1 ]
then
  VERBOSITY_OPTION=" -loglevel=verbose"
elif [ VERBOSITY_LEVEL -eq 2 ]
then
  VERBOSITY_OPTION=" -loglevel=debug"
fi

# Check to see if a filename was provided
# $# will contain the number of arguments
# If the number of arguments is not 1
if [ $# -ne 1 ]
then
  echo 'No File Provided';
  showUsage;
fi

# $1 will contain the first argument
# Check to see if $1 is not a file
if [ ! -f "$1" ]
then
  echo 'File Not Found';
  showUsage;
fi

# Check to see if avconv is installed
which avconv
if [ $? -ne 0 ]
then
  echo 'avconv not found'
  echo 'This script requires avconv.'
  echo 'Please install it and try again'
  exit 99
fi

# Check to make sure output file doesn't exist
OUTPUT_FILE="$1.mkv"
if [ -f "$OUTPUT_FILE" ]
then
  echo 'Warning: The output file already exists: $OUTPUT_FILE'
  echo 'Please rename that file'
  exit 98
fi

# Convert the video
avconv -i "$1" $VERBOSITY_OPTION -c:v libx264 \
  -preset medium -crf 20 -c:a mp3 "$OUTPUT_FILE";
if [ $? -ne 0 ]
then
  echo 'There was a problem converting the video.'
  echo 'Your file may be corrupt.'
  echo 'Please verify the video file.'
  exit 97;
fi