Including passwords and keys in your script is bad for many reasons, security being number 1. All of these should be stored in another file, preferably a hidden file, that will never make it into your source control. That is just a universal no-no. Luckily, you can include another file's contents into your script.

Include Variables

Including a file with variables for passwords and keys is the way to go: [bash] #!/bin/bash source .env echo "Password in .env file is $PASSWORD" [/bash] Here, it just pulls in the entire file '.env'. For this file, just create some bash variables and those will be available after the source command in your script. Here's an example: [bash] PASSWORD="mySuperSecretPassWord111" [/bash] You do need to escape normal characters here, just like defining a variable in your script.

Include Functions

Your include can also have functions defined in it. [bash] # functions.sh include file # Grab the local ip address getLocalIp() { IP=`ifconfig | \ grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*' | \ grep -Eo '([0-9]*\.){3}[0-9]*' | \ grep -v '127.0.0' | \ head -n1` echo $IP } [/bash]

Shorthand way to include a file

There's a quicker way to include a file, by replacing source with a period. Here's the example above updated to use this method: [bash] #!/bin/bash . .env echo "Password in .env file is $PASSWORD" [/bash]

Bash has a lot of builtins, or internal functions, you can call. While most of them are very similar to utility programs available, the distinction is that no additional process is spawned. It may not sound like much, but a linux operating system has a limit to the number of processes that can run. If a server happens to run out of processes, but you still have a terminal open, then you still have a way to work with the system.

I/O builtins

In previous bash script posts, I've used the echo function. Here are some examples of that and more: [bash] # echo something to the screen echo "something to the screen" # read in from the keyboard, line-by-line while read LINE; do #Do some processing here echo "$LINE" done # read in file.txt, line-by-line while read LINE < file.txt; do #Do some processing here echo "$LINE" done # append to file.txt echo "Appending a line to the end" >> file.txt # echo a formatted line COUNT=1 NAME="Barry" printf "Hi %s! The count is %d" "$NAME" "$COUNT" [/bash]

Filesystem builtins

Bash also provides many of the fs builtins used every day. [bash] # Change directory cd /dir/to/enter # List dir contents ls -l # Print the current directory pwd [/bash] There are many more bash builtins than I've listed here, but these are the most commonly used.

There are a lot of utilities available to a bash script. Most will be installed by default, which can make scripting a breeze. However, if it's not installed it can mean some unexpected behavior. First and foremost, you'll need to check to see if the program is installed.

Request the user install a program

When a program is not available/installed, you want to notify the user and exit. Here's an example: [bash] which avconv if [ $? -ne 0 ] then echo 'avconv not found' echo 'This script requires avconv.' echo 'Please install it and try again' exit 99 fi [/bash] Here, we're searching for 'avconv'. You can swap 'avconv' with any utility. Make sure you do this for the utilities discussed here, as some may not be present.

scp - Copy of ssh

This is one of the easiest commands, because it uses the same format as regular 'cp'. [bash] #Copy file_to_copy.txt to the server's /path/for/file/ scp file_to_copy.txt user@example.com:/path/for/file/ #Copy file_to_copy.txt to the server's /path/to/file/ and name it remote_file_name.txt scp file_to_copy.txt user@example.com:/path/for/file/remote_file_name.txt #Copy file_to_copy.txt to the server's /path/to/file/, using port 22222 for the ssh connection scp -P 22222 file_to_copy.txt user@example.com:/path/for/file/ [/bash] As you can see, it's pretty straightforward. The one trick here is specifying the port, which always comes right after 'scp' on the command line.

ssh

You might not think ssh is useful in a script, but it does allow you to run a command remotely. [bash] #Run ls on the remote server ssh user@example.com ls #Run cp on the remote server ssh user@example.com "cp file1.txt file2.txt" [/bash] Basically, any command you want to run on the server is run and ssh exits immediately.

nslookup

nslookup checks DNS for a hostname. This is great if you want to know if a domain name exists or not, just look at the error code returned ($?). [bash] nslookup example.com if [ $? -ne 0 ] then echo 'Domain name not found, exiting' exit $? fi [/bash]

ping

Pinging a server can check to see if it's up. Some servers don't respond to pings, so it's not 100%. If you control the server and have ping open on purpose, this works great! [bash] ping example.com if [ $? -ne 0 ] then echo 'Server ping returned an error, exiting' exit $? fi [/bash]

curl and wget

curl and wget are very similar, so I'm going to cover them together. Basically they grab a webpage from a server. The main difference is where the page is placed by default. wget will save the page in a file and curl will output the page to stdout. [bash] #Grab the html into a variable HTML_CONTENTS=`curl http://example.com` #Save the html to a file wget http://example.com [/bash] My suggestion is to stick to curl if you want the contents in your script, because wget may append to the filename if there is a conflict.

rsync

Last, but not least, is rsync. This will sync the contents of a folder to another folder, possibly on a different computer. If files are the same, no transfer will take place, saving you on bandwidth. Here are some examples: [bash] # Copy everything under a folder locally rsync -av /folder/copying/from/ /folder/copying/to/ # Copy everything under a folder to a server rsync -av /folder/copying/from/ user@example.com:/folder/copying/to/ # Copy to a server using 22222 as the ssh port rsync -av -e "ssh -p 22222" /folder/copying/from/ user@example.com:/folder/copying/to/ [/bash]

Do Your Homework

These tools provide a massive amount of options you can pass in on the command line. Check out the man pages for each for more information.

When writing bash scripts, sometimes you'll want to do a search and replace on a file. If you already know regular expressions, this is super easy! If not, read on, because that's what's being covered in this post.

Regular Expressions

A search and replace expressions can be simple. The simplest is just changing out text directly. Here's an example:

s/One/Two/g

The forward slash '/' divides the expression into 4 parts. The first part is just the 's', which means we are going to search and replace. The second part is the match expression. In this case, we want to match the text 'One'. The third part is the replace expression. Here, 'Two' is the replacement string. The fourth part gives some extra flags. Here, it's just 'g', which means perform this globally and replace all instances of 'One'. Other flags include 'i' for case-insensitive searches and 'm' for a multi-line match.

Special Match Characters

In the match expression, there are some special characters that will match certain things.

Character	Meaning
Period '.'	Matches any character
Carot '^'	Matches the start of a line
Dollar Sign '$'	Matches the end of a line
Question Mark '?'	Means the preceding character may be present or absent
Star '*'	Means the preceding character may be absent, present, or repeat over and over
Plus Sign '+'	Means the preceding character is present and may repeat over and over

Here is an example:

s/^MyKey=.*$/MyKey=NewValue/g

In this example, we are looking for 'MyKey=' at the beginning of a line in the file. The '.*$' at the end of the match will match the rest of the line, so this will replace 'MyKey=ASDF' and 'MyKey=Value' with 'MyKey=NewValue'. If you want to search for that actual character, simply escape it by adding a backslash before it.

S/This is a sentence\./This sentence has no period now /g

Special Matches

There are some special escaped characters that we can use for matches:

Match String	What is matches
\s	Spaces and Tabs
\d	Numbers (does not include period, dollar sign, percent, negative sign, etc.)
\w	All alphabet characters, numbers and the underscore
\S	Everything except spaces and tabs
\D	Everything except numbers
\W	Everything except alphabet characters, numbers, and the underscore

These can be used in combination with each other and the special characters above. Here are some examples:

s/^\w+/Replace the first word/g

s/^\d+/Numbers!/g

s/^\s\s\s\s/  /g

Using Regular Expressions with sed

Now that we've covered the basics, here is how to use them with sed. [bash] sed -i -e 's/^MyLine.*$/YourLine/g' file_to_update.txt [/bash] This time, we're replacing all lines that start with 'MyLine' to 'YourLine'. The -i parameter tells sed to do an inline replace. Removing this causes the output to go to your terminal. The -e parameter tells sed that a regular expression is next. It's really that easy to use sed. The hard part is crafting the the replace expression.

In the previous post, the crontab was covered. Systemd gives another option, timers. This works basically the same way, it's just a job scheduler.

Creating a timer

Creating a timer is simply creating a file. This will live in "~/.local/share/systemd/user". Let's name it example.timer.

[Unit]
Description=ExampleTimer which runs every 10 minutes

[Timer]
OnCalendar=*:0/10
Unit=example.service

The first parts are pretty basic, the description and the program to run (ExecStart). The OnCalendar= part is where you setup the schedule. It has this format:

OnCalendar=[DayOfWeek] [Year-Month-Day] [Hour:Minute:Second]

• DayOfWeek - 3 character day of the week (i.e. Wed,Thu,Sun)
• Year-Month-Day - Standard calendar dates
• Hour:Minute:Second - Standard 24-hour time

Each value may contain a star to act as a wildcard and match any value. Each part is optional, but at least one piece is required.

Enabling the timer

You have to enable the timer via systemctl for it to be active: [bash] systemctl --user enable example-timer [/bash]

OnCalendar Examples

Here are some examples, with their crontab equivalent:

#Run every minute
#Cron: * * * * *
OnCalendar=*:*

#Run at the top of every hour
#Cron: 0 * * * *
OnCalendar=*:0

#Run at noon every day
#Cron: 0 12 * * *
OnCalendar=12:0

#Run at 4:25am, every Monday
#Cron: 25 4 * * 1
OnCalendar=Mon 4:25

#Run 2 hours before Christmas day
#Cron: 00 22 24 12 *
OnCalendar=*-12-24 22:00

#Runs Every Monday morning at midnight
#Cron: 0 0 * * 1
OnCalendar=Mon 00:00