Shell Scripting Tricks

Understand those secret shell script tricks with this quick and easy tutorial.


Introduction

I'm sure you've seen some strange stuff when looking at shell scripts. This tutorial is a just a quick overview of some those secret tricks that are hard to figure out what they are with just a Google search. I primarily use the BASH, ZSH and ASH shells, so these are things that I believe work with those; I won't specifiy whether these things work in other shells or whether they are POSIX.

Terminology

Here is some basic shell terminology you need to know.

What This Tutorial Covers

What This Tutorial Covers
  1. Tests
  2. Redirection & Piping
  3. Parameter & Other Expansions
  4. Subshells
  5. The Set Command
  6. Special Variables

What You Need For This Tutorial

What You Need For This Tutorial

A terminal if you want to play around with these tricks.


Tests

By tests, I'm just referring to how 'if' evaluates expressions. The way shell scripts evaluate expressions is particularly unique.

Output Redirection & Piping

File descriptors are just numbers that are shortcuts to file streams. You always start with three, stdin (0), stdout (1), stderr (2). You can use redirection to move these around.

Redirection is just moving output from one file stream to another file stream, while Pipe is the redirection of the output from one command to another command.

Shell Expansions & Subshells

Expansions are when some code is converted to something else when evaluated. Here I'll go over several types of expansions (except parameter expansion, which is described in the next section below).

Filename Expansion, otherwise known as Globbing, are some characters you can use to expand a filename into multiple files.

Tilde & Dash Expansion

Brace Expansion

Subshells, Command Substitution, & Parallel Processing.

Arithmetic Expressions

Parameter Expansion

There are many tricks for condensing logic when using parameters. Here are the most common ones I have seen.

  • Use the parameter's value
  • 
    variable=${parameter}
            
  • If parameter is null, use the word
  • 
    variable=${parameter:-'word'}
            
  • If parameter is not null, use the word
  • 
    variable=${parameter:+word}
            
  • If parameter is null, set parameter to the value of the word
  • 
    ${parameter:='word'}
            
  • If parameter is null, print the error message to stderr. If not in interactive mode, the script exits.
  • 
    ${parameter:?'error message'}
            
  • Use just a substring of the parameter. The parameter can be a variable name or the number of the positional parameter to use. Offset = where to start, Length = how many chars from the start.

    Offsets can be negative (offset from end of word) and lengths can be negative (number of chars going backwards)
  • 
    ${parameter:offset}
    ${parameter:offset:length}
            
  • If the pattern matches from the beginning of the parameter, remove the match and expand the rest. If no match, use the whole parameter.

    A single # means remove the shortest pattern match, and a double ## means remove the longest pattern match.
  • 
    ${parameter#pattern}
    ${parameter##pattern}
            
  • The same as the pattern matching with #, except match from the end of the parameter instead.
  • 
    ${parameter%pattern}
    ${parameter%%pattern}
            
  • The same as the pattern matching with #, except replace the match with the word.
  • 
    ${parameter/pattern/word}
            

Set

Set is a built in shell command. It's used to define the functionality around parameters. I'm going to go over a few of the more widely used commands.


# Set new positional parameters (they replace the old ones, they don't append)
set -- these are new positional parameters

# If a command in your script fails, exit the script
set -e

# You can always change the '-' to a '+' to turn the feature off
set +e

# If you try to expand a variable that does exist, exit the script
set -u

# If a command in a series of piped commands fail, 
# return its exit status instead of the exit status of the last command
set -o pipefail

# Print the commands that are run prepended by a '+' sign,
# followed by their output. Useful for debugging.
set -x

# Do not allow redirection to overwrite existing files.
set -C
      

There is of course, also the unset built in command. But it is much simpler. You use it just to unset variables


unset myVariable
      

Special Variables

When you start a script in shell, there are certain predefined special variables.


# The name of the script or command
$0

# The arguments passed to the script
$1 $2 $3

# All of the arguments passed to the script
$*

# The entire command line, or in other words, 
# the script name and all of its arguments
$@

# The number of arguments passed to the script
$#

# The options (flags) the shell has on to set how it behaves
$-

# Exit value of the last command
$?

# Process ID of the last command
$!
      

Done!

That's the weird looking stuff I know about in shell scripting.