From: Subject: Unix shell scripting with sh/ksh Date: Tue, 15 Apr 2003 11:18:34 +0200 MIME-Version: 1.0 Content-Type: text/html; charset="windows-1250" Content-Transfer-Encoding: quoted-printable Content-Location: http://www.dartmouth.edu/~rc/classes/ksh/print_pages.shtml X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 Unix shell scripting with sh/ksh

Unix shell scripting with sh/ksh

Course Handout: (last Update 28 February 2003)=20



These notes may be found at http://www.dartmouth.ed= u/~rc/classes/ksh.=20 The online version has many links to additional information and may be = more up=20 to date than the printed notes=20

UNIX shell scripting with sh/ksh

The goals of this class are to enable you to:

Assumptions

It is assumed that you already know how to:=20 Example = commands are=20 shown like = this
Output from=20 commands is shown like this
Optional items are shown in = brackets,=20 [ like this ]=20

Some descriptions in these notes have more detail available, and are = denoted=20 like this:

More details of this item would appear here. The printed = notes=20 include all of the additional information

These notes are updated from time to time. The "development" set of = notes are=20 here. (Dartmouth only)=20

Richard Brittain, Dartmouth College Computing Services.
=A9 2002 = Dartmouth=20 College.
Comments and questions, contact Richard.Brittain @ = dartmouth.edu

Text-only= Printable
Frames No Frames
Introduction

Research = Computing=20 Home

1. What is a shell script
2. Why use shell scripts
3. History
4. Feature comparison
5. Other scripting languages
6. ksh vs sh
7. Basics
8. Style
9. Common external commands (1)
10. Common external commands (2)
11. More external commands
12. Variables
13. Preset Variables
14. Arguments
15. Manipulating Variables
16. Command line options
17. Command substitution
18. Conditional Tests
19. Flow control
20. Flow control (contd.)
21. Basic I/O
22. Functions
23. Advanced I/O
24. Coprocesses
25. Signals
26. Security
27. Miscellaneous
28. Examples
29. References
Printable version
(single file)=20

(1)

What is a Shell Script

Example

A shell script can be as simple as a sequence of = commands that=20 you type regularly. By putting them into a script, you reduce them to a = single=20 command. 
	#!/bin/ksh
	date
	cd
	pwd
	du -k

(2)

Why use Shell Scripts

Shell scripts are extensively used by:=20

AUTOMATE, AUTOMATE, AUTOMATE

(3)

History of Shells

In the = beginning.....=20
sh=20
aka "Bourne" shell, written by Steve Bourne at AT&T Bell Labs = for the=20 very earliest versions of Unix. Very small, very simple, and very = few=20 internal commands, so it called external programs for even the = simplest=20 of tasks. The big advantage is that it is always available on = everything that=20 looks vaguely like Unix.=20
csh=20
The "C" shell. Written by Bill Joy at Berkeley (who went on to = found Sun=20 Microsystems). Many things in common with the Bourne shell, but many=20 enhancements to make interactive use much nicer. The syntax of the = internal=20 commands used only in scripts is very different from "sh", and is = similar to=20 the "C" programming syntax.=20
tcsh=20
The "TC" shell. Freely available, written mostly by various folks = at=20 Cornell University, and based on "csh". It has many additional = features to=20 make interactive use more convenient. The disadvantage is that until = recently,=20 it was not installed by default on many versions of Unix. Not many = people=20 write scripts in [t]csh. We use it as the default shell for new = accounts on=20 all of our public systems.=20
ksh=20
The "Korn" shell, written by David Korn of AT&T Bell Labs (now = Lucent). Written as a major upgrade to "sh", it is compatible with it, = but has=20 many more internal commands for the most frequently used functions. It = also=20 incorporates most of the same features from tcsh which enhance = interactive use=20 (command line history recall etc.). This shell is now available on = most=20 systems. It was slow to gain acceptance because earlier versions were=20 encumbered by AT&T licensing.=20
POSIX/XPG4=20
Standards comittees worked over the Korn shell and slightly = modified some=20 specifications to define a "standard" shell which systems meeting the = POSIX=20 spec. must comply with. On some systems, /bin/sh is actually now a = POSIX=20 compliant shell, fully compatible with Bourne shell, but with most of = the ksh=20 features. This standard was created partly because the early ksh was = not fully=20 100% backward compatible with sh, and so caused trouble if it simply = replaced=20 it. On Solaris, the "alternate" commands which differ slightly in = behaviour=20 from traditional SunOS commands are located in /usr/xpg4/bin=20
bash=20
The "Bourne again" shell. Written as part of the GNU/Linux Open = Source=20 effort, it is basically a functional clone of sh, with additional = features to=20 enhance interactive use, add POSIX compliance, and partial ksh = compatability.=20 It is the default shell on Linux.=20
zsh=20
A freeware functional clone of sh, with parts of ksh, bash and = POSIX=20 compliance, and some new interactive command-line editing features. It = is=20 installed as the default shell on early MacOSX systems (later ones = have bash)=20

(4)

Comparison of shell features

All the = shells just=20 listed share common features, and the major differences in syntax = generally only=20 affect script writers. It is very common to use one shell (e.g. tcsh) = for=20 interactive use, but another (sh or ksh) for writing scripts.=20

Core Similarities

Principal Differences

between sh + derivitives, and csh + = derivitives.=20

(5)

Other Scripting Languages

There are = many other=20 programs which read a file of commands and carry out a sequence of = actions. The=20 "#!/path/to/program" = convention=20 allows any of them to be used as a scripting language to create new = commands.=20 Some are highly specialized, and some are much more efficient than the=20 equivalent shell scripts at certain tasks. There is never only one way = to=20 perform a function, and often the choice comes down to factors like:=20 Some major = players (all=20 of these are freely available) in the general purpose scripting = languages are:=20
perl=20
The most used scripting language for Web CGI applications and = system=20 administration tasks. Perl is harder to learn, and is not installed by = default=20 on all systems (but most modern systems have it now). It is more = efficient and=20 has an enormous library of functions available. You could use Perl for = almost=20 all scripting tasks, but the syntax is very different to the shell = command=20 line=20
awk=20
A pattern matching and data (text and numeric) manipulation tool. = Predates=20 perl. Installed on all Unix systems. Often used in combination with = shell=20 scripts.=20
python=20
An object-oriented scripting language. Not widely installed.=20
tcl/tk=20
Tool Command Language. Another general purpose scripting language. = The=20 "tk" component is a scripted interface to standard X-windows graphical = components, so the combination is often used to create graphical user=20 interfaces.

(6)

ksh vs sh

Ksh is essentially a = superset of sh.=20 For maximum portability, even to very old computers, you should stick to = the=20 commands found in sh. Where possible, ksh-specific features will be = noted in the=20 following pages. In general, ksh runs a little faster (sometimes a lot = faster)=20 and is often more readable because logic can be expressed more cleanly = user the=20 newer ksh-specific syntax.=20

The philosophy of separate Unix tools each performing a single = operation was=20 followed closely by the designers of the original shell, so it had very = few=20 internal commands and used external tools for very trivial operations = (like=20 echo). Ksh internally = performs=20 many of the basic string and numeric manipulations and conditional = tests.=20 Occasional problems arise because the internal versions of some commands = like=20 echo are not fully = compatible=20 with the external utility they replaced.

The action taken every time a shell needs to run an external = program is to locate the program (via $PATH), fork(), which creates a = second=20 copy of the shell, adjust the standard input/output for the external = program,=20 and exec(), which replaces the second shell with the external program. = This=20 process is computationally expensive (relatively), so when the script = does=20 something trivial many times over in a loop, it saves a lot of time if = the=20 function is handled internally.

If you follow textbooks on Bourne shell programming, all of the = advice should=20 apply no matter which of the Bourne-derived shells you use. = Unfortunately, many=20 vendors have added features over the years and achieving complete = portability=20 can be a challenge. Explicitly writing for ksh (or bash) and insisting = on that=20 shell being installed, can often be simpler.=20

The sh and ksh man pages use the term special = command=20 for the internal commands - handled by the shell itself.=20

(7)

Basic script syntax

The most basic = shell script=20 is a list of commands exactly as could be typed interactively, prefaced = by the=20 #! magic header. All = the parsing=20 rules, filename wildcards, $PATH searches etc. which were summarized = above,=20 apply. In addition:=20
# as the first=20 non-whitespace character on a line=20
flags the line as a comment, and the rest of the line is = completely=20 ignored. Use comments liberally in your scripts, as in all other forms = of=20 programming.=20
\ as the last = character on=20 a line=20
causes the following line to be logically joined before = interpretation.=20 This allows single very long commands to be entered in the script in a = more=20 readable fashion. You can continue the line as many times as needed. =
This is actually just a particular instance of \ being to = escape, or=20 remove the special meaning from, the following character. =
; as a separator = between=20 words on a line=20
is interpreted as a newline. It allows you to put multiple = commands on a=20 single line. There are a few occasions when you must = do this,=20 but often it is used to improve the layout of compound commands.=20
Example 1: display, text=20

Every command has a value or exit status which it = returns=20 to the calling program. This is separate from any output generated. The = exit=20 status of a shell script can be explicitly set using exit N, or it defaults to the = value of=20 the last command run.

The exit status is an integer 0-255. Conventionally = 0=3Dsuccess and=20 any other value indicates a problem. Think of it as only one way for=20 everything to work, but many possible ways to fail. If the command was = terminated by a signal, the value is 128 plus the signal value. =

(8)

Style

Shell scripts are very = frequently written=20 quickly for a single purpose, used once and discarded. They are also as=20 frequently kept and used many times, and migrate into other uses, but = often do=20 not receive the same level of testing and debugging that other software = would be=20 given in the same situation. It is possible to apply general = principles=20 of good software engineering to shell scripts.=20
  • Preface scripts with a statement of purpose, author, date and = revision=20 notes=20
  • Use a revision control system for complex scripts with a long = lifetime=20
  • Assume your script will have a long lifetime unless you = are=20 certain it won't=20
  • Document any non-standard external utilities which your script = needs=20
  • Document your scripts with inline comments - you'll need them in a = few=20 months when you edit it.=20
  • Treat standard input and output in the normal way, so that your = script can=20 be used in combination with other programs=20
  • Be consistent in the format of your output, so that other programs = can=20 rely on it=20
  • Use options to control behaviour such as verbosity of output. = Overly=20 chatty programs are very hard to combine with other utilities=20
  • Test (a lot)

When not to use shell scripts

  • If an existing tool already does what you need - use it.=20
  • If the script is more than a few hundred lines, you are probably = using the=20 wrong tool.=20
  • If performance is horrible, and the script is used a lot, you = might want=20 to consider another language.

(9)

A Toolkit of commonly used external commands=20 (1)

The following commands are very frequently used in shell = scripts.=20 Some of them will be used in the following examples. This is just a = brief recap=20 -- see the man pages for details on usage.=20

Most of these commands will happily operate on a one or more named = files, or=20 will operate on a stream of data from standard input if no files are = named.=20

mkdir; rmdir=20
make directory and remove directory.=20
rm; cp; mv=20
Remove (delete), Copy and Move (rename) files.=20
touch=20
Update the last modifed timestamp on a file, to make it appear to = have=20 just been written. If the file does not exist, a new zero-byte file is = created, which is often useful to signify that an event has occurred.=20
grep=20
Search a file for lines containing character patterns. The = patterns can be=20 simple fixed text, or very complex regular expressions. The name comes = from=20 "Global Regular Expression and Print" -- a function from the Unix = editors=20 which was used frequently enough to warrant getting its own program.=20
sed=20
Stream Editor. A flexible editor which operates by applying = editing rules=20 to every line in a data stream in turn. Since it makes a single pass = through=20 the file, keeping only a couple of lines in memory at once, it can be = used=20 with infinitely large data sets. It is mostly used for global search = and=20 replace operations. It is a superset of 'tr', 'grep', and 'cut', but = is more=20 complicated to use.=20
ls=20
list contents of a directory, or list details of files and = directories.=20
cat=20
Copy and concatenate files; display contents of a file=20
sort=20
Sort data alphabetically or numerically.=20
uniq=20
Remove duplicate lines.=20
head; tail=20
Display the beginning of a file, or the end of it.=20
wc=20
Count lines, words and characters in a file.=20
tr=20
Transliterate - perform very simple single-character edits on a = file.=20
find=20
Search the filesystem and find files matching certain criteria = (name=20 pattern, age, owner, size, last modified etc.)=20
mail=20
Send mail, from a file or standard input, to named recipients. = Since=20 scripts are often used to automate long-running background jobs, = sending=20 notification of completion by mail is a common trick.

(10)

A Toolkit of commonly used external commands=20 (2)

The following commands are very frequently used in shell = scripts.=20 Some of them will be used in the following examples. This is just a = brief recap=20 -- see the man pages for details on usage.=20

Most of these commands will happily operate on a one or more named = files, or=20 will operate on a stream of data from standard input if no files are = named.=20

awk=20
A pattern matching and data manipulation utility, which has its = own=20 scripting language. It also duplicates much functionality from=20 'sed','grep','cut','wc', etc. Complex scripts can be written entirely = using=20 awk, but it is frequently used just to extract fields from lines of a = file=20 (similar to 'cut').=20
tee=20
Make a duplicate copy of a data stream - used in pipelines to send = one=20 copy to a log file and a second copy on to another program. (Think = plumbing).=20
xargs=20
Apply multiple filename arguments to a named command and run it. = Xargs is=20 often used in combination with "find" to apply some command to all the = files=20 matching certain criteria. Since "find" may result in a very large = list of=20 pathnames, using the results directly may overflow command line = buffers. Xargs=20 avoids this problem, and is much more efficient than running a command = on=20 every pathname individually.=20
diff=20
Compare two files and list the differences between them.=20
id=20
Print the user name and UID and group of the current user (e.g. to = distinguish priviledged users before attempting to run programs which = may fail=20 with permission errors)=20
who=20
Display who is logged on the system, and from where they logged = in.=20
uname=20
Display information about the system, OS version, hardware = architecture=20 etc.=20
cut=20
Extract selected fields from each line of a file. Often awk is = easier to=20 use, even though it is a more complex program.=20
paste=20
Merge lines from multiple files into tab-delimited columns.=20
join=20
Perform a join (in the relational database sense) of lines in two = sorted=20 input files.=20
dc=20
Desk Calculator - an RPN calculator, using arbitrary precision = arithmetic=20 and user-specified bases. Useful for more complex arithmetic = expressions than=20 can be performed internally or using expr=20
bc=20
A preprocessor for dc which=20 provides infix notation and a C-like syntax for expressions and = functions.=20
compress; gzip, zip; = tar=20
Various utilities to compress/uncompress individual files, combine = multiple files into a single archive, or do both.=20
logger=20
Place a message in the central system logging facility. Scripts = can submit=20 messages with all the facilities available to compiled programs. =

(11)

More external commands (mostly used with = sh)

The=20 following commands are needed when programming in sh, but are usually replaced = by=20 internal commands with ksh or bash.=20
echo=20
Echo the arguments to standard output -- used for messages from = scripts.=20 Some versions of "sh", and all csh/ksh/bash shells internalized = "echo".=20 Conflicts sometimes arise over the syntax for echoing a line with no = trailing=20 CR/LF. Some use "\c" and some use option "-n". To avoid these = problems, ksh=20 also provides the "print" command for output.=20
test; [=20
The conditional test, used extensively in scripts, is also an = external=20 program which evaluates the expression given as an argument and = returns true=20 (0) or false (1) exit status. The name "[" is a link to the "test" = program, so=20 a line like:
if [ -w logfile ]=20
actually runs a program "[", with arguments "-w = logfile=20 ]", and returns a true/false value to the "if" command. In ksh and = most newer=20 versions of sh, "[" is replaced with a compatible internal command. = Ksh also=20 provides the internal "[[" operator.=20
expr=20
The "expr" command takes an numeric or text pattern expression as = an=20 argument, evaluates it, and returns a result to stdout. Bourne shell = has no=20 built-in arithmetic operators or string manipulation. e.g.
expr 2 + 1
expr 2 '*' '(' 21 + 3 ')' =
Used=20 with text strings, "expr" can match regular expressions and extract = sub=20 expressions. Similar functionality can be achived with sed e.g.
expr SP99302L.Z00 :=20 '[A-Z0-9]\{4\}\([0-9]\{3\}\)L\.*'=20
basename = pathname=20
Returns the base filename portion of the named pathname, = stripping=20 off all the directories=20
dirname = pathname=20
Returns the directory portion of the named pathname, = stripping off=20 the filename

(12)

Shell Variables

Scripts are not very = useful if=20 all the commands and options and filenames are explicitly coded. By = using=20 variables, you can make a script generic and apply it to different = situations by=20 changing the variables. Variable names consist of letters and numbers=20 ([a-zA-Z0-9_], must start with a letter), and are case sensitive. = Several=20 special variables are used by the system -- you can use these, but may = not be=20 able to change them. The special variables use uppercase names, or = special=20 characters. Using lowercase names for your own variables is safest.=20

Setting and exporting variables

srcfile=3Ddataset1=20
Creates (if it didn't exist) a variable named "srcfile" and sets = it to the=20 value "dataset1". If the variable already existed, it is overwritten.=20 Variables are treated as text strings, unless the context implies a = numeric=20 interpretation. You can make a variable always be treated as a number. = Note=20 there must be no spaces around the "=3D".=20
set=20
Display all the variables currently set in the shell=20
unset srcfile=20
Remove the variable "srcfile"=20
srcfile=3D=20
Give the variable a null value, (not the same as removing it).=20
export srcfile=20
Added srcfile to the list of variables which will be made = available to=20 external program through the environment. If you don't do this, the = variable=20 is local to this shell instance.=20
export=20
List all the variables currently being exported - this is the = environment=20 which will be passed to external programs.

Using variables; simple modifiers.

$srcfile=20
Prefacing the variable name with $ causes the = value of the=20 variable to be substituted in place of the name.=20
${srcfile}=20
If the variable is not surrounded by whitespace (or other = characters that=20 can't be in a name), the name must be surrounded by "{}" braces so = that the=20 shell knows what characters you intend to be part of the name.=20

Example: 

datafile=3Dcensus2000
echo $datafile_part1.sas    # Tries to find $datafile_part1, which =
doesn't exist
echo ${datafile}_part1.sas  # This is what we intended
${datafile:-default}=20
Substitute the value of $datafile, if it has been set, otherwise = use the=20 string "default". This is an easy way to allow for optional variables, = and=20 have sensible defaults if they haven't been set. If $datafile was=20 uninitialized, it remains so.=20
${datafile:=3Ddefault}=20
Similar to the above, except if "datafile" has not been = initialized, do=20 so.

Variable assignment command prefix

It is possible to set a = variable just=20 for the duration of a single command using the syntax:
var=3Dvalue command args=20

(13)

Preset Shell Variables

Several special = variables=20 are used by the system -- you can use these, but may not be able to = change them.=20 The special variables use uppercase names, or punctuation characters. = Some=20 variables are set by the login process and inherited by the shell (e.g. = $USER), while others are used = only by=20 the shell. These are some of the more commonly used ones:=20

$USER, $LOGNAME=20
Preset to the currently logged-in username.=20
$PATH=20
The list of directories that will be searched for external = commands. You=20 can change this in a script to make sure you get the programs you = intend, and=20 don't accidentally get other versions which might have been installed. =
$TERM=20
The terminal type in which the shell session is currently. Usually = "xterm"=20 or "vt100". Many programs need to know this to figure out what special = character sequences to send to achieve special effects.=20
$PAGER=20
If set, this contains the name of the program which the user = prefers to=20 use for text file viewing. Usually set to "more" or "less" or = something=20 similar. Many programs which need to present multipage information to = the user=20 will respect this setting (e.g. man). This isn't actually = used by the=20 shell itself, but shell scripts should honour it if they need to page = output=20 to the user.=20
$EDITOR=20
If set, this contains the name of the program which the user = prefers to=20 use for text file editing. A program which needs to have the user = manually=20 edit a file might choose to start up this program instead of some = built-in=20 default (e.g. "crontab -e". This also determines the default=20 command-line-editing behaviour.=20
$PWD=20
Always set the current working directory (readonly)=20
$OLDPWD=20
The previous directory (before the most recent cd command)=20
$IFS=20
Internal Field Separators: the set of characters (normally space = and tab)=20 which are used to parse a command line into separate arguments. This = may be=20 set by the user for special purposes, but things get very confusing if = it=20 isn't changed back.=20
$$ (readonly)=20
Set to the process ID of the current shell - useful in making = unique=20 temporary files, e.g. /tmp/$0.$$=20
$PPID (readonly)=20
Set to the process ID of the parent process of this shell - useful = for=20 discovering how the script was called.=20
$! (readonly)=20
Set to the process ID of the last command started in background - = useful=20 for checking on background processes.=20
$? (readonly)=20
Set to the exit status of the last command run, so you can test = success or=20 failure. Every command resets this so it must be saved immediately if = you want=20 to use it later.=20
$-=20
Set to the currently set options flags.=20
$SECONDS (ksh, = readonly)=20
Integer number of seconds since this shell was started. Can be = used for=20 timing commands.=20
$RANDOM (ksh, = readonly)=20
Every time it is valuated, $RANDOM returns a random = integer in=20 the range 0-32k=20
$LINENO (ksh, = readonly)=20
Always evaluates to the current line number of the script being = executed -=20 useful for debugging.

(14)

Command Line (positional) arguments

To = customize=20 the behaviour of a script, you can give it any number of arguments on = the=20 command line. These are often filenames, but can be interpreted by the = script in=20 any way. Options are often specified using the "-flag" convention used = by most=20 Unix programs, and a ksh command getopts is available to help = parse=20 them. The shell expands wildcards and makes variable and command = substitutions=20 as normal, then parses the resulting words by whitespace (actually = special=20 variable $IFS), and = places the=20 resulting text strings into the positional variables as = follows:=20
$0, $1, $2, ... = $9=20
The first 9 arguments are made available directly as $1-$9. To = access more=20 than 9, use shift, or = $*, $@. The variable $0 contains the name of the = script=20 itself.=20
${10}, ${11}, ... =
Positional arguments greater than 9 are set by ksh (and bash). = Remember to=20 use braces to refer to them.=20
shift=20
discard $1 and renumber all the other variables. "shift N" will shift N = arguments at=20 once.=20
$#=20
contains the number of arguments that were set (not including $0). =
$*=20
contains all of the arguments in a single string, with one space=20 separating them.=20
$@=20
similar to $*, but if used in quotes, it effectively quotes each = argument=20 and keeps them separate. If any argument contains whitespace, the = distinction=20 is important.
e.g. if the argument list is: a1 a2 "a3 which contains spaces"=20 a4
then: $1=3Da1, $2=3Da2, $3=3Da3 which contains = spaces,=20 $4=3Da4
and: $*=3Da1 a2 a3 which contains spaces a4
and: = "$@"=3D"a1" "a2" "a3=20 which contains spaces" "a4"

Only using the form "$@" preserves = quoted=20 arguments. If the arguments are being passed from the script directly to = some=20 other program, it may make a big difference to the meaning.=20

Setting new arguments

The set command, followed by a = set of=20 arguments, creates a new set of positional arguments. This is often used = (if the=20 original arguments are no longer needed) to parse a set of words = (possibly using=20 different field separators). Arguments may be reset any number of times. =

Example

# Find an entry in the password file
pwent=3D`grep '^richard:' /etc/passwd`
# The "full name" and other comments are in
# field 5, colon delimited
IFS=3D:
set $pwent
echo $5

(15)

Manipulating Variables (ksh only)

Text variables

The pattern in the following uses the same = wildcards as for filename matching.=20
${#var}=20
returns the length of $var in characters=20
${var%pattern}=20
removes the shortest suffix of $var patching pattern=20
${var%%pattern}=20
removes the longest suffix of $var patching pattern=20
${var#pattern}=20
removes the shortest prefix of $var patching pattern=20
${var##pattern}=20
removes the longest prefix of $var patching pattern =

Numeric variables

$(( integer expression=20 ))=20
The $(( ... )) construction interprets the contents as an = arithmetic=20 expression (integer only). Variables are referenced by name without = the "$".=20 Most of the arithmetic syntax of the 'C' language is supported, = including bit=20 manipulations (*,/,+,-,|,&,<<,>>, parentheses for = changing=20 precedence).

Examples

datapath=3D/data/public/project/trials/set1/data= file.dat=20
filename=3D${datapath##*/}=20
filename is set = to=20 "datafile.dat" since the longest prefix pattern matching "*/" = is the=20 leading directory path (compare basename)=20
path=3D${datapath%/*}=20
path is set to=20 "/data/public/project/trials/set1" since the shortest suffix = pattern=20 matching "/*" is the filename in the last directory (compare dirname)=20

i=3D$((i+1))=20
often used in while loops=20

(16)

Command line options to ksh

Startup = options.=20 ksh -options scriptname =
-x=20
echo line to stderr before executing it=20
-n=20
read commands and check for syntax errors, but do not execute.=20
-a=20
all variables are automatically exported=20
set -x=20
Set an option within a shell script=20
$-=20
contains the currently set option letters

(17)

Command Substitution

sh syntax

`command`=20
A command (plus optional arguments) enclosed in backticks is = executed and=20 the standard output of that command is substituted. If the command = produces=20 multiline output, the newlines are retained. If the resultant string = is=20 displayed, unquoted, using echo=20 newlines and multiple spaces will be removed.

ksh syntax

$(command)=20
This syntax is functionally the same as backticks, but commands = can be=20 more easily nested.=20
$(<file)=20
This is equivalent to `cat=20 file`, but implemented internally for efficiency. =

Examples

echo Today is `date`

file=3D/etc/hosts
echo The file $file has $(wc -l < $file) lines

echo This system has host name $(</etc/nodename)

(18)

Conditional tests for [...] and [[...]]=20 commands

Most of the useful flow-control operators involve = making=20 some conditional test and branching on the result (true/false). The test = can be=20 either the test = command, or its=20 alias, [, or the ksh = built-in=20 [[ ... ]] command, = which has=20 slightly different options, or it can be any command which returns a = suitable exit status. Zero is taken to be "True", while any = non-zero value=20 is "False". Note that this is backwards from the C language convention.=20

File tests

-e file=20
True if file exists (can be of any type).=20
-f file=20
True if file exists and is an ordinary file.=20
-d file=20
True if file exists and is a directory.=20
-r file=20
True if file exists and is readable
Similarly, -w =3D writable, -x =3D executable, -L =3D is a symlink.=20
-s file=20
True if file exists and has size greater than zero=20
-t = filedescriptor=20
True if the open filedescriptor is associated with a = terminal=20 device. E.g. this is used to determine if standard output has be = redirected to=20 a file.

Character string tests

-n "string"=20
true if string has non-zero length=20
-z "string"=20
true if string has zero length
With [, = the=20 argument must be quoted, because if it is a variable that has a null = value,=20 the resulting expansion ( [ -z ] ) is a syntax error. An expansion = resulting=20 in "" counts as a null string. For [ only, a quoted string = alone is=20 equivalent to the -n test, e.g. [ "$var" ]. In older shells for = which [ is an external program, = the only=20 way to test for a null string is:
if [=20 "X$var" =3D "X" ]
$variable =3D = text=20
True if $variable matches text.=20
$variable < = text=20
True if $variable comes before (lexically)=20 text
Similarly, >=20 =3D comes after

Arithmetic tests

$variable -eq = number=20
True if $variable, interpreted as a number, is equal to=20 number.=20
$variable -ne = number=20
True if $variable, interpreted as a number, is not = equal to=20 number.
Similarly, -lt =3D less than, -gt =3D greater than =

Additional tests for [[...]] (ksh)

$variable =3D = pattern=20
True if $variable matches pattern. If pattern = contains no wildcards, then this is just an exact text match. The same = wildcards as used for filename matching are used. This is a much = simpler=20 regular expression match than available with tools like grep or expr.=20
file1 -nt file2=20
True if file1 is newer than file2.
Similarly = -ot =3D older than=20
file1 -ef file2=20
true if file1 is effectively the same as file2, = after=20 following symlinks and hard links.

Negating and Combining tests

Tests may be negated by prepending = the=20 ! operator, and = combined with=20 boolean AND and OR operators using the syntax:=20
conditional -a = conditional,=20 conditional -o conditional
AND and OR syntax for test and [=20

conditional &&=20 conditional, conditional || conditional=20
AND and OR syntax for [[ ...=20 ]]
Parentheses may be inserted to resolve=20 ambiguities or override the default operator precedence rules.=20

Examples

if [[  -x /usr/local/bin/lserve && -w =
/var/logs/lserve.log ]]; then
   /usr/local/bin/lserve >> /var/logs/lserve.log &
fi

pwent=3D`grep '^richard:' /etc/passwd`
if [ -z "$pwent" ]; then
   echo richard is not in the password file
fi

(19)

Flow Control and Compound Commands

A = list=20 in these descriptions is a simple command, or a pipeline. The value of = the=20 list is the value of the last simple command run in it.
A list can also be a set of simple commands or = pipelines=20 separated by ";,&,&&,||,|&". For the compound commands = which=20 branch on the success or failure of some list, it is usually = [ or [[, but can be anything. =

list &&=20 list=20
Execute the first list. If true (success), execute the = second one.=20
list || = list=20
Execute the first list. If false (failure), execute the = second one.=20

Example: 

    mkdir tempdir && cp workfile tempdir

    sshd || echo "sshd failed to start"
if list; then list ; = elif=20 list; else list; fi=20
Execute the first list, and if true (success), execute the = "then"=20 list, otherwise execute the "else" list. The "elif" lists are optional =

Example: 

    if [ -r $myfile ]
    then
       cat $myfile
    else
       echo $myfile not readable
    fi
while list; do; list; = done=20
until list; do; list; = done=20
Execute the first list and if true (success), execute the = second=20 list. Repeat as long as the first list is true. The = until form just negates the = test.=20

Example: 

    count=3D0
    max=3D10
    while [[ $count -lt $max ]]
    do
        echo $count
        count=3D$((count + 1))
    done
for identifier [ in = words ];=20 do; list; done=20
Set identifier in turn to each word in words and = execute the=20 list. Omitting the "in words" clause implies using $@, = i.e. the=20 identifier is set in turn to each positional argument.=20

Example: 

    for file in *.dat
    do
        echo Processing $file
    done
As with most programming languages, there are often = several ways=20 to express the same action. Running a command and then explicitly = examining=20 $? can be equivalent to = some of=20 the above.=20

(20)

Flow Control and Compound Commands = (contd.)

case word in pattern) = list;; esac=20
Compare word with each pattern) in turn, and = executes the=20 first list for which the word matches. The = patterns=20 follow the same rules as for filename wildcards.
(Ksh only) A pattern-list is a list of one or more = patterns=20 separated from each other with a |. Composite patterns can be formed = with=20 one or more of the following:=20
?(pattern-list)=20
Optionally matches any one of the given patterns.
*(pattern-list)=20
Matches zero or more occurrences of the given patterns.
+(pattern-list)=20
Matches one or more occurrences of the given patterns.=20
@(pattern-list)=20
Matches exactly one of the given patterns.=20
!(pattern-list)=20
Matches anything, except one of the given patterns.=20

Example: 

    case $filename in
    *.dat)
        echo Processing a .dat file
        ;;
    *.sas)
        echo Processing a .sas file
        ;;
    *)
        # catch anything else that doesn't match patterns
        echo "Don't know how to deal with $filename"
        ;;
    esac
break [n]=20
Break out of the current (or n'th) enclosing loop. Control jumps = to the=20 next statement after the loop=20

continue [n];=20
Resume iteration of the current (or n'th) enclosing loop. Control = jumps to=20 the top of the loop, which generally causes re-evaluation of a while or processing the = next element=20 of a for.=20

. filename=20
Read the contents of the named file into the current shell and = execute as=20 if in line. Uses $PATH to locate the file, and can be passed = positional=20 parameters. This is often used to read in shell functions that are = common to=20 multiple scripts. There are security implications if the pathname is = not fully=20 specified.=20

( ... ) Command = grouping=20
Commands grouped in "( )" are executed in a subshell, with a = separate=20 environment (can not affect the variables in the rest of the script). = Some of=20 the compound commands run in implicit subshells, which affects setting = of=20 variables which may be needed later.

(21)

Basic I/O

Any simple command (or shell function, or compound command) may have = it's=20 input and output redirected using the following operators.=20
> = filename=20
Standard ouput (file descriptor 1) is redirected to the named = file. The=20 file is overwritten unless the noclobber option is set. = The file is=20 created if it does not exist.=20
>> = filename=20
Standard ouput is appended to the named file. The file is created = if it=20 does not exist.=20
>| = filename=20
Output redirect, and override the noclobber option, if set. =
< = filename=20
Standard input (file descriptor 0) is redirected to the named = file. The=20 file must already exist.=20
command | command [ | command=20 ...]=20
Pipe multiple commands together. The standard output of the first = command=20 becomes the standard input of the second command. All commands run=20 simultaneously, and data transfer happens via memory buffers. No more = than one=20 command in a pipeline should be interactive (attempt to read from the=20 terminal). This construct is much more efficient than using temporary = files,=20 and most standard Unix utilities are designed such that they work well = in=20 pipelines.

Shell scripts can generate output directly or read input into = variables using=20 the following commands:=20

echo=20
Print arguments, separated by spaces, and terminated by a newline, = to=20 stdout. Use quotes to preserve spacing. Echo also understands C-like = escape=20 conventions, but beware that the shell may process backslashes before = echo sees them (may need to = double=20 backslash). Internal in most shells, but was originally external. = 
 \b   backspace
 \c   print line without new-line (some versions)
 \f   form-feed
 \n   new-line
 \r   carriage return
 \t   tab
 \v   vertical tab
 \\   backslash
 \0n  where n is the 8-bit character whose ASCII=20
      code is the 1-, 2- or 3-digit octal number=20
      representing that character.
-n=20
suppress newline

print (ksh)=20
Print arguments, separated by spaces, and terminated by a newline, = to=20 stdout. Print observes the same escape conventions as echo. It is = internal in=20 ksh.=20
-n=20
suppress newline=20
-r=20
raw mode - ignore \-escape conventions=20
-R=20
raw mode - ignore \-escape conventions and -options except -n. =

read var1 var2 = rest=20
read a line from stdin, parsing by $IFS, and placing the words = into the=20 named variables. Any left over words all go into the last variable. A = '\' as=20 the last character on a line removes significance of the newline, and = input=20 coninues with the following line.=20
-r=20
raw mode - ignore \-escape conventions

<< = [-]string=20 ("here documents")=20
redirect input to the temporary file formed by everything up the = matching=20 string at the start of a line.

Example 

cat <<EOF
 This text will be fed to
 the "cat" program as standard
 input.  It will also have variable
 and command substitutions performed.
 I am logged in as $USER and today is `date`
EOF

cat <<"EndOfInput"
 This text will be fed to
 the "cat" program as standard
 input.  Since the text string marking the=20
 end was quoted, it does not get variable and command
 subsitutions.
 I am logged in as $USER and today is `date`
 The terminating string must be at the start of a line.
EndOfInput

(22)

Shell Functions

All but the earliest = versions of=20 sh allow you define shell functions, which are visible only to = the shell=20 script and can be used like any other command. Shell functions take = precedence=20 over external commands if the same name is used. Functions execute in = the same=20 process as the caller, and must be defined before use (appear earlier in = the=20 file).=20
  • Defining functions=20
    function identifier { = list;=20 }=20
    identifier() { = list;=20 }=20
    Equivalent syntax for defining a function. The identifier = follows the=20 rules for variable names, but uses a separate namespace.

    Example: 

    die()
{
   # Print an error message and exit with given status
   # call as: die status "message" ["message" ...]
   exitstat=3D$1; shift
   for i in "$@"; do
      print -R "$i"
   done
   exit $exitstat
}
  • Calling functions.
    Functions are called like any other = command. The=20 output may be redirected independantly of the script, and arguments = passed to=20 the function. Shell option flags like -x are unset in a function - you = must=20 explicitly set them in each function to trace the execution.=20

    Example: 

    die 1 "file not writeable" "check permissions"
  • Global vs local variables
    A function may read or modify any = shell=20 variable that exists in the calling script. Such variables are global. =

    (ksh only) Functions may also declare local variables in the = function using=20 typeset. Local = variables are=20 visible to the current function and any functions called by it.=20

    Example: 

    vprint()
{
   # Print or not depending on global "$verbosity"
   # Change the verbosity with a single variable.
   # Arg. 1 is the level for this message.
   level=3D$1; shift
   if [[ $level -le $verbosity ]]; then
      print -R $*
   fi
}

verbosity=3D2
vprint 1 This message will appear
vprint 3 This only appears if verbosity is 3 or higher

  • Returning from functions=20
    return [n], = exit [n]=20
    Return from a function with the given value, or exit the whole = script=20 with the given value.
    Without a return, the function = returns when it=20 reaches the end, and the value is the exit status of the last command = it ran.=20

  • Reuseable functions
    By using only command line arguments, not = global=20 variables, and taking care to minimise the side effects of functions, = they can=20 be made reusable by multiple scripts. Typically they would be placed = in a=20 separate file and read with the . operator.

(23)

Advanced I/O

Unix I/O is performed by = assigning=20 file descriptors to files or devices, and then using those descriptors = for=20 reading and writing. Descriptors 0, 1, and 2 are always used for stdin, = stdout=20 and stderr respectively. Stdin defaults to the keyboard, while stdout = and stderr=20 both default to the current terminal window.=20
  • Redirecting stdout, stderr and other file descriptors for the = whole script=20
    exec > outfile < infile=20
    with no command, the exec=20 just reassigns the I/O of the current shell.=20
    exec = n>outfile=20
    The form n<, n> opens file descriptor n instead of the = default=20 stdin/stdout. This can then be used with read=20 -u or print=20 -u.

  • Explicitly opening or duplicating file descriptors=20
    >&n=20
    standard output is moved to whatever file descriptor n is = currently=20 pointing to=20
    <&n=20
    standard input is moved to whatever file descriptor n is = currently=20 pointing to=20
    n>file=20
    file descriptor n is opened for writing on the named = file.=20
    n>&1=20
    file descriptor n is set to whatever file descriptor 1 is = currently=20 pointing to.
    e.g. echo "Error: = program=20 failed" >&2 writes an error message to = stderr.=20

  • Printing to file descriptors (more efficient than = open/append/close) (ksh)=20
    print -u n = args=20
    print to file descriptor n.=20
    -p=20
    write to the pipe to a coprocess (opened by |&)

  • Reading from file descriptors other than stdin (ksh)=20
    read -u n var1 var2=20 rest=20
    read a line from file descriptor n, parsing by $IFS, and placing = the=20 words into the named variables. Any left over words all go into the = last=20 variable.=20
    -p=20
    read from the pipe to a coprocess (opened by |&)

  • Closing file handles=20
    <&-=20
    standard input is explicitly closed=20
    >&-=20
    standard output is explicitly closed
    For example, to = indicate to=20 another program downstream in a pipeline that no more data will be = coming. All=20 file descriptors are closed when a script exits.
I/O = redirection=20 operators are evaluated left-to-right. This makes a difference in a = statement=20 like: >filename = 2>&1=20

(24)

Coprocesses and Background = jobs

Scripts can=20 start any number of background jobs (any external command), which run in = parallel with the parent script, and asynchronously. Interaction with = background=20 jobs is tricky. You can use signals, named pipes, or disk files for=20 communication.=20
command = &=20
Start command as a background process. Control returns = immediately=20 to the shell.=20
bgpid=3D$!=20
The special variable $!=20 contains the process ID of the last background job that was started. = You can=20 save that and examine the process later (ps -p=20 $bgpid) or send it a signal (kill -HUP $bgpid). =

Coprocesses (ksh only) are a way of starting a separate = process which=20 runs asychronously, but has stdin/stdout connected to the parent script = via=20 pipes.=20

command |& =
Start a coprocess with a 2-way pipe to it=20
read -p = var=20
Read from the pipe to the coprocess, instead of standard input=20
print -p = args=20
Write to the pipe connected to the coprocess, instead of standard = output=20
Multiple coprocesses can be handled by moving the special = file=20 descriptors connected to the pipes onto standard input and output, and = or to=20 explicitly specified file descriptors.=20
exec <&p=20
The input from the coprocess is moved to standard input=20
exec >&p=20
The output from the coprocess is moved to standard output =

Examples

  1. A script wants to save a copy of all output in a file, but also = wants a=20 copy to the screen. This is equivalent to always running the script as =
    script | tee=20 outfile 
    if [[ -t 1 ]] ; then
  # Only do this if fd 1 (stdout) is still connected
  # to a terminal

  print "Setting up output to tee into $0.out"

  # We want the standard output of the "tee" process=20
  # to go explicitly to the screen (/dev/tty)
  # and the second copy goes into "logfile"

  tee logfile >/dev/tty |&

  # Our stdout all goes into this coprocess
  exec 1>&p=20
fi
  2. Start a coprocess to look up usernames in some database. It is = faster to=20 run a single process than to run a separate lookup for each user. = 
    # start the coprocess
dndlookup -fname -u |&

# move the input/output streams so we=20
# can use other coprocesses too
exec 4>&p
exec 5<&p

# read the names from a file
cat namefile | while read pwentry; do
  print -u4 $uname
  read  -u5 dndname
  case $dndname in
  *many matches*)
    # handle case where the name wasn't unique
    ;;
  *no match*)
    # handle case where the name wasn't found
    ;;
  *)
    # we seem to have a hit - process the
    # canonical named retrieved from dndlookup
    ;;=20
  esac
done

# We've read all the names, but the coprocess
# is still running.  Close the pipe to tell it
# we have finished.
exec 4>&-

(25)

Delivering and Trapping Signals

Unix = signals=20 (software interrupts) can be sent as asynchronous events to shell = scripts, just=20 as they can to any other program. The default behaviour is to ignore = some=20 signals and immediately exit on others. Scripts may detect signals and = divert=20 control to a handler function or external program. This is often used to = perform=20 clean-up actions before exiting, or restart certain procedures. = Execution=20 resumes where it left off, if the signal handler returns. Signal traps = must be=20 set separately inside of shell functions. Signals can be sent to a = process with=20 kill.=20
trap handler sig=20 ...=20
handler is a command to be read (evaluated first) and = executed on=20 receipt of the specified sigs. Signals can be specified by name = or=20 number (see kill(1)) e.g. HUP, INT, QUIT, TERM. A Ctrl-C at the = terminal generates a INT.
  • A handler of - = resets the=20 signals to their default values=20
  • A handler of '' = (null)=20 ignores the signals=20
  • Special signal values are as follows:=20
    EXIT=20
    the handler is called when the function exits, or when the whole = script=20 exits. The exit signal has value 0.=20
    ERR (ksh)=20
    the handler is called when any command has a non-zero exit = status=20
    DEBUG (ksh)=20
    the handler is called after each command. =

Example: 

#!/bin/ksh

trap huphandler  HUP
trap ''          QUIT
trap termhandler TERM
     =20
huphandler()
{
   print 'Received SIGHUP'
}

termhandler()
{
   print 'Received SIGTERM'
   exit 1
}


while true; do
   sleep 5
   print -n "$SECONDS "
done
Exit handlers can be defined to clean up temporary files or reset = the=20 state of devices. This can be useful if the script has multiple possile = exit=20 points.=20

(26)

Security issues in shell scripts

Shell = scripts=20 are often used by system administrators and are run as a priviledged = user.=20
  • Don't use Set-UID scripts. Many system don't allow a script to be = made=20 set-uid. It is impossible to ensure that a set-uid script cannot be=20 compromised. Use wrapper programs like sudo instead.=20

  • Always explicitly set $PATH=20 at the start of a script, so that you know exactly which external = programs=20 will be used.=20

  • Be careful what you put in /tmp, or other public-writeable = directory.=20 Don't write without checking to see if the file exists first. Often = scripts=20 will write to a fixed, or trivially generated temporary filename in = /tmp. If=20 the file already exists and you don't have permission to overwrite it, = the=20 script will fail. If you do have permission to overwrite it, you will = delete=20 the previous contents. Safe scratch files can be made by creating a = new=20 directory, owned and writeable only by you, then creating files in = there.=20

    Example:
    A link is created by another user in /tmp: = /tmp/scratch ->=20 /vmunix
    A root user runs a script that blindly writes a scratch = file to=20 /tmp/scratch, and deletes the operating system.=20

  • Check error status of everything you do.=20

  • Don't trust user input=20
    • contents of files=20
    • data piped from other programs=20
    • file names. Output of filename generation with = wildcards, or=20 directly from ls or = find

Example:
Consider the effects of a file named "myfile;cd /;rm *" = if=20 processed, unquoted, by your script.=20

One possible way to protect against weirdo characters in file names: = 

# A function to massage a list of filenames=20
# to protect weirdo characters
# e.g. find ... | protect_filenames | xargs command
#
# We are backslash-protecting the characters \'" ?*;
protect_filenames()
{
   sed -es/\\\\/\\\\\\\\/g \
       -es/\\\'/\\\\\'/g   \
       -es/\\\"/\\\\\"/g   \
       -es/\\\;/\\\\\;/g   \
       -es/\\\?/\\\\\?/g   \
       -es/\\\*/\\\\\*/g   \
       -es/\\\ /\\\\\ /g
}

(27)

Miscellaneous internal commands

The = shells (ksh=20 in particular) have many more internal commands. Some are used more in=20 interactive shells. The commands listed here are used in scripts, but = don't=20 conveniently fit elsewhere in the class.=20
eval args=20
The args are read as input to the shell and the resulting command=20 executed. Allows "double" expansion of some constructs.=20
exec command args =
The command is executed in place of the current shell. = There is no=20 return from an exec. I/O redirection may be used. This is also used to = change=20 the I/O for the current shell.=20
:=20
The line is variable-expanded, but otherwise treated as a comment. = Sometimes used as a synonym for "true" in a loop.=20 

while :; do
  # this loop will go forever until broken by=20
  # a conditional test inside, or a signal
done

unset var = ...=20
Remove the named variables. This is not the same as setting their = values=20 to null.=20

typeset [+/- options] [=20 name[=3Dvalue] ] ... (ksh only)=20
Set attributes and values for shell variables and functions. When = used=20 inside a function, a local variable is created. Some of the options = are:=20
-L[n]=20
Left justify and remove leading blanks. The variable always has = length n=20 if specified.=20
-R[n]=20
Right justify and fill with leading blanks. The variable always = has=20 length n if specified.=20
-Z[n]=20
As for -R, but fill with zeroes if the value is a number=20
-i=20
The named variable is always treated as an integer. This makes=20 arithmetic faster. The reserved word integer is an alias for = typeset -i.=20
-l=20
Lower-case convert the named variables=20
-u=20
Upper-case convert the named variables=20
-r=20
Mark the variables as readonly=20
-x=20
Export the named variables to the enviroment=20
-ft=20
The variables are taken as function names. Turn on execution = tracing.=20

(28)

Some longer examples

The class accounts have a ksh/Blinn = directory=20 containing all of the example scripts in the book by Blinn (1995, see=20 references). Some of these are reproduced below.=20

  • MailPkg: display, text
    Tar, compress, split and uuendcode a set = of files=20 for mailing. (Blinn)=20

  • ptree (original) display, text
  • ptree (ksh) display, text
    Runs "ps" to get a process listing and = then=20 reformats to show the process family hierarchies. The original example = is pure=20 Bourne shell and inefficient. The ksh version is a fairly simple = translation=20 to use ksh internal commands where possible, and avoid writing scratch = files,=20 and runs very much faster. (Blinn).=20

  • pickrandom display, text
    Selects a random file from a directory. = Uses the=20 ksh RANDOM feature.

This entire tutorial was created from individual HTML pages using a = content=20 management system written as ksh scripts (heavily using sed to edit the = pages),=20 coordinated by make.=20

  • buildhtml: display, text
    Turns a stand-alone HTML file into a = component of=20 the tutorial, with navigation links and frames.=20

  • buildslidelist: display, text
    Creates the slide list which appears as = the left=20 frame of the tutorial.

(29)

References, Resources, Man pages = etc.

The=20 standard man pages for sh and=20 ksh are quite complete, = but not=20 easy to learn from. The following is a sampling of the many available = books on=20 the subject. The Bolsky and Korn book might be viewed as the standard=20 "reference". The Blinn book is Bourne shell, but everything in it should = work=20 for either shell.
The links are to publisher's web sites, or = Amazon.com.=20

Books

  • The New KornShell Command And Programming Language, by = Morris I.=20 Bolsky, David G. Korn (Contributor). More=20 info

  • Learning the Korn Shell (A Nutshell Handbook) by Bill = Rosenblatt,=20 Mike Loukides (Editor). More=20 info

  • Korn Shell Programming by Example, by Dennis O'Brien, David = Pitts=20 (Contributor). More=20 info

  • The Korn Shell Linux and Unix Programming Manual (With = CD-ROM) by=20 Anatole Olczak. More=20 info

  • Portable Shell Programming: An Extensive Collection of Bourne = Shell=20 Examples by Bruce Blinn. More=20 info

  • Teach yourself Shell Programming in 24 Hours, by S. = Veeraraghavan.=20 SAMS 2nd Edn. (2002) More=20 info

  • Unix Power Tools, by S. Powers, J. Peek, T. O'Reilly, M. = Loudikes=20 et al. More=20 info

Online Resources



Unix shell scripting with sh/ksh: Course = Handout
(last Update   28 February = 2003)  =A9Dartmouth College=20     http://www.dartmouth.edu/~rc/classes/ksh=20