shellscript.html

Note: This page was written for FreeBSD. Most of example should be usable on Linux and Cygwin, too

Why

Well, who's going to bother about shell script these days, huh ? It's only me ? Maybe so.
But more I learn about shell script, more I have fun with it.
It's simple, fast enough, powerfull enough for many tasks, including for CGI !
It's one of the lightest CGI scripting language, and very easy to use.

Other than that, you can manage your machine and web site with shell script very easliy !

How

Using shell script is easy ... on the web server (Linux, FreeBSD, etc),
or on your machine with Cygwin.
(Cygwin lets you use popular Unix commands on Windows Machine.)

First, we'll see the example of very simple CGI.

To use shell script as a cgi, your should start the file with these two lines.

#!/bin/sh
echo -e "Content-type: text/html\n"

And then, set the permission of file to 700 (on the server with SuExec, like PowWeb),
with FTP, WebFTP, sitemanager, etc. (Use 755 if CGI runs as nobody)

To output texts and html codes, use "echo".

#!/bin/sh
echo -e "Content-type: text/html\n"

echo '<HTML><BODY>
<H1>Title</H1>
Text here.
</BODY></HTML>'

# END
# Any line starting with '#' is a comment and ignored.

Notice that "echo" will output until the 2nd single quote
if the text is started with a single quote.

You must have a space between "echo" and the 1st single quote.

If you want to use sigle quotes in the text, add \ after it. 
In other words, '\ will show a single quote in the string surounded by single quotes.
  Ex. echo 'Text with quote '\here'\.'

You can also use double quotes "" for delimiting echo output.
In this case, variables will be expanded. 
And you can use single quotes without special attention.

Next, to include any file, you use "cat".

# include a file in the same directory
cat header.html

# include a file in te subdirectory of current directory
cat inc/footer.html

# Or use absolute path. (It can be any file extension.)
cat /www/U/USER/htdocs/template/toppage.tpl

Both combined, you can do like this.

#!/bin/sh
echo -e "Content-type: text/html\n"

# include header part
cat header.html

# output some text (between quotes)
echo '<center>
Here is a simple dynamic page <BR>
served by shellscript!
</center>'

# include footer part from 'inc' directory
cat inc/footer.html

# END

shellscript pages can be cleaner than SSI, PHP, or Perl,
and simple and powerful enough for many tasks !


Also, it uses a lot less resources than PHP or Perl
and thus it loads pretty rapidly !
The difference will be even greater when the server starts to suffer from heavier load.
While a shellscript page may manage to load, php version may timeout.

As it uses less resources, shellscript pages are less vulnerable
when you get massive requests from search engines, robots, attackers, whatever.

In addition to that, the skills and knowledge leaned with shellscript pages
are very usefull in site management and system administration.
It can be used in Perl and other languages, too.

More of simple examples to come .....

Shell script code bits

In the shell script programming, if you do things with internal command,
it will use less resources and run quicker.
So, I thought about how to do things WITHOUT using external command
like test (or [ ] ), expre, sed, and so on.

Actually, most of common simple task can be done only with internal commands.


Note: These are based on FreeBSD /bin/sh. Please read the manual!.
If you are using Linux or cygwin bash shell, you have more internal commands
(with the price of heavier resource usage ...)
Also, bash does things a little differently.

For those who intersted in how far we can go to minimize things,
there is a shell subset ENTIRELY written in x86 assembler.
Take a look at this thread.

Debugging

By the use of exec, and set options.

#!/bin/sh
echo -e "Content-type: text/html\n\n<PRE>"

# This will redirect error output (stderr) to the standard output (stdout)
exec 2>&1

# turn on verbose option
set -v

# and/or turn on trace option
set -x

echo "Let's make an error"
ls badabudibuahaha

# to turn off all options
set -

By using function and redirect.

#!/bin/sh

echo -e "Content-type: text/html\n\n<PRE>"

# define a function
myfunc (){
  # Put your code here. Some errors will be shown when it's executed.
  echo "myfunc"
  
  # shell function definition can be nested.
  anotherfunc () echo "blah"; return 1
  
}

# execute it with stderr redirected to stdout
myfunc 2>&1

By invoking from onother script.

#!/bin/sh
# Another option would be invoking the script from another one
# to see the stderr output.
#

echo -e "Content-type: text/html\n\n<PRE>"

./myscript.cgi 2>&1

# Optionally, you can turn on the tracing. (And maybe -v , verbose)
/bin/sh -x myscript.cgi 2>&1

# Or run just syntax checking
/bin/sh -n myscript.cgi 2>&1

General tips for performance tunig if you care about that

Note: You don't have to worry about these in most small script
without recursion or looping.

Use builtins if available, espacially in the loop.

Builtins for /bin/sh FreeBSD 4.9: 
     alias, bg, bind, break, builtins, case,
     cd, chdir, command, continue, do, done, echo,
     elif, else, esac, eval, exec, exit, export,
     false, fc, fg, fi, for, getopts, hash,
     if, jobid, jobs, pwd, printf, read, readonly, set, setvar,
     shift, test, then, trap, true, type, ulimit, umask, unalias, unset,
     until, wait, while
 
Note: printf isn't in the list of manpage for builtins, 
      but it's shell builtin according to 'type' command

Avoid subshell in a long tight loop if possible.
Subshell is invoked with:
      external command
      (any_command ; internal_command ;)
      `any_command`


Keep the file descripter open for repeated output.

# open a file as descriptor 9
exec 9>somefile.txt
   
# repeated output
for f in *.html
do
 echo $f >&9
 wc $f >&9
done
   
# close file
exec 9>-
   
Any other ideas?

String processing

Basic string processiong. See the manual for more details.

# Set 'abc/de/hhh' to a variable $STR
# Note that to set value, you must not put $ before var name.
STR='abc/def/hhh'

# Cut off after the last '/' and anything behind.
#   Useful in obtaining directory part ot a path (dirname)
echo ${STR%/*} ;#==> abs/def


# Cut off after the first '/' and anything behind.
echo ${STR%%/*} ;#==> abs

# Cut off up to and including the first '/'.
echo ${STR#*/} ;#==> def/hhh

# Cut off up to and including the last '/'.
#   Useful in obtaining filename part ot a path (basename)
echo ${STR##*/} ;#==> hhh

Take our the first two characters.
substr($STR, 0, 2) in some languages.


#   Cut out the first two characters,  ==> c/def/hhh
#   and then cut out that part 'c/def/hhh' from the original 'abc/def/hhh'
echo ${STR%${STR#??}} ;#==> ab

Take our the last two characters substr($STR, $#STR-2 ,2)

#   Cut out the last two characters,  ==> abc/def/h
#   and then cut out that part 'abc/def/h' from the original 'abc/def/hhh'
echo ${STR#${STR%??}} ;#==> hh

Then there are many default value setting methods

# If $HOME is not set ro NULL, set $DOCUMENT_ROOT minus '/*' and expand.
echo ${HOME:=${DOCUMENT_ROOT%/*}}

To obtain /www/U/USER type $HOME of PowWeb

#   fisrt, set a variable with the output of whoami command using backticks``.
#   Then, use the default setting method combined with 
#   other string manipulation to obtain U/USER part.
U=`whoami`
echo ${HOME:=/www/${U%${U#?}}/U}

Variables can act as both Numeric and String type

# Set 45 to $STR
STR=45
# Set the result of calculation $STR * 4 to $STR
#   Note that arithmetic operation must be placed in the $(( and ))
STR=$(($STR * 4))
# cut out last two digits.
echo ${STR%??}

Short associative array (or dict, hash,list, whatever)

# connect the key and value with underscore _
# separate each elements by a space
# You can change the code to use different connector $KV and separator $SEP.
#  ex. kv='==>';SEP=','

# Make the dict.
X='Not_Valid 1_abcd 2_ef 3_abc 4_123 5_/s'

# Set key to seach
N=3 

$KV='_'
$SEP=' '
# Get the value by:
#  remove up to and including the key
XX=${X#*$N$KV}
#  remove everything after the value
echo ${XX%%$SEP*}

# If you give invalid key, it will return 'Not_Valid' in this case.
# I guess, you can construct linked list and B-tree and whatever 
# with thi kind of trick.
# Probably, quick enough for small data (Limited by the env size).

Using case ) ;; esac for string processing and conditional structure.


# You should put the variable in the double quots to prevent it 
#  from expanded any further.
case "$STR" in
  '') 
    echo "EMPTY" 
    ;;
  as*) 
    echo "Starts with 'ab'" 
    ;;
  *ef) 
    echo "Ends with 'ef'" 
    ;;
   aa|bb|cc) 
    echo "'aa','bb', or 'cc'"
    ;; 
  [abc]?) 
    echo "Start with 'a', 'b', or 'c' and ends with anyother charcter" 
    ;;
  *[!0-9]*) 
    echo "Contains non-digit" 
    ;;
  ????) 
    echo "Four characters" 
    ;;
  *$PAT) 
    echo "Ends with $PAT" 
    ;;
  *) 
    echo "Anything. Default" 
    ;;
esac

case ..esac in one line


# More or less normal way of writing.
case "$STR" in 
  '')
    echo "EMPTY"
    ;;
  as*)
    echo "Starts with 'ab'"
    ;;
esac

# In compact form.
case "$STR" in '')echo "EMPTY";;as*)echo "Starts with 'ab'";;esac

for(C=7;C > 0;C++){...} type of loop without test

On FreeBSD 4.9, "test" is a builtin and we don't need to do following.


# Avoid 'test [ ]' to speed up the loop if the test isn't builtin
# by using internal shell function to check the value !

# shell function "nz arg" will return True (0) if the "arg" is non-zero
nz () case $1 in 0)return 1;;*) return 0;;esac

C=7
while nz $C
do
 # do anything here 
 echo $C
 
 # decriment $C
 C=$(($C - 1))
done

# Other functions that can help "while" loop, 
# <b>as well as "if ;then  else   ;fi"</b>

# 1 argument
nz () case $1 in 0)return 1;;*) return 0;;esac 
nil () case $1 in 0|'')return 0;;*) return 1;;esac
pos () case $(($1 + 0)) in -*)return 1;;*) return 0;;esac
neg () case $(($1 + 0)) in -*)return 0;;*) return 1;;esac

# 2 argments. "eq" and "ne" can compare both string and numbers.
eq () case $2 in $1)return 0;;*) return 1;;esac
ne () case $2 in $1)return 1;;*) return 0;;esac
gt () case $(($2 - $1)) in -*)return 0;;*) return 1;;esac
lt () case $(($1 - $2)) in -*)return 0;;*) return 1;;esac

# Use the value of the argument as return value
val () return $1

# An alternative mothod using case ...esac for breaking out of the loop
# Essencially the same as the above. Maybe less intutive.

C=7
while :; do 
 case "$C" in ))::0)break;;esac
 # any command here
 echo "In theloop No.$C"
 C=$(($C - 1))
done

# To use while for infinite loop, use 'while :' 
# ':' is a internal commamd that returns TRUE

# BE CAREFULL with while loop.
# Unless you put a solid loop stopper, it can go into infinite loop.

# If you have doubt, put "sleep 1" to slow down the loop
# so that you can kill it by browser or killing cgi.

# Or use emergency stop counter untill all debugging is done.

# Another alternative method for small loops

for C in 0 1 2 3 4 5 6 7 8
do
 # do anything here 
 echo $C
done

# This method is more convenient if you know the number of looping in advance
# and if it's a small number.

Replace all occurance of $P with $R

P='\'
R='__'
S='xyz\sample\text\end'

while :
do 
  S="${S%$P*}$R${S##*$P}"
  case "$S" in
    *"$P"*)
      continue
      ;;
    *)
      break
      ;;
  esac
done
# ==> xyz__sample__text__end

# Compact form
while :; do S="${S%$P*}$R${S##*$P}"
  case "$S" in));;*"$P"*)continue;;*)break;;esac;done


   
# As a shell function
#
# Take 2 or 3 arguments in from of:
#   repl "search_pat" "replace" ["string"]
# If the 3rd argument is missing, it will try to use $Repl.
# The result will be stored in $Repl.
#
# Only very basic error checking is done.

repl (){
  local P=$1
  local R=$2
  local S=$3
  S=${S:-Repl}
  case $P in '')return 1;;esac
  case $S in '')return 2;;esac
  while :; do S="${S%$P*}$R${S##*$P}"
    case "$S" in));;*"$P"*)continue;;*)break;;esac;done
  Repl=$S
}
repl "pat" "replaceed" "this pat and that pattern"
echo $Repl
# ==> this replaceed and that replaceedtern

'?' x $C ---> $P = '????'

C=4
p='?'

while :
do 
  P=$P$P
  case "$C" in
    ${#P})
      break
      ;;
    esac
done

# Compact form
while :;do P=$P$P;case "$C" in));;${#P})break;;esac;done

More complex examples

Please see myini.cgi and other tools
found in the tools section such as:

Some of these use all the trick and more to read a file and parse it 
to store some data in a list/array,
and then use it to find and replace lines in another file.

An easy job for a sed command script, but a little complicated for 
a PURE shell script without any external commands!

Helpful Links

http://www.troubleshooters.com/codecorn/shellscript/

Very nice tutorial with many examples.
Some of code does not apply to old sh. (String replacements, "let", and so on)

http://c2.com/cgi/wiki?UnixShellPatterns

From Wiki site.

http://www.shelldorado.com/

Lot's of tips. Mostly with external commands.
I guess I'm one of very few person obsessed with the idea of avoiding
external commands as much as possible.

http://www.onesmartclick.com/programming/shell-programming.html

A link page with many mnay links.

Questionable color of this page is dictated by blueberry cream cake, my favorite dessert.