Stream I/O in Qshell

by Ted Holt

Standard Files

Most Qshell commands read a stream of characters and write a stream of characters. The Qshell interpreter is a good example. In an interactive Qshell session, the input to Qshell is the keyboard, and the output device is the display. The session is a conversation. Each time you press the Enter key, Qshell reads whatever you typed, attempts to carry out your instructions, and gives you output, even if that output is only a prompt.

But Qshell doesn't require that the input and output devices be the keyboard and display. They may be disk files instead. In Unix systems, the output device may be a printer. So far, Qshell does not support output to printer files, but maybe it will someday. The ability of a script's input and output to be assigned to different devices at run time is known as "device independence." Contrast this to an RPG program, where a file is assigned to a specific device type in the F-spec (although a file may be overridden to a different device, within limitations, at run time.)

The default input and output files are known as standard input (stdin) and standard output (stdout), respectively. There is a second default output file, standard error (stderr), to which Qshell directs error messages. Like standard output, standard error is directed to the display by default when Qshell runs interactively.

Input to Qshell Utilities

Some Qshell utilities, such as true and false, take no input at all. These are rare. Other utilities, such as date and test, take no input from files. That is not to say that such utilities take no input at all, because they can process command-line arguments.

Many Qshell utilities are designed to process a stream of input characters. Some of them accept input only from standard input. Good examples are the read (if we ignore file descriptors for now) and tr utilities.

But most Qshell utilities that get input from files can read stdin and other files. For example, grep, which looks for patterns in files, has the following syntax:

grep [options] pattern [files]

The brackets indicate optional arguments. Notice that the last argument, files, is in brackets. You may specify a list of files for grep to search, but you don't have to specify any. If you don't list any file names, grep searches stdin.

Redirection

Stdin, stdout, and stderr may be assigned to devices through a process known as "redirection." Redirection is somewhat like an override in OS/400. Under OS/400 V5R1, Qshell supports eight redirection operators, three of which are the ones most used.

The < operator redirects stdin. For example, suppose you want to display the contents of the file mydata.txt in all uppercase letters in an interactive Qshell session. You would use the tr (translate) utility to translate lowercase letters to their uppercase equivalents.

tr  [:lower:]  [:upper:]  <mydata.txt

The tr utility reads stdin, which would normally be the keyboard. The redirection operator makes tr read the disk file mydata.txt instead. The output goes to the display.

If you want the output in a disk file instead, you can use the > or >> redirection operator. The > operator replaces the contents of the disk file, while >> appends to the existing data. Both operators create the output file if necessary.

Consider the following tr (translate) command:

tr [:lower:] [:upper:] <mydata.txt >myprint.file

The capitalized version of mydata.txt replaces whatever is already stored in myprint.file if myprint.file already exists. If myprint.file does not exist, it is created.

If you use the > operator but no output is produced, the file is cleared. You can use this technique with the null utility, which is represented by a single colon (:), to clear a file.

 
:>mydata.txt

Keep in mind that Qshell commands can access database files. Suppose you have an RPG IV program that you converted from RPG III with the CVTRPGSRC command. It's all in uppercase, but you'd like to make it mostly lowercase. You could convert it to lowercase with a command like this:

tr [:upper:] [:lower:] \
   </qsys.lib/mylib.lib/qrpglesrc.file/mypgm.mbr \
   >/qsys.lib/mylib.lib/qrpglesrc.file/mypgm2.mbr 

Then you'd go back and capitalize the necessary parts by hand.

The backslashes (\) that end the first two lines in this example tell Qshell that the command is not finished yet. Normally you would key the entire command before pressing Enter. Since I had to break the command to fit it on this page, I decided to use continuation characters in this example.

To redirect stderr, put a 2 in front of the redirection operator.

cp  $var1  $var2   2>serr.txt

If the copy fails, the error message will be sent to the file serr.txt. If serr.txt does not exist, it is created. If it does exist, its contents are replaced with error messages.

Pipes and Pipelines

A pipe channels the output of one command to the input of another. It is represented by the vertical bar ( | ).

In the following example, the ls and tr commands are connected by a pipe.

ls -l m* | tr [:lower:] [:upper:]

The ls utility produces a directory listing of all files whose names begin with a lowercase letter m, but the directory listing is not sent to the display. Rather, it is sent to tr, which reads the directory listing and converts all lowercase letters to uppercase. Since the output of tr is not redirected with a redirection operator or pipe, the output is sent to the display (in an interactive Qshell session.)

Pipes can be combined with redirection operators. Here's the last example again, but this time the output is directed to the file mylist.txt:

ls -l m* | tr [:lower:] [:upper:] >mylist.txt

It's typical in Unix environments to string a series of commands together using pipelines. Here's an example:

liblist | grep "CUR$" | cut -c 1-10 | sed 's/ //g' | dsplibdir.qsh

Here's how it works.

The liblist utility writes the library list to stdout.

QSYS        SYS
QUSRSYS     SYS
QHLPSYS     SYS
QSHELL      PRD
SMITHS      CUR
QTEMP       USR
SMITHO      USR
SMITHS      USR
SMITHD      USR
CURTISLIB   USR

The grep utility reads this data and selects only the records that end with CUR. It looks for CUR at the end of the record because of the dollar ($) sign at the end of the search string. Grep extracts one record, the one with the name of the current library. Notice that the last record, CURTISLIB, is not selected, even though it contains the string CUR.

SMITHS      CUR

The cut utility reads all the records (all one of them, in this case) and extracts the data in columns 1 through 10. It writes one record, containing the name of the current library with trailing blanks, to stdout.

The stream editor, sed, replaces all blanks with nothing (null values). It writes one record, containing the name of the current library without trailing blanks, to stdout.

SMITHS

The output is sent to Qshell script dsplibdir.qsh, which displays the contents of a library in directory format. Here is the script:

#! /bin/qsh                   
while read libname                  
   do
   ls /qsys.lib/$libname.lib 
   done

The result of the long pipeline is a listing of the contents of the current library in directory format.

Notice that script dsplibdir.qsh is coded to read all the records in the input stream. In this case there will be at most one library to be listed. There may be none. But dsplibdir.qsh was written to process the entire input stream, as is typical of Unix scripts.

Redirection Operators and Pipes

It is common for beginning shell users to confuse redirection operators and pipes. There is a great difference between the following two commands:

myscript.qsh > somefile
myscript.qsh | somefile

In the first line, the output of myscript.qsh will be stored in somefile. In the second command, the output of myscript.qsh is to be read by a process (a program or shell script) called somefile. Since files in Unix systems may contain any type of data, including programs, using an output redirection operator instead of a pipe before a program or script destroys or corrupts the program or script.

It will help you to use these properly if you always look at the filename to which stdout is being redirected. If the file is a program or script, use a pipe. If it is a data file, use a redirection operator.

It will also help if you think of a pipe operator as a symbol for a temporary file. Imagine that the command preceding the pipe symbol writes to a temporary file and the command following the pipe symbol reads the temporary file.

Tee

You may decide, perhaps for debugging purposes, that you would like to see the data that travels through a pipe. If so, use the tee utility, which reads stdin and writes to stdout and one or more other files.

Here is the long pipeline example again, with an added tee command (shown in red).

liblist | tee tee.out  | grep "CUR$" | \
cut -c 1-10 | sed 's/ //g' | dsplibdir.qsh

The tee utility passes the output of liblist along to grep, as before, but at the same time makes a copy in the file tee.out. After the pipeline finishes, use cat or some other program to view tee.out and determine what input grep was given.

Stream I/O and the Philosophy of Scripting

As a rule, scripts should not be written in such a way that they converse with the user. For instance, suppose you want the user to enter his name when a script begins to run. You could use an echo and read in the script, like this:

echo "Please enter your name:"
read name

But this degree of interactivity would make it difficult or impossible to use the script in a pipeline. It is better to provide a way to specify a command-line option.

myscript.qsh -n 'Joe Smith'

In this case, the programmer writes the script so that it interprets the parameter following the -n option as a user's name.

if [ "$1" ] && [ "$2" ] && [ $1 = "-n" ] 
   then name=$2                         
   else name=$LOGNAME                   
fi                                     

If the first and second positional parameters are defined and the first parameter is -n, the second parameter provides the value for the variable name; otherwise, name gets its value from the Qshell predefined variable LOGNAME, which is the user who is running the shell. There is no need to use echo and read statements to include the user's name.

Working with Other Files

You can make a Qshell script use other files in addition to stdin, stdout, and stderr. Identify each file with a file descriptor, which is a single-digit integer number. File descriptors 0, 1, and 2 are already associated with stdin, stdout, and stderr, respectively; so you will need to use numbers 3 through 9 for other files.

To open or close a file, use the exec utility.

exec 3<mydata.txt 

Qshell opens the file mydata.txt for input with file descriptor 3.

exec 3<&-

The &- sequence tells Qshell to close the file assigned to file descriptor 3.

The following script works similarly to the cat utility. It first reads stdin, then reads any files listed as input parameters. Notice the two exec utilities, which open and close files, and the -u parameter of the read utility. The u (unit) switch tells Qshell to read from the file opened under file descriptor 3.

#! /bin/qsh                    
# read all of stdin            
while read line                
  do echo Stdin: $line; done   
# read files listed in parms   
while [ "$1" ]                 
  do                           
     exec 3<$1 # open file     
     while read -u 3 line      
       do echo $1: $line; done 
     exec 3<&- # close file    
     shift # next file         
  done                         

Overriding Redirection

You can temporarily override stdout on a line-by-line basis. Read the following lines of code:

echo "Something" # this goes to stdout
if [ -z "$1" ] 
   then date >>errorlog.txt 
        echo "Script: $0" >> errorlog.txt 
        echo "Parm 1 missing, default used."  >> errorlog.txt
fi
while read line       
  do
#    ... more lines here
    echo $whatever # this goes to stdout
  done

Notice that the first and last echo commands write to stdout. The output from those commands will go to the display, to a file, or into a pipe, depending on how stdout is directed. Now take a look at the if block. Three commands--date and two echo commands--do not write to stdout, but append to errorlog.txt.

exec 3<this.article;exec 3<&-

Stream I/O cannot take the place of database in a business environment, but it's not intended to. I am using Qshell utilities more and more to process source physical file members. Rather than looking at Qshell as some sort of competitor to OS/400, I consider it a new set of tools that is available on every iSeries machine with which I work.

References