I/O Redirection and Piping

I/O Redirection

You can redirect or pipe i/o in much the way you might under cmd.exe. Here’s a simple example redirecting stdout from the word count of the famous “Hello, world” program. cat just copies from any files you tell it or, by default, from stdin to stdout.

74 E% cd hello
75 D% ls
hello.c      hello.exe
76 D% cat hello.c
#include <stdio.h>
main ()
   {
   printf("Hello, world.\n");
   }
77 D% wc hello.c >hello.wc
78 D% cat <hello.wc
        5        8       72  hello.c

(wc tells us that hello.c has 5 lines, containing 8 words, totaling 72 characters.)

If the file you write to with “>“ exists, it’s first truncated to zero length (discarding the old contents); if the file doesn’t exist, it’s created. With “<“, it’s an error if the file doesn’t exist.

Data can be appended to a file with the “>>“ operator:

79 D% echo that^'s all folks >>hello.wc
80 D% cat hello.wc
        5        8       72  hello.c
that's all folks
81 D% _

When you append with “>>“, if the file exists, data is written onto the end; if it doesn’t exist, it’s created. (The single quote character has special meaning to the shell on the command line; the special meaning is turned off by the shell’s escape character,”^”.)

noclobber

Not everyone is comfortable with letting the shell glibly toss away an existing file if you type “>“ when you meant “>>“ or lose it somewhere if you mistype an existing filename with “>>“. The noclobber variable lets you tell the shell you want this to be caught, so you can decide if this was really what you meant.

If you set noclobber, you have to type “>!” to redirect to an existing file:

81 D% set noclobber = 1
82 D% echo trash this file > hello.c
csh:  Couldn't open 'hello.c' as a redirected standard output.

Come to think of it, let’s not overwrite that file.

Similarly if you want to append to something that doesn’t already exist:

83 D% echo appended data >> newdata
csh:  Couldn't open 'newdata' as a redirected standard output.
84 D% echo appended data >>! newdata
85 D% cat newdata
appended data
86 D% rm newdata

Protection Attributes

If a file has any of the special protection attributes, hidden, read-only or system, set, you cannot overwrite it by redirecting i/o to it. Even when you type “!”, you still can’t. Before you can redirect to it, you must clear all these attribute bits.

87 D% ls -l zork
-SHAR  Feb 23  13:16         0  zork
88 D% echo new zork data >! zork
csh:  Couldn't open 'zork' as a redirected standard output.
89 D% chmod -R zork
90 D% echo new zork data >! zork
csh:  Couldn't open 'zork' as a redirected standard output.
91 D% chmod -SH zork
92 D% ls -l zork
---A-  Feb 23  13:16         0  zork
93 D% echo new zork data > zork
94 D% _

Stdout and Stderr

Redirecting both stdout and stderr together is done by adding an ampersand. For example, using echo’s “-2” option to deliberately write to stderr and parentheses for a simple grouping:

94 D% (echo -2 error; echo standard) > zork
error
95 D% cat zork
standard
96 D% (echo -2 error; echo standard) >& zork
97 D% cat zork
error
standard
98 D% _

Separately redirecting stderr and stdout to different files is a little tricky: first you redirect them both, then redirect stdout by itself. Here’s an example running the C compiler with stdout to log and stderr going to errors.

98 D% cl hello.c >& errors > log

You can type as many i/o redirections in a row as you like. The shell evaluates them one after another. If you redirect to a new file, then redirect to something else, the effect is just like touch’ing the file.

Pipes

Pipes are a way of connecting a series of activities together so that the output of one is read as input to the next. Each of the activities runs asynchronously and concurrently with the others. Data is passed completely in memory and is very fast.

The syntax is similar to i/o redirection in its use of the “&” character. To pipe just stdout, use “|” by itself:

99 D% ls -L | more

To pipe both stdout and stderr together, use “|&”:

100 D% cl hello\hello.c |& more

The leftmost part of the pipeline is evaluated directly by the shell’s current thread. The successive right parts are evaluated by child threads. (This is so that piping a command that lists status information on the current thread through a filter like more operates sensibly.) Each part of the pipeline can be an arbitrarily complex statement, perhaps even run in a separate window.

Pipes are much faster and more responsive than with vanilla Windows due to improved buffering and scheduling technology. A long pipeline finishes much faster. Also, when you type ^C to interrupt, it comes back immediately without a lot of nuisance messages.

Command Substitution

A particularly novel way of piping statements together is to use the output of one as command line arguments of another. This is called command substitution and you indicate it by typing backquotes, `...`, around a command.

101 D% ls +a
.              hello          zork
..             memos
102 D% echo `ls +a`
. hello zork .. memos
103 D% _

When command substitution is done, all the extra “white space” (space characters, tabs and newlines) is squeezed out. Also, any ANSI escape sequences that might have turned on highlighting or color, etc., are deleted. You just get the list of words the backquoted command wrote to stdout. In this example, the order of the files is a bit scrambled when the line ends are removed; the -1 (numeric one) single column option can fix this. (Try it again using ls +a1 inside the backquotes.)

Command substitution is especially useful anywhere you need to give a list of filenames as arguments to a command. Here’s an example using ls to give a detailed listing of itself:

103 D% whereis ls
D:\Program Files\Hamilton C shell 2003\bin\ls.exe
104 D% ls -l `whereis ls`
---A-- Jun 24  8:28      86016  d:\Program Files\Hamilton C shell 2003\Bin\ls.exe

If there are any variable substitutions inside the backquotes, they’re done by the child, not the parent. This lets you easily embed for loops and other programming constructs into the command substitution.

Inside backquotes, only the backquote character needs to be escaped to avoid having it processed by the parent thread.

Inline Data

A novel variation on i/o redirection is inline data, also called “here” documents: literal text you want the shell to feed a command as stdin. Here’s an example:

105 D% cat <<eof
106 D? (this is the inline data)
107 D? eof
(this is the inline data)
108 D% _

The “<<“ operator is followed by a string the shell is asked to look for to mark the end of the inline data. The end-of-data string can be virtually anything you like, including wildcard characters, dollar signs, etc.; their normal meaning is turned off and they’re treated as ordinary literal characters. Only quote or escape characters have any special meaning, which is to turn off substitutions in the inline text (as we’ll discuss in a moment). Continuation lines as the shell collects the inline data get a different prompt, controlled by the prompt2 variable. Once the data has been collected in memory, it’s written through a pipe to the command.

One very convenient use of inline data is when you want to quickly search for any one of a number of important words in a large library. E.g., to scan for some specific strings in a set of C files:

108 D% fgrep -ns <<xxx ~\sh\*.c
109 D? Open
110 D? Close
111 D? Read
112 D? Write
113 D? xxx
:  search results

In situations where the inline data is being created inside a larger structure, the data is assumed to start on the first line following a break between statements. For example, inside a for loop:

114 D% for i = 1 to 3 do
115 D? cat <<eof; echo i = $i
116 D? (this is the inline data)
117 D? eof
118 D? end
(this is the inline data)_
i = 1
(this is the inline data)_
i = 2
(this is the inline data)_
i = 3

If you want to put several inline i/o redirections on the same line, type the associated inline data sections, each with its own terminating string, in the same left-to-right order in which they appeared.

So far, we’ve just shown examples involving static text. But it’s also possible to ask the shell to do command and variable substitutions on the inline text:

119 D% cat << ***
120 D? The ^$home directory is $home.
121 D? Today's date is `date`.
122 D? ***
The $home directory is d:\Nicki
Today's date is Wed Jul 2 2003 9:04:11.46.
123 D% _

Although substitutions and escape characters inside the here document are processed, quotes (both single and double) are not.

The C shell implements here documents by spawning a child thread to do any substitutions and write the results into a pipe feeding the current thread as it continues to evaluate the statement. If the here document contains references to shared variables, they’ll be evaluated by that other thread. And unless they’re local variables, the values will not be snapshotted when the here document thread is created. If the current thread (or any other thread) continues to make changes to a variable after the here document thread is spawned but before it evaluates the variable, the here document will contain the new, not the old value.

Command and variable substitution and escape processing inside a here document is turned off if any part of the end-of-data string following the << is quoted (with single, double or backquotes) or escaped:

123 D% cat <<^***
124 D? The ^$home directory is $home.
125 D? Today's date is `date`.
126 D? ***
The ^$home directory is $home
Today's date is `date`.
127 D% _

Inline Data in Scripts

Inline data can be especially useful if you’re writing a script file or passing commands to the shell through a pipe. In either of these cases, the low-level ReadFiles to the Windows kernel cannot be depended on to stop at the end of a line because pipes and files are considered block-oriented rather than line-oriented like the keyboard. If too many characters are read, there’s no simple way to back up. For this reason, it’s not realistic to write a script where a child process is supposed to inherit stdin pointed into the script file. In a script file, this is not reliable:

:
:
csh
echo hello
exit
:
:

The file descriptor the child process inherits will likely not be pointing at the “echo hello”; when it exits, the parent will likely not find it pointed just past the “exit”. This type of script should be written as:

:
:
csh <<eof
echo hello
exit
eof
:
: