Remove Blank Lines from a Stream File

Published: October 4, 2006

Hey, Ted:

Working with IFS files, especially those we get from other systems, greatly increases my appreciation of physical files. Opening a stream file is like opening a Christmas present--we never know what we'll get. Anyway, one source typically sends us stream files that often have blank lines in them, which messes up our process. Is there any way to delete those blank lines?

--Ray

Well, Ray, I don't know of a way to delete them, but if you're willing to copy the stream file to another stream file and omit the blank lines in the process, you can take advantage of the sed stream editor in Qshell. The following command reads data file FileWithBlankLines and writes to FileWithoutBlankLines.

sed  '/^ *$/d'  FileWithBlankLines > FileWithoutBlankLines

Let me run through the expression. The caret (^) means that the pattern must start at the beginning of a line. The dollar sign ($) means that pattern must end at the end of a line. The asterisk means zero or more occurrences of the preceding character, which is a blank. So, the expression means there must be zero or more blanks from the beginning through the end of the line. The lowercase d indicates that lines that match the pattern are to be deleted from the output stream. The result is that you get a stream file without blank lines.

The cryptic sed expression reminds me of the comments I received recently from faithful and perspicacious reader Shalom Carmel:

"I like tools like sed and awk because they give me greater flexibility, despite their awk-wardness and sed-itionary character. . . . I was sed-uced by the power of this tool."

--Ted