sed & awk

sed & awk are tools used by users, programmers and administrators, sed, so called because it is a stream editor, perfect for applying a series of edits to a number of files, awk, named after its developers, Aho, Weinberger and Kernighan, is a programming language that permits easy manipulation of structured data and the generation of formatted reports. This book covers the original awk and the new awk (nawk) as well as the GNU project's gawk.

The book begins with an overview and a tutorial that demonstrate a progression in functionality from grep to sed to awk. All three programs share a similar command-line syntax, accepting user instructions in the form of a script. Because all three programs use UNIX regular expressions, an entire chapter is devoted to understanding UNIX regular expression syntax.

The first chapter on sed describes how to write a sed script using only a few sed commands. The chapters that follows divide the sed command set into basic and advanced commands. The basic commands are ones that parallel manual editing actions while the advanced commands introduce simple programming constructs. Among the advanced commands are those that manipulate the hold space, a set-aside temporary buffer.

The first of five chapters on awk introduces the primary features of the language and how to write simple scripts. The next chapter looks at common programming constructs. The third chapter explains how to use awk's built-in functions as well as how to write user-defined functions that are available with nawk. The next chapter also covers advanced topics and includes a discussion of awkcc, and awk -to-C translator. The last chapter covers the development of an application that takes awk to its limits, the processing of an index.

Finally, there is a miscellany of user-contributed sed and awk scripts that demonstrate a wide range of scripting styles and techniques.

1. Power Tools for Editing
• May You Solve Interesting Problems
• A Stream Editor
• A Pattern-matching Programming Language
• Three Hurdles to Mastering Sed and Awk
2. Understanding Basic Operations
• Awk, by Sed and Grep, out of Ed
• Command-line Syntax
• Scripting
• Sample Mailing List
• Using Sed
• Specifying Simple Instructions
• Script Files
• Using Awk
• Using Sed and Awk Together
3. Understanding Regular Expression Syntax
• That's an Expression
• A Line Up of Characters
• The Ubiquitous Backslash
• A Wildcard
• Writing Regular Expressions
• Character Classes
• Repeated Occurrences of a Character
• What's the Word? Part I
• Positional Metacharacters
• A Span of Characters
• Alternative Operations
• Grouping Operations
• What's the Word? Part II
• Your Replacement is Here
• Limiting the Extent
• I Never Metacharacter I Didn't Like
4. Writing Sed Scripts
• Applying Commands in a Script
• The Pattern Space
• A Global Perspective on Addressing
• Grouping Commands
• Testing and Saving Output
• testsed
• runsed
• Four Types of Sed Scripts
• Multiple Edits to the Same File
• Making Changes Across a Set of Files
• Extracting Contents of a File
• Edits to Go
• Getting to the PromiSed Land
5. Basic Sed Commands
• About the Syntax of Sed Commands
• Comment
• Substitution
• Replacement of Metacharacters
• Delete
• Append, Insert and Change
• List
• Stripping Out Non-Printable Characters from Nroff Files
• Transform
• Print
• Print Line Number
• Next
• Reading and Writing Files
• Checking Out Reference Pages
• Quit
6. Advanced Sed Commands
• Multi-line Pattern Space
• Append Next Line
• Multi-line Delete
• Multi-line Print
• A Case for Study
• Hold That Line
• Advanced Flow Control Commands
• Branching
• The Test Command
• One More Case
• To Join the Phrase
7. Writing Scripts for Awk
• Hello World
• Awk's Programming Model
• Pattern Matching
• Describing Your Script
• Records and Fields
• Expressions
• Averaging Student Grades
• System Variables
• Working with Multi-line Records
• Balance the Checkbook
• Relational and Boolean Operators
• Getting Information About Files
• Formatted Printing
• Passing Parameters into a Script
• Information Retrieved
• Importing Shell variables into an Awk Script
• Finding a Glitch
8. Conditionals, Loops, and Arrays
• Conditional Statements
• Conditional Operator (nawk)
• Looping
• While Loop
• Do Loop (nawk)
• For Loop
• Deriving Factorials
• Other Statements That Affect Flow Control
• Arrays
• Associative Arrays
• Testing for Membership in an Array (Nawk)
• A Glossary Lookup Script
• Using split to Create Arrays
• Making Conversions
• Deleting Elements of an Array (Nawk)
• An Acronym Processor
• Multi-dimensional Arrays (Nawk)
• System Variables That Are Arrays (Nawk)
• An Array of Command Line Parameters
• An Array of Environment Variables
9. Functions
• Arithmetic Functions
• Trigonometric Functions
• Integer Functions
• Random Number Generation (nawk)
• Pick'em
• String Functions
• Substrings
• String Length
• Substitution Functions (nawk)
• Match Function (nawk))
• Writing Your Own Functions (nawk)
• Writing a Sort Function
• Maintaining a Function Library
• Another Sorted Example
10. The Bottom Drawer
• The getline Function
• Reading Input from Files (Nawk)
• Assigning the Input to a Variable (nawk)
• Reading Input from a Pipe
• The Close Statement (nawk)
• The system() Function (nawk)
• A Menu-based Command Generator
• Directing Output to Files and Pipes
• Directing Output to a Pipe (Nawk)
• Special Filenames
• Working with Multiple Files
• Generating Columnar Reports
• Gawk Talk
• Debugging
• Make a Copy
• Before and After Photos
• Finding Out Where the Problem Is
• Commenting Out Loud
• Slash and Burn
• Getting Defensive About Your Script
• Limitations
• Invoking Awk Using the #! Syntax
• A Trick for Mac Text Files
• Using awkcc
11. Full-featured Applications
• An interactive Spell Checker
• BEGIN Procedure
• Main Procedure
• End Procedure
• Supporting Functions
• The spellcheck Shell Script
• Generating a Formatted Index
• The Masterindex Program
• Standardizing Input
• Sorting the Entries
• Handling Page Numbers
• Mergin Entries with the Same Keys
• Formatting the Index
• Spare Details of the Masterindex Program
• How to Hide a Special Character
• Rotating Two Parts
• Finding a Replacement
• A Function For Reporting Errors
• Handling See Also Entries
• Alternative Ways to Sort
12. A Miscellany of Scripts

1. Sed Quick Reference
• Command Line Syntax
• Syntax of Sed Commands
• Pattern Addressing
• Sed Regular Expression Metacharacters
• Sed Command Summary
2. Awk Quick Reference
• Command Line Syntax
• Shell Wrapper for invoking Awk
• Records and Fields
• Format of a Script
• Patterns
• Regular Expressions
• Expressions
• Statements and Functions
• Awk Command Summary
3. Supplement for Chapter 11
• Full Listing of spellcheck.awk
• Listing of Masterindex Shell Script
• Documentation for Masterindex
• Background Details
• Coding Index Entries
• Output Format
• Compiling a Master Index

A very good book about the subject.