06 Text Processing

Learning objectives

Command	Purpose	Meaning	Basic usage	Example
`sort`	Sort text lines	Sorts lexically or numerically	`sort file`	`sort names.txt`
`uniq`	Remove adjacent duplicate lines	Short for unique; usually used with `sort`	`uniq file`	`sort names.txt \| uniq`
`cut`	Extract text columns	Selects fields by character position or delimiter	`cut [options] file`	`cut -d ":" -f 1 /etc/passwd`
`awk`	Analyze text and process columns	Suitable for field-oriented text processing	`awk 'rules' file`	`awk '{print $1}' app.log`
`sed`	Stream text editor	Commonly replaces, removes, or prints lines	`sed 'expression' file`	`sed 's/old/new/g' file.txt`
`tr`	Translate or delete characters	Performs character-level transformations	`tr 'old_chars' 'new_chars'`	`printf '%s\n' abc \| tr '[:lower:]' '[:upper:]'`
`xargs`	Turn NUL-delimited input into arguments	Ordinary newlines cannot represent every legal filename; on GNU/Linux use with `-print0`/`-0`	`find ... -print0 \| xargs -0 command`	`find . -type f -name '*.log' -print0 \| xargs -0 -- wc -l`

Sort and deduplicate:

sort names.txt | uniq
sort names.txt | uniq -c

Extract the first field of a colon-separated file:

cut -d ":" -f 1 /etc/passwd

Extract the first log field:

awk '{print $1}' app.log

Replace text in output:

sed 's/error/ERROR/g' app.log

Turn lowercase into uppercase:

printf '%s\n' 'linux' | LC_ALL=C tr '[:lower:]' '[:upper:]'

Pass found files safely to another command:

find . -type f -name '*.log' -exec wc -l -- {} +
find . -type f -name '*.log' -print0 | xargs -0 -- wc -l

uniq removes only adjacent duplicate lines, so use sort before uniq in the usual case.
sed 's/old/new/g' file prints a transformed result by default; it does not modify the source file.
sed -i is not POSIX. GNU and BSD/macOS differ on -i and backup-suffix syntax. This course first writes to a new file and compares it instead of editing in place.
awk splits fields on whitespace by default; $1 is the first field and $2 is the second.
An ordinary newline pipeline cannot safely represent every legal filename. The portable first choice is find -exec ... {} +; GNU/Linux can use -print0 | xargs -0.
Sorting, case conversion, and character classes depend on locale. Set LC_ALL=C explicitly when reproducible byte order is required.

sed 's/error/ERROR/g' -- app.log > app.log.new
cmp --silent -- app.log app.log.new
cmp_rc=$?
printf 'content_changed=%s\n' "$cmp_rc"

Inspect app.log.new first, then decide whether to replace the original with a controlled mv. Do not redirect output back into the input file.

I can explain why uniq usually requires sorted input.
I can handle filenames containing spaces and newlines without bare find | xargs.
I know that sed in-place editing and locales have platform differences.