Deduplicate Lines in Order With awk

Drop repeated lines but keep the first occurrence where it appeared.

LinuxtextscriptingcliLow risk

Run awk '!seen[$0]++' file to print each unique line once, in order.

01Command

Command

$ awk '!seen[$0]++' file.txt

02What it does

sort -u reorders data and uniq only removes adjacent duplicates, but this awk idiom remembers every line seen and prints the first occurrence in place. That order preservation matters for logs and generated lists, and it is trivial to script.

03Risk

RiskLow risk

RequirementsPreinstalled on Linux (GNU awk)

!Warning: The whole file's unique lines are held in memory; extremely high-cardinality inputs need a database or sort instead.

04Undo

✓No undo needed — this command does not change persistent state.

05Sources

gawk.1.en.html manual page (Debian bookworm)
Debian · technical · 2026-09-20
Evidence: Documents associative arrays, used here to remember seen lines.

Last verified: 2026-10-13 · Category: Developer

06Related solutions

Create and Apply Patches with diff -u and patch

Linuxdiffpatchtext

Generate a unified diff between two files, apply it to another copy with patch, and reverse it with patch -R.

Low risk ✓ 2026-09-17