Diffing the Fine Print
You have agreed to thousands of pages of contracts you never read. So have I. We clicked the button:
Nobody reads it. The button might as well say Agree and Do Not Install The Thing You Opened This Window To Install. And the companies writing these know it, which is why the interesting move isn’t the first version you accept — it’s the next one. The terms you agreed to in 2021 are not the terms you’re living under today, and no one sent you a changelog.
So I wanted a changelog. Point it at version A and version B of an agreement, and tell me — in plain English — what changed and who it favors. That’s it. That’s the tool.
It turned out to be a great little lesson in the difference between a thing that works on your test data and a thing that works on the world.
Why diff is useless here
The naive approach is diff old.txt new.txt. Try it on two versions of a real
contract and you get a wall of red and green that means nothing, because legal
documents don’t change the way code changes:
- A paragraph rewraps and every line reads as modified.
- Someone inserts section 4 and every following section renumbers. Diff screams; nothing substantive happened.
- An arbitration clause moves from §12 to §3.
The unit of a contract isn’t the line. It’s the clause. I wanted a tool that aligns clauses across versions — matching them by content, so a clause that keeps its wording survives renumbering and reordering — and then shows a word-level redline only where the meaning actually moved.
The shape: deterministic core, judgment on top
I split it clean down the middle:
old ─┐
├─► extract ─► normalize ─► segment ─► align ─► redline ─► JSON ─► Claude
new ─┘ ───────── deterministic Go, no LLM ───────── the semantic layer
A small Go binary does the boring, reproducible, testable half: pull text out of
.html/.txt, canonicalize it, split it into clauses, align the two versions,
and emit a word-level redline as JSON. Then a Claude skill reads that JSON and
does the half a regex can’t: tag each change by category, decide whether it favors
you or the vendor, rate severity, and write it up. Determinism for the mechanical
work; a language model for the judgment. Neither pretending to be the other.
On my own hand-written test fixtures, it was beautiful. Then I fed it a real one.
Reality, exhibit A: the iTunes EULA
I grabbed Apple’s Standard EULA — the “Licensed Application End User License
Agreement” every App Store app ships under — and, to get an old version, the
Wayback Machine. (Its CDX API lists every capture; add
id_ to a timestamp and you get the raw archived HTML.) Two versions, four years
apart. I ran the tool.
It found two clauses. The entire agreement, minus a scrap of website chrome, collapsed into one giant blob.
The problem: my segmenter knew how to spot 1., Section 9, and ALL-CAPS
banners. Apple’s EULA uses lettered sections with the heading and body on the
same line — a. Scope of License: Licensor grants…. My clean little fixtures had
never contained such a thing. So I taught it lettered and inline-colon headings,
and it jumped to a proper eleven clauses.
Then the diff told me something I didn’t expect. The legal body was identical.
Every apparent change was the website around the contract — the 2017 footer
versus today’s, iTunes Gift Cards becoming Apple Gift Card, © 2017 becoming
©. So I taught the extractor to isolate the page’s <main> content and throw
away nav, header, and footer. Clean text in, clean text out.
And the answer held, and got sharper: Apple’s Standard EULA has not changed one word from December 2017 to today. I sampled the archive across nine years. Every version I could cleanly extract hashed to the same bytes. The Wayback Machine has 630 distinct captures of that page. The legal text in all 630 is the same document. What changed 630 times was the website; the contract underneath sat perfectly still.
I built a tool to watch a EULA evolve over a decade, and its first real subject never moved. Which is its own kind of finding.
Reality, exhibit B: the one that actually moved
So I pointed it somewhere with a pulse: Apple’s Apple Cash & Apple Payments Inc. privacy notice, first archived version (November 2021) against the live page today.
It found one clause again. Different reason this time: privacy notices don’t number anything. They use bare Title-Case heading lines — “Setting Up Apple Cash”, “Verifying Your Identity” — that carry no number, no letter, no caps signal. A third heading style, a third patch. (This one needed a guard so it wouldn’t mistake an ordinary capitalized sentence for a heading.) After that: ten clauses, and a diff with actual teeth.
This notice morphed. Eight of ten sections rewritten in under five years, and the direction was all one way — more data, more purposes. A few of the things the tool surfaced, each quoted straight from the redline:
- A new purpose appeared. The 2021 version used your Apple Cash data to run the service and prevent fraud. Today: “Apple may also use this information to improve other Apple products and services, for marketing, and for fraud and security purposes.” Marketing is new.
- Location entered the fraud pipeline: “If you have enabled Location Services for Wallet, your device will also evaluate information about your device’s location… The output of the assessments, but not the underlying data, will be sent to Apple.”
- On-device behavioral risk scoring, “shared identifiably with Apple Payments Inc.”, that simply wasn’t there before.
None of that is in a press release. It’s in a Title-Case section three quarters of the way down a page you’ve never opened. That’s the whole point of the tool.
The part I got wrong, and the fix that mattered
Notice the pattern: three real documents, three times the deterministic layer quietly fell on its face. Lettered headings. Website chrome. Title-Case headings. Each time it didn’t error — it cheerfully returned one clause and a confident, useless answer.
That last bit is the real bug. A tool that fails loudly is annoying. A tool that fails silently — that hands you a garbage single-clause “diff” wearing the same face as a good one — is dangerous, because you’ll believe it. My deterministic normalization and segmentation were never going to be sufficient for arbitrary real-world contracts. No pile of regexes is. The mistake was letting them pretend they were.
The fix isn’t more heuristics. It’s honesty. The tool now scores its own segmentation and reports a confidence level:
1 -> 1 clauses | +0 added -0 removed ~1 modified >0 moved =0 unchanged
! segmentation confidence: LOW
- old: under-segmented — one clause holds most of the text; treat clause diff as unreliable
- new: under-segmented — one clause holds most of the text; treat clause diff as unreliable
When confidence isn’t high — when the whole document collapses into one clause —
the skill is told to stop trusting the clause diff and fall back: pull the
cleaned text of each version (eula-diff --text) and re-read it whole, by meaning,
rather than present a structural diff it knows is bogus. The deterministic layer’s
job isn’t to be right about everything. It’s to be cheap, reproducible, and never
quietly hide signal. Now it admits when it can’t parse, and something smarter
takes over.
I also stopped hand-maintaining a list of typographic gremlins and let Unicode do it — NFKC normalization folds ligatures, full-width characters, and exotic spaces, and a pass strips the invisible format characters (soft hyphens, zero-width spaces, byte-order marks) that love to masquerade as changes in text pulled from PDFs and web pages.
What it is now
A dependency-light Go binary that turns two versions of an agreement into a clause-aligned, word-level redline with a confidence score — and a Claude skill that reads that redline, or falls back to the raw text when the redline can’t be trusted, and tells you in plain language what changed and who it favors. The determinism handles what determinism is good at. The model handles judgment. The confidence score is the honest seam between them.
The code is on GitHub. It is a drafting aid and a curiosity engine, not a lawyer, and it will tell you so before it tells you anything else.
None of this makes anyone read the fine print. But it means the next time the button says Agree, something can read the delta for you — and, just as important, tell you when it couldn’t.