Seven Places to Fix One Bug
Last time I built a fleet-wide
go mod why — build-time SBOMs loaded into a graph, so I could ask where an
interesting dependency enters a fleet and along which paths. Pointed at
gopkg.in/yaml.v3 and
CVE-2022-28948, it gave a clean
answer:
repos_naming_it, repos_reaching_it, repos_actually_importing_it
8, 7, 1Seven repositories affected. One — platform-telemetry — actually contains the
import statement.
Which raises the question I assumed had an obvious answer: where do I fix it?
I assumed “at the source, obviously.” I was wrong, in an interesting way.
Minimal version selection changes the answer
Go doesn’t resolve versions with a solver. It uses minimal version selection: for each module, take the highest version that anything in the build graph requires. Not the newest published — the highest one somebody actually asked for. Deterministic, boring, and it has a consequence that matters enormously here.
Since Go 1.17’s module pruning writes every transitive dependency into every
go.mod as // indirect, all seven affected repos already name yaml.v3 in their
own manifest. And a version you name in your own manifest is a version you can
raise. If svc-checkout says v3.0.1, then v3.0.1 wins — regardless of what
platform-telemetry says.
A leaf repo can upgrade a transitive dependency without the library in the middle cooperating, or even knowing about it. Here it is happening:
$ cd fleet/svc-checkout
$ go list -m gopkg.in/yaml.v3
gopkg.in/yaml.v3 v3.0.0 # inherited from platform-telemetry
$ go get gopkg.in/yaml.v3@v3.0.1
$ go list -m gopkg.in/yaml.v3
gopkg.in/yaml.v3 v3.0.1 # patched
$ grep yaml ../platform-telemetry/go.mod
require gopkg.in/yaml.v3 v3.0.0 # never touched
$ go build ./... && echo OK
OKI want to be blunt rather than hedge, because if you’ve ever merged a Dependabot PR
you already know this: the seven-repo fix works. Seven pull requests, each
raising one // indirect line, and the CVE is gone from all seven builds. No
coordination. No waiting on the platform team. No library release.
So the “naive” answer — seven places — is correct. That isn’t the problem.
The problem is that it doesn’t stay fixed
Run that play as a fleet practice and it stops being seven PRs. It’s seven PRs per advisory, times your CVE rate, times your repo count. Seven review queues, seven deploy windows, seven teams to chase. And when it lands:
- every one of those repos now carries a hand-maintained version override that
future upgrades and
go mod tidyruns will churn against platform-telemetrystill declaresv3.0.0, so the next repo anyone creates inherits the vulnerable version on day one- if only five of the seven land, the fleet sits in a mixed state you can’t assert compliance over
It’s a patch applied at seven leaves instead of at the root, and the root still says what it always said. The fix works; it just has no memory.
That’s the annoying case. There’s also a case where the seven-PR fix isn’t available at all.
Two shapes of remediation
Class A — a version bump within the same module path. v3.0.0 → v3.0.1. Any
of the seven can do it alone. MVS does the rest.
Class B — a source change. Dropping the dependency, a major-version migration
(gopkg.in/yaml.v3 → yaml.v4 is a different module path, not a version bump),
or an API break the calling code has to absorb.
In class B, none of the seven can do anything. The import statement lives in
platform-telemetry’s source, and only platform-telemetry can change it. The
remediation surface collapses from seven to one.
Same CVE, same seven repos, completely different plan — and the deciding factor is a
property of the fix, not of your dependency graph. Nothing in any go.mod tells
you which one you’re in. That’s a judgement call about the upgrade itself, and it’s
the first thing worth establishing when an advisory lands.
When it’s class B, order stops being optional
For class A, the seven PRs are independent and can land in any order. For class B they aren’t independent at all, because the fix has to propagate outward from the one repo that can make it.
That’s a topological question, and with the graph loaded it’s a query:
wave, repo_count, release_together
1, 1, [platform-telemetry]
2, 2, [platform-authz, platform-httpkit]Telemetry ships first because nothing blocks it. httpkit and authz follow — and
they can go concurrently, since neither depends on the other. Then the services pick
up the new middle layer.
Get that order wrong and the cost is concrete rather than theoretical. Ship httpkit
before telemetry and you release httpkit twice: once for whatever you thought
you were fixing, again to pick up the telemetry that actually fixed it. Two of the
three internal libraries here sit above telemetry, so any ordering that doesn’t start
at the bottom wastes releases.
One subtlety worth flagging, since it nearly gave me a wrong answer: you cannot read
this ordering off the SBOM’s requirement graph. Pruning means platform-httpkit’s
go.mod lists yaml.v3 // indirect, so the SBOM contains a direct httpkit→yaml
edge and every module looks one hop from the target. Reachability survives that
distortion; depth does not. The waves above come from edges tagged as genuinely
declared, recovered by parsing go.mod — which is the trap from the last
post, showing up again in a new
disguise.
So what do you actually do
Not “always fix at the source.” That’s too clean, and it’s wrong for the urgent case.
For a class-A advisory that’s actively burning — drop-in compatible, no API change — the seven PRs are fine, and they’re faster than waiting on a library release plus two waves of propagation. Ship them. Stop the bleeding.
Then fix the source, so it stays fixed. Emergency remediation and durable remediation are different jobs with different shapes, and conflating them is how you end up either sitting on a live CVE waiting for a library release, or patching the same seven leaves every quarter forever.
The graph doesn’t make that decision. What it does is make both options legible: it tells you the seven and the one, and the fact that those are different numbers is the whole point. Without it you’re picking a remediation strategy from a list of package names, which is a bit like choosing a route from a list of street names.
The fleet, the CI step, the loader and the queries are at
husobee/sbom-fleet. The go get transcript
above is reproducible against it — v3.0.0 really is vulnerable, v3.0.1 really is
the fix, and platform-telemetry really doesn’t change.