mats/fancy-dns
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Make the API response contract a test; fix what it found

Browse Source 
The console's TypeScript types are hand-written and apiFetch ends in
`JSON.parse(text) as T`, so nothing in the toolchain compared them with the
Go structs. A field that lost its json tag, or a response a struct never had,
showed up as an empty table in a browser and nowhere else: the Go suite was
green and the console was wrong. internal/app/contract_test.go now asserts the
exact JSON key set of every admin endpoint, driven through the real router
with the real hook implementations, plus a test that drives the two write
endpoints with the exact payload web/src builds.

The drift it found, all four fixed here:

blocklist.Source and SourceMeta had no json tags at all, so GET /api/blocklists
sent Go field names ("URL"/"Format") where the console expects snake_case: the
Sources table rendered blank rows and its per-row toggle posted
`name: undefined`. Tagged to match ParseStats.

acl.GroupSummary was flat where the console reads a nested `match` object — the
same shape config.Group has — so every group card read "matches nothing". The
four match fields are nested now, under a new acl.Match.

PUT /api/clients answered 400 `json: unknown field "ptr_suffixes"` to the
console's payload, because config.Group and config.Match carried yaml tags only
and the decoder runs with DisallowUnknownFields. Both structs have json tags
now; the same applies to config.UpstreamServer and UpstreamBootstrap.

PUT /api/upstreams answered 400 `json: unknown field "health"`, because the
console round-trips the whole server object it read, runtime health fields
included. They are accepted and ignored, because the server owns them, and they
are `yaml:"-"` so settings.yaml does not gain runtime state.

Also fixed, each with a test:

A duration on the wire was a nanosecond count. config.Duration had a
MarshalYAML but no MarshalJSON, so bootstraps[].refresh arrived as 30000000000
while the console typed the field as a string, and healthcheck_interval was a
float of seconds in one place and a string in another. Durations on the wire
are duration strings everywhere now — the same spelling config.yaml uses — and a
bare JSON number is still accepted and means seconds, not nanoseconds.
TestDurationJSONForm used to pin the old behaviour and now pins the new one,
saying in its comment that the old behaviour was the bug.

A log row's key set depended on the row. api.LogEntry was omitempty on
client_group, response, upstream and list, so a query with no ACL group
arrived without the key and the console rendered "undefined" where the column
should be empty. The contract test can only pin one shape if the server emits
one, so nothing on that struct is omitempty any more.

A dead peer stayed in the cluster peer list forever. Cluster.seen was only ever
added to and deleted from on an explicit `bye`, so a node that crashed — or
whose Valkey connection died — was reported as a live peer for the life of the
process, and its stale blocklist digest was flagged as divergence forever.
Peers() now reaps peers unheard for peer_ttl*4 and stamps each state with the
receiver's own clock, so clock skew between two machines can neither reap a
healthy peer nor keep a dead one. Discover() ages out the same way on the
publisher's own last_seen stamp, because the keyed Valkey record outlives a
crash: a live node keeps refreshing the key's TTL, so a dead node's field stayed
readable for the life of the fleet.

peer_ttl: 0 would have crashed the process. time.NewTicker(PeerTTL/2) panics on
a non-positive period and `EXPIRE key 0` deletes the heartbeat key, so a config
that left peer_ttl unset would have taken the process down at Start, or made
every instance believe it is alone in the fleet. Both go through peerTTL() and
peerDeadline(), which fall back to the default.

Two cluster tests raced on the wire, not on the product, and one was
environment-dependent rather than wrong. A node that joined after another's
heartbeat was in flight hears about it on the next tick, so "b does not see a"
read immediately after "a sees b" is a coin flip; the divergence test waited for
a digest on the wrong node's view of the fleet; and 127.0.0.1:1 is only
unbindable outside a container that sets
net.ipv4.ip_unprivileged_port_start=0, where it binds and the partial-bind test
fails with both transports up. That test now occupies a port with a live
listener in the test process, which is a deterministic EADDRINUSE on every
machine and every uid.

internal/server/counters_test.go covers the counter-key layer that no other
test touched: the "<transport>.<field>" keyspace, that the aggregate folds only
the known fields, and that the snapshot is safe while counters move.

Two pinned behaviours changed on purpose, each with a test that says so:

An RPZ record did not cover the name it names. rpzRule gave a bare owner name
the subdomain form (*.name), so analytics.example.org. in a policy zone blocked
cdn.analytics.example.org. and left analytics.example.org. resolving normally —
and an rpz-passthru. record did not allowlist its own domain either. RPZ
semantics say the record applies to the domain and everything under it, so a
bare owner now takes the apex form **.name; an owner written with an explicit *.
is still strict, so a zone that wants subdomain-only behaviour can still ask for
it. This changes which queries get blocked for an existing policy zone, which is
why it wanted a decision. TestExampleRPZZoneCoversTheApexItNames pins both
halves and the example zone now says which half it relies on.

metrics.OutcomeLocal had no producer. The engine reported a local-zone answer as
qlog.ActionAllow, which outcomeFor mapped to OutcomeUpstream, so every locally
answered query was counted as an upstream one and the outcome="local" series was
never published — the dashboard's local column was permanently empty.
Response.Local is now set on the local-zone path and outcomeFor consults it
first. That moves published counters, so the README's metrics section says so
too.
This commit is contained in:
developer committed 2026-09-27 18:42:59 +00:00
1 parent e8c5b414d9
commit 9881d2a095
23 files changed
+1302 -242
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

STATUS.md
+110 -50
View File
@@ -1,9 +1,11 @@
# fancy-dns — development status and continuation plan
Last updated after the round that finished the test coverage of every package
that has logic worth covering, fixed two live bugs and pinned the open
questions. `AGENTS.md` holds the decisions, invariants and working rules; this
file holds state and the plan.
Last updated after the round that made the **API response contract a test**
(`internal/app/contract_test.go` pins the exact JSON key set of every admin
endpoint, and fixed the four drift items it exposed), decided the two pinned
open questions, and finished the cluster package's peer-reaping gap. `AGENTS.md`
holds the decisions, invariants and working rules; this file holds state and the
plan.
## Goal
@@ -79,11 +81,11 @@ Cache is a single shard-local lookup; only a miss touches the network.
|---|---|---|
| `internal/acl` | 26 | group compilation, matching, first-group-wins |
| `internal/api` | 55 (156 cases) | the whole route table, three auth modes, CORS, rate limiting, SSE, settings, error branches, pinned limitations |
| `internal/app` | 25 | wiring, hooks, the API response contract, integration |
| `internal/app` | 25 + the contract suite | wiring, hooks, **the API response contract**, integration |
| `internal/blocklist` | 28 | four parsers, the set, the manager, the shipped example zone |
| `internal/cache` | 27 | LRU, TTL policy, serve-stale, bbolt round-trip |
| `internal/cluster` | in progress | Valkey lease, pub/sub, divergence |
| `internal/config` | 22 | Duration/Size, `${VAR}`, unknown keys, the shipped example configs |
| `internal/cluster` | 30 | Valkey lease, pub/sub, divergence, dead-peer reaping |
| `internal/config` | 22 | Duration/Size, `${VAR}`, unknown keys, the JSON wire form, the shipped example configs |
| `internal/engine` | 26 | the pipeline end to end, plus the pins |
| `internal/localzone` | 18 | wildcards, delegation, CNAME following |
| `internal/metrics` | 10 | the metric names, which are the contract with the dashboards |
@@ -116,8 +118,86 @@ struct never had, showed up as an empty table in a browser and nowhere else.
`internal/app` now has contract tests that assert the exact JSON key set of
every admin endpoint — see the fix list.
### Decided this round (behaviour changed on purpose, each with a test)
- **An RPZ record did not cover the name it names.** `rpzRule` gave a bare
owner name the subdomain form (`*.name`), so `analytics.example.org.` in a
policy zone blocked `cdn.analytics.example.org.` and left
`analytics.example.org.` resolving normally, and an `rpz-passthru.` record
did not allowlist its own domain either. RPZ semantics say the record applies
to the domain and everything under it, so a bare owner now takes the apex
form `**.name`; an owner written with an explicit `*.` is still strict, so a
zone that wants subdomain-only behaviour can ask for it. This changes which
queries get blocked for an existing policy zone — re-read your zones if they
relied on the old behaviour. `TestExampleRPZZoneCoversTheApexItNames` pins
both halves, `TestParseRPZ` pins the compiled forms, and the example zone now
says which half it relies on.
- **`metrics.OutcomeLocal` had no producer.** The engine reported a local-zone
answer as `qlog.ActionAllow`, which `outcomeFor` mapped to
`OutcomeUpstream`, so every locally answered query was counted as an upstream
one and the dashboard's local column was permanently empty. `Response.Local`
is now set on the local-zone path and `outcomeFor` consults it first, so a
local answer publishes `outcome="local"` and is *not* counted as upstream.
This moved published counters, which is why it was a decision: the README's
metrics section and the dashboard notes are updated in the same change.
`TestLocallyAnsweredQueryIsCountedAsLocal` replaces the pin.
## Fixed this round
- **A dead peer stayed in the cluster peer list forever.** `Cluster.seen` was
only ever added to and deleted from on an explicit `bye`, so a node that
crashed — or whose Valkey connection died — was reported as a live peer for
the life of the process, and its stale blocklist digest was flagged as
divergence forever. `Peers()` now reaps peers unheard for `peer_ttl * 4`
(four missed heartbeats) and stamps each state with the receiver's own clock,
so clock skew between two machines can neither reap a healthy peer nor keep a
dead one. `Discover()` ages out the same way on the publisher's own
`last_seen` stamp, because the keyed Valkey record outlives a crash: a live
node keeps refreshing the key's TTL, so a dead node's field stayed readable
for the life of the fleet. `TestDeadPeerAgesOutOfPeers` (which was failing on
the tree as handed over — the reaping it asked for did not exist) now passes.
- **`peer_ttl: 0` would have crashed the process.** `time.NewTicker(PeerTTL/2)`
panics on a non-positive period, and `EXPIRE key 0` *deletes* the heartbeat
key, so a config that left `peer_ttl` unset would have taken the process down
at `Start` (or made every instance believe it is alone in the fleet). Both go
through `peerTTL()`/`peerDeadline()`, which fall back to the default.
- **Two cluster tests raced on the wire, not on the product.** A node that
joined after another's heartbeat was in flight hears about it on the next
tick, so "b does not see a", read immediately after "a sees b", is a coin
flip; and `TestStatusReportsDivergentPeers` waited for a digest on the wrong
node's view of the fleet. Both tests were wrong, not the code.
- **The partial-bind test only worked unprivileged.**
`TestStartReportsAPartialFailureButServes` used `127.0.0.1:1` as a port that
cannot be bound, which is not true in any container with
`net.ipv4.ip_unprivileged_port_start=0` — it binds, and the test failed with
both transports up. It now occupies a port with a live listener in the test
process, which is a deterministic `EADDRINUSE` on every machine and every
uid.
- **Four API/console contract breaks, all found by the new contract test:**
`blocklist.Source` and `SourceMeta` had no `json` tags at all, so the Sources
table rendered blank rows and its per-row toggle posted `name: undefined`;
`acl.GroupSummary` was flat where the console reads a nested `match`, so every
group card read "matches nothing"; `PUT /api/clients` answered 400
`json: unknown field "ptr_suffixes"` to the payload the console sends, because
`config.Group` and `config.Match` had yaml tags only; and `PUT /api/upstreams`
answered 400 `json: unknown field "health"`, because the console round-trips
the whole server object it read, runtime health fields included. The write
structs now carry json tags, the health fields are accepted and ignored (the
server owns them), and `acl.Match` is nested.
- **A duration on the wire was a nanosecond count.** `config.Duration` had a
`MarshalYAML` but no `MarshalJSON`, so `bootstraps[].refresh` arrived as
`30000000000` while the console typed the field as a string, and
`healthcheck_interval` was a float of seconds in one place and a string in
another. Durations on the wire are duration strings everywhere now, which is
the same spelling config.yaml uses; a bare JSON number is still accepted and
means **seconds**, not nanoseconds. `TestDurationJSONForm` pins all of it and
says the old behaviour was the bug.
- **A log row's key set depended on the row.** `api.LogEntry` was `omitempty`
on `client_group`, `response`, `upstream` and `list`, so a query with no ACL
group arrived without the key and the console rendered "undefined" where the
column should be empty. The contract test can only pin one shape if the
server emits one, so nothing on that struct is `omitempty` any more.
- **Every query from a client that asks for EDNS0 NSID got SERVFAIL.**
`dns.EDNS0_NSID.Nsid` is a *hex string* in miekg/dns v1.1.73 — `pack`
hex-decodes it, `unpack` hex-encodes it — and the engine put the literal
@@ -219,32 +299,14 @@ every admin endpoint — see the fix list.
## Pinned, not fixed (each one has a test that says so out loud)
The standing decision for this round was: **pin, do not change**. Every item
below is behaviour someone may want to change deliberately; each has a test
that fails when it changes, and says what to do when it does.
The standing decision for the round after this one was: **pin, do not change**.
Every item below is behaviour someone may want to change deliberately; each has
a test that fails when it changes, and says what to do when it does.
- **Unknown config keys are silently ignored** (a settled decision, see the
table above). `TestLoadIgnoresUnknownKeys` in `internal/config` also pins the
asymmetry that makes this surprising: a typo is invisible, while a known key
with a bad *type* is a startup failure.
- **`metrics.OutcomeLocal` has no producer.** The engine reports a local-zone
answer as `qlog.ActionAllow`, which `outcomeFor` maps to `OutcomeUpstream`, so
every locally answered query is counted as an upstream one and the
`outcome="local"` series is never published — the dashboard's local column is
permanently empty. `TestPinnedLocalAnswersAreCountedAsUpstream` asserts the
upstream counter moves by one and the local series stays absent. Fixing it
moves published counters, so it wants a decision and a dashboard note.
- **An RPZ record does not cover the name it names.** `rpzRule` gives a bare
owner name the subdomain form (`*.name`), and a subdomain-form rule matches
the name's children but not the name itself, so `analytics.example.org.` in a
policy zone blocks `cdn.analytics.example.org.` and leaves
`analytics.example.org.` resolving normally. RPZ semantics say the record
applies to the domain and everything under it, so `**.name` is what belongs
here — a one-character change in `rpzRule` that alters which queries get
blocked. Verified end to end against a running binary: the subdomain is
NXDOMAIN, the apex is not. Pinned by `TestExampleRPZZoneApexIsNotCovered`,
which also shows the same gap on the allow side (an `rpz-passthru.` record
for a domain does not allowlist the domain itself).
- **Auth configuration is frozen at `New`.** `api.base_path`,
`api.trusted_proxies` and the authenticator (`auth.mode`, `api_keys`) are
captured once; a reload re-reads CORS only. Writing a new API key through
@@ -293,31 +355,29 @@ that fails when it changes, and says what to do when it does.
## Continuation plan, in order
1. **`internal/cluster` suite** — in progress. The rolling heartbeat, the Valkey
lease and leader handoff, pub/sub config sync, divergence detection and
reconnect, against `miniredis`.
2. **The API response contract** — in progress on the Go side
(`internal/app` contract tests + the fixes listed above).
3. **The console side of that contract** — the next piece of work, and it should
be done against the canonical key list from step 2: the `qtype` fields that
are numbers on the wire but strings in TypeScript (the server also sends
`qtype_name`), `blocked_pct` arriving as 0..100 from the dashboard and 0..1
over SSE, the `whoami.can_write` / `read_only` flags that are fetched and
never used (a read-only key still shows every Save control), a `latency`
block the dashboard never receives, the cache "Lookups"/"Last seen" columns
reading `count`/`last_seen` where the API sends `hits`, and the refresh toast
that says "0 sources queued" because the API returned null.
4. **The remaining server tests** — in progress: shutdown ordering (in-flight
queries, idempotent `Stop`, a cancelled context, the port actually being
released), the stats counters, and a table for the pure helpers.
5. **Decide the open questions** in the pinned list above, one at a time, each
with a changelog note: the RPZ apex, `OutcomeLocal`, the frozen auth config,
the DoH client identity, CORS wildcards.
6. **Stop hand-writing the console's types.** The contract tests make the Go
1. **The console side of the contract** — the Go side is authoritative now
(`internal/app/contract_test.go`); `web/src/lib/types.ts` and the pages have
to be brought onto it. The known items: the Cache page's "Lookups"/"Last
seen" columns read `count`/`last_seen` where the server sends `hits` (there
is no last-seen data at all), the Dashboard reads `qps`/`total_queries` where
the server sends `queries_per_sec`/`queries_total` and never sends a
`latency` block, `whoami.can_write`/`read_only` are fetched and never used so
a read-only key still shows every Save control, `blocked_pct` arrives as
0..100 from the dashboard and 0..1 over SSE, and the upstreams page has to
cope with durations being strings on the wire.
2. **Finish the remaining server tests** — the counter-key layer is now covered
by `internal/server/counters_test.go`; what is left is a look at each
transport's shutdown ordering with a slow query in flight, and a `Stop`
goroutine-leak assertion.
3. **Decide the rest of the pinned list**, one at a time, each with a changelog
note: the frozen auth config, the DoH client identity, CORS wildcards, the
UDP query lost at `Stop`, `doh.sent` never bumped.
4. **Stop hand-writing the console's types.** The contract tests make the Go
side authoritative; the cheapest durable answer is generating
`web/src/lib/types.ts` from the Go structs (or an OpenAPI document emitted by
the server) so `apiFetch` is typed against something real instead of
`JSON.parse(text) as T`.
`JSON.parse(text) as T`. `internal/app/dump_shapes_test.go` (`DUMP_SHAPES=1`)
is the stopgap that already exists.
## Design notes worth preserving