Finish the server shutdown tests; fix the two bugs they found

internal/server was the last package whose shutdown was tested by assumption:
four transports, three shutdown shapes, and every one of them can strand a
goroutine that a client is holding open. Both bugs below were live in the tree
as handed over and neither had a test.

An idle client could hold the whole shutdown open. The stream accept loop waits
for its connections before Stop returns, and a connection parked in ReadMsg with
no deadline in front of it only ends when the client goes away — ctx.Err() is
checked between reads, so cancelling the context cannot reach a read that is
already blocked. One connection that connected and said nothing was enough to
make Stop sit there for the caller's entire grace period (15s in main) and then
return context.DeadlineExceeded, which the app reports as "shutdown completed
with errors"; and the cache snapshot the app takes right after Stop never
happened, because the orchestrator had given up first. It is a shutdown DoS that
costs the attacker one connection. The fix is a read deadline in the past on
every accepted connection when the context is cancelled, and deliberately not a
close: a query already inside the pipeline is past its read, so its answer still
goes out, which is the invariant TestInFlightQueryIsAnsweredAcrossStop pins. The
accept loop keeps a small registry of live connections for this — the accept
path, not the query path, and no query ever takes that mutex.

A UDP query in flight at Stop was lost and nothing waited for it. The stream
paths register their connections with s.wait; servePacket did not register at
all, so a UDP query already inside the pipeline was gone twice over: nothing
waited for it, and the answer had nowhere to go once it was ready, because a
datagram is written to the very socket the query arrived on. The visible damage
was not the lost answer (the client retries) but the app, which snapshots the
cache and closes the store the moment Stop returns — a query still in the
pipeline could be writing to a store on its way out. Packets are now counted in
s.wait, and the UDP listener's close wakes the read loop with an expired
deadline and then waits, bounded by udpDrainGrace (250ms), for the loop and the
packets it spawned before closing the socket. The bound is deliberate: a
shutdown may not spend its grace period on one slow upstream, and past the bound
the answer is dropped, which is what the client would see anyway.

Stop returning did not mean the process was done, so nothing could tell a
leaking shutdown from a clean one.
TestStopLeavesNoGoroutineRunningServerCode counts goroutines whose stack still
has a frame in the package — not runtime.NumGoroutine(), which in a test binary
is net/http, crypto/tls and the race detector talking over each other — and it
found the idle-connection bug above. TestStopIsNotBlockedByAnIdleStreamConnection
sets a 60s DoT idle timeout on purpose so a config value cannot hide the
regression behind a lucky tick, and TestInFlightQueryIsAnsweredAcrossStop now
covers all four transports. harness.idleStream exists because a test that only
dials a stream transport races the accept loop: Stop can close the listener
while the connection is still in the backlog, the server never accepts it, and a
shutdown assertion that needs a live server-side goroutine passes for the wrong
reason.

AGENTS.md gains the two durable invariants (no step in the shutdown may wait for
a client; every query goroutine is registered with s.wait, UDP included) and the
two test-hygiene traps. STATUS.md drops the UDP query lost at Stop from its
pinned list, since it is now a test rather than an accident, and the plan starts
at the pinned decisions that are actually left.

STATUS.md

@@ -1,11 +1,12 @@
 # fancy-dns — development status and continuation plan
 
-Last updated after the round that made the **API response contract a test**
-(`internal/app/contract_test.go` pins the exact JSON key set of every admin
-endpoint, and fixed the four drift items it exposed), decided the two pinned
-open questions, and finished the cluster package's peer-reaping gap. `AGENTS.md`
-holds the decisions, invariants and working rules; this file holds state and the
-plan.
+Last updated after the round that finished the **server shutdown paths**:
+every transport's ordering with a query in flight is now a test, and it found
+two real bugs — an idle client could hold a shutdown for the whole grace
+period, and a UDP query in flight was neither waited for nor answered. The
+round before that brought the console onto the Go side of the API contract, so
+both halves of that are done. `AGENTS.md` holds the decisions, invariants and
+working rules; this file holds state and the plan.
 
 ## Goal
 
@@ -81,17 +82,17 @@ Cache is a single shard-local lookup; only a miss touches the network.
 |---|---|---|
 | `internal/acl` | 26 | group compilation, matching, first-group-wins |
 | `internal/api` | 55 (156 cases) | the whole route table, three auth modes, CORS, rate limiting, SSE, settings, error branches, pinned limitations |
-| `internal/app` | 25 + the contract suite | wiring, hooks, **the API response contract**, integration |
+| `internal/app` | 30 (incl. the contract suite) | wiring, hooks, **the API response contract**, integration |
 | `internal/blocklist` | 28 | four parsers, the set, the manager, the shipped example zone |
 | `internal/cache` | 27 | LRU, TTL policy, serve-stale, bbolt round-trip |
-| `internal/cluster` | 30 | Valkey lease, pub/sub, divergence, dead-peer reaping |
+| `internal/cluster` | 20 | Valkey lease, pub/sub, divergence, dead-peer reaping |
 | `internal/config` | 22 | Duration/Size, `${VAR}`, unknown keys, the JSON wire form, the shipped example configs |
 | `internal/engine` | 26 | the pipeline end to end, plus the pins |
 | `internal/localzone` | 18 | wildcards, delegation, CNAME following |
 | `internal/metrics` | 10 | the metric names, which are the contract with the dashboards |
 | `internal/qlog` | 18 | ring, bbolt, top domains, sampling |
 | `internal/resolver` | 10 | pool, breakers, health, policies |
-| `internal/server` | 36 | four listeners, UDP deadline reset, truncation, DoH limits, identity, shutdown, stats |
+| `internal/server` | 57 | four listeners, UDP deadline reset, truncation, DoH limits, identity, counters, **shutdown and leak** |
 | `internal/zonefile` | 29 | the master-file parser and the shipped example zone |
 | `web` | 0 | typecheck and `oxlint` clean (0 errors), no test runner |
 | `cmd/fancy-dns` | 0 | exercised by `make smoke` and the CI smoke job instead |
@@ -109,14 +110,26 @@ Cache is a single shard-local lookup; only a miss touches the network.
 - `make web-check` — console typecheck and production build.
 - `make bench` — cache and blocklist benchmarks.
 
-### Test coverage is no longer the interesting gap
+### The contract gap is closed on both sides
 
-The API **response contract** is. The console's types are hand-written and
-`apiFetch` ends in `JSON.parse(text) as T`, so nothing in the toolchain compared
-them with the Go structs; a field that lost its `json` tag, or a response a
-struct never had, showed up as an empty table in a browser and nowhere else.
-`internal/app` now has contract tests that assert the exact JSON key set of
-every admin endpoint — see the fix list.
+The API **response contract** used to be the interesting gap. The console's
+types are hand-written and `apiFetch` ends in `JSON.parse(text) as T`, so
+nothing in the toolchain compared them with the Go structs; a field that lost
+its `json` tag, or a response a struct never had, showed up as an empty table in
+a browser and nowhere else. `internal/app` now has contract tests that assert
+the exact JSON key set of every admin endpoint (see the fix list), and the
+console has been brought onto them — so the two sides agree *today*, but only
+because someone compared them once. The durable fix is still open: see plan
+item 2.
+
+### The server's shutdown is now pinned behaviour, not folklore
+
+It was the last part of `internal/server` tested by assumption. Four
+transports, three shutdown shapes (a datagram socket, a stream accept loop with
+configurable idle timeouts, and an `http.Server` that drains politely), and
+every one of them can strand a goroutine a client is holding open. The
+invariants that came out of it are in AGENTS.md; the two bugs it found are in
+the fix list below.
 
 ### Decided this round (behaviour changed on purpose, each with a test)
 
@@ -144,6 +157,47 @@ every admin endpoint — see the fix list.
 
 ## Fixed this round
 
+- **A client that connected and said nothing held the whole shutdown open.**
+  The stream accept loop waits for its connections before `Stop` returns, and a
+  connection parked in `ReadMsg` with no deadline in front of it only ends when
+  the client goes away — `ctx.Err()` is checked *between* reads, so cancelling
+  the context cannot reach a read that is already blocked. One idle TCP
+  connection (or one DoT connection, with `timeouts.tls` set high) was enough to
+  make `Stop` sit there for the caller's entire grace period — 15s in `main` —
+  and then return `context.DeadlineExceeded`, which the app reports as
+  "shutdown completed with errors". Worse, the cache snapshot the app takes
+  right after `Stop` never happened: the orchestrator had already given up. It
+  is a shutdown DoS that costs the attacker one connection, and nothing tested
+  it. The fix is a read deadline in the past on every accepted connection when
+  the context is cancelled, and deliberately *not* a close: a query already
+  inside the pipeline is past its read, so its answer still goes out, which is
+  the invariant `TestInFlightQueryIsAnsweredAcrossStop` pins.
+  `TestStopIsNotBlockedByAnIdleStreamConnection` sets a 60s DoT idle timeout on
+  purpose, so a config value cannot hide the regression behind a lucky tick.
+- **A UDP query in flight at `Stop` was lost, and nothing waited for it.** The
+  stream paths register their connections with `s.wait`, so `Stop` waits for
+  them; `servePacket` did not register at all, so a UDP query already inside
+  the pipeline when the socket closed was gone twice over — nothing waited for
+  it, and the answer had nowhere to go even once it was ready, because a
+  datagram is written to the very socket the query arrived on. The visible
+  damage was not the lost answer (the client retries) but the app: it snapshots
+  the cache and closes the store the moment `Stop` returns, so a query still in
+  the pipeline could be writing to a store that was about to close. Packets are
+  now counted in `s.wait`, and the UDP listener's close waits (bounded by
+  `udpDrainGrace`, 250ms) for the read loop and the packets it spawned before it
+  closes the socket. A shutdown must not spend its grace period on one slow
+  upstream, hence the bound: past it the answer is dropped, which is what the
+  client would see anyway. Pinned on all four transports by
+  `TestInFlightQueryIsAnsweredAcrossStop/{tcp,dot,udp,doh}`.
+- **`Stop` returning did not mean the process was done.** Nothing asserted that
+  the server's goroutines were gone, so a shutdown that leaked one was
+  indistinguishable from one that did not. `TestStopLeavesNoGoroutineRunningServerCode`
+  counts goroutines whose stack still has a frame in this package — not
+  `runtime.NumGoroutine()`, which in a test binary is net/http, crypto/tls and
+  the race detector talking over each other — and it found the idle-connection
+  bug above, since the check is the only way to see a goroutine that has no
+  client left to answer.
+
 - **A dead peer stayed in the cluster peer list forever.** `Cluster.seen` was
   only ever added to and deleted from on an explicit `bye`, so a node that
   crashed — or whose Valkey connection died — was reported as a live peer for
@@ -336,12 +390,6 @@ a test that fails when it changes, and says what to do when it does.
   TCP/DoT.** `TestUDPQuestionlessMessageReachesTheEngine` pins the UDP half (the
   engine turns it into a FORMERR); the stream path returns instead. Probably
   right for both, but it is an asymmetry worth knowing about.
-- **A UDP query in flight at `Stop` can be lost.** The stream paths register
-  with `s.wait`, so `Stop` waits for them; `servePacket` does not, so a UDP
-  query already inside the pipeline when the listener closes may never be
-  answered. That is arguable — UDP has no connection to hang up on and the
-  client will retry — but the asymmetry with TCP/DoT is not pinned anywhere, and
-  it should be a decision rather than an accident.
 - **`doh.sent` is never bumped.** The DoH handler writes its answer through
   `net/http`, so the `sent` counter is only meaningful on UDP/TCP/DoT, and the
   console's transport table shows `sent: 0` for DoH. Pinned as current
@@ -355,29 +403,25 @@ a test that fails when it changes, and says what to do when it does.
 
 ## Continuation plan, in order
 
-1. **The console side of the contract** — the Go side is authoritative now
-   (`internal/app/contract_test.go`); `web/src/lib/types.ts` and the pages have
-   to be brought onto it. The known items: the Cache page's "Lookups"/"Last
-   seen" columns read `count`/`last_seen` where the server sends `hits` (there
-   is no last-seen data at all), the Dashboard reads `qps`/`total_queries` where
-   the server sends `queries_per_sec`/`queries_total` and never sends a
-   `latency` block, `whoami.can_write`/`read_only` are fetched and never used so
-   a read-only key still shows every Save control, `blocked_pct` arrives as
-   0..100 from the dashboard and 0..1 over SSE, and the upstreams page has to
-   cope with durations being strings on the wire.
-2. **Finish the remaining server tests** — the counter-key layer is now covered
-   by `internal/server/counters_test.go`; what is left is a look at each
-   transport's shutdown ordering with a slow query in flight, and a `Stop`
-   goroutine-leak assertion.
-3. **Decide the rest of the pinned list**, one at a time, each with a changelog
+1. **Decide the rest of the pinned list**, one at a time, each with a changelog
    note: the frozen auth config, the DoH client identity, CORS wildcards, the
-   UDP query lost at `Stop`, `doh.sent` never bumped.
-4. **Stop hand-writing the console's types.** The contract tests make the Go
+   question-less message asymmetry, `doh.sent` never bumped. (The UDP query lost
+   at `Stop` left this list when it was fixed.)
+2. **Stop hand-writing the console's types.** The contract tests make the Go
    side authoritative; the cheapest durable answer is generating
    `web/src/lib/types.ts` from the Go structs (or an OpenAPI document emitted by
    the server) so `apiFetch` is typed against something real instead of
    `JSON.parse(text) as T`. `internal/app/dump_shapes_test.go` (`DUMP_SHAPES=1`)
    is the stopgap that already exists.
+3. **Decide whether the shutdown grace belongs in config.** `main` hard-codes
+   15s for the whole `Stop`, and the drain inside it is 250ms of that. A
+   Kubernetes `terminationGracePeriodSeconds` and a resolver's own notion of
+   "long enough to answer what is in flight" are the same number, and right now
+   only one of them is visible. The four-transport invariant is pinned, so
+   changing the bound is cheap — but the knobs do not exist yet.
+4. **Read the pinned "not fixed" list in this file for anything that has since
+   become untrue.** Each entry claims a behaviour a reader may act on; a claim
+   that no longer matches the code is worse than a missing one.
 
 ## Design notes worth preserving
 
@@ -399,6 +443,11 @@ a test that fails when it changes, and says what to do when it does.
 - **The UDP listener resets its read deadline every iteration.** A deadline set
   once expires for good and turns the loop into a spin that never reads another
   packet (caught by `TestBlocklistToggle`).
+- **Shutdown order is: cancel, close every listener, then wait.** The close is
+  not interruptible by the caller's context, because releasing the ports is the
+  work the next process start needs. Everything after it is interruptible, and
+  a cancelled context is expected to be able to cut the *wait* short — which is
+  why no step in the shutdown may depend on a client doing anything.
 - **Only the Valkey lease holder fetches lists**, so N instances do not hit a
   mirror N times.
 - **Serve-stale** answers get TTL 1; an expired upstream answer is never cached;
@@ -422,6 +471,16 @@ a test that fails when it changes, and says what to do when it does.
 - Assert on invariants, not on what a concurrent goroutine happened to observe.
   "A reader must have seen address X" is a flake; "every list a reader sees is
   one of the published ones" is a real property and cannot flake.
+- **A test that only dials a stream transport races the accept loop.** Stop can
+  close the listener while the connection is still in the backlog, the server
+  never accepts it, and any shutdown assertion that needs a live server-side
+  goroutine passes for the wrong reason. Exchange one query first, then leave
+  the connection alone — that is what `harness.idleStream` is for.
+- **Count leaked goroutines by stack, not by `runtime.NumGoroutine()`.** A test
+  binary shares its process with net/http, crypto/tls and the race detector; a
+  baseline count is noise. "Does any goroutine still have a frame in this
+  package?" is a property of the product, and it needs the caller's own
+  goroutine filtered out by id or every run finds exactly one.
 - `tsc -b` replays **cached diagnostics** from `node_modules/.tmp/*.tsbuildinfo`.
   After a large edit, `rm node_modules/.tmp/*.tsbuildinfo`.
 - `vite build` empties `outDir` and **deletes `internal/web/dist/.gitkeep`**.