Finish the server shutdown tests; fix the two bugs they found
build / build-push (push) Successful in 22s
build / build-push (push) Successful in 22s
internal/server was the last package whose shutdown was tested by assumption: four transports, three shutdown shapes, and every one of them can strand a goroutine that a client is holding open. Both bugs below were live in the tree as handed over and neither had a test. An idle client could hold the whole shutdown open. The stream accept loop waits for its connections before Stop returns, and a connection parked in ReadMsg with no deadline in front of it only ends when the client goes away — ctx.Err() is checked between reads, so cancelling the context cannot reach a read that is already blocked. One connection that connected and said nothing was enough to make Stop sit there for the caller's entire grace period (15s in main) and then return context.DeadlineExceeded, which the app reports as "shutdown completed with errors"; and the cache snapshot the app takes right after Stop never happened, because the orchestrator had given up first. It is a shutdown DoS that costs the attacker one connection. The fix is a read deadline in the past on every accepted connection when the context is cancelled, and deliberately not a close: a query already inside the pipeline is past its read, so its answer still goes out, which is the invariant TestInFlightQueryIsAnsweredAcrossStop pins. The accept loop keeps a small registry of live connections for this — the accept path, not the query path, and no query ever takes that mutex. A UDP query in flight at Stop was lost and nothing waited for it. The stream paths register their connections with s.wait; servePacket did not register at all, so a UDP query already inside the pipeline was gone twice over: nothing waited for it, and the answer had nowhere to go once it was ready, because a datagram is written to the very socket the query arrived on. The visible damage was not the lost answer (the client retries) but the app, which snapshots the cache and closes the store the moment Stop returns — a query still in the pipeline could be writing to a store on its way out. Packets are now counted in s.wait, and the UDP listener's close wakes the read loop with an expired deadline and then waits, bounded by udpDrainGrace (250ms), for the loop and the packets it spawned before closing the socket. The bound is deliberate: a shutdown may not spend its grace period on one slow upstream, and past the bound the answer is dropped, which is what the client would see anyway. Stop returning did not mean the process was done, so nothing could tell a leaking shutdown from a clean one. TestStopLeavesNoGoroutineRunningServerCode counts goroutines whose stack still has a frame in the package — not runtime.NumGoroutine(), which in a test binary is net/http, crypto/tls and the race detector talking over each other — and it found the idle-connection bug above. TestStopIsNotBlockedByAnIdleStreamConnection sets a 60s DoT idle timeout on purpose so a config value cannot hide the regression behind a lucky tick, and TestInFlightQueryIsAnsweredAcrossStop now covers all four transports. harness.idleStream exists because a test that only dials a stream transport races the accept loop: Stop can close the listener while the connection is still in the backlog, the server never accepts it, and a shutdown assertion that needs a live server-side goroutine passes for the wrong reason. AGENTS.md gains the two durable invariants (no step in the shutdown may wait for a client; every query goroutine is registered with s.wait, UDP included) and the two test-hygiene traps. STATUS.md drops the UDP query lost at Stop from its pinned list, since it is now a test rather than an accident, and the plan starts at the pinned decisions that are actually left.
This commit is contained in:
1 parent
3a199bb310
commit
b6d70a50fe
5 files changed
+526
-41
No files matched your search
@@ -83,6 +83,21 @@ Break these and you break the design, not just the test.
|
||||
truncated forever.
|
||||
- **The UDP read deadline is reset every iteration.** A deadline set once expires
|
||||
for good and turns the loop into a spin that never reads another packet.
|
||||
- **No step in the shutdown may wait for a client.** `Stop` is: cancel, close
|
||||
every listener (not interruptible — releasing the ports is the work the next
|
||||
process start needs), then wait for the goroutines (interruptible by the
|
||||
caller's context). The wait is the step that breaks when something depends on
|
||||
a client: a connection parked in a read with no deadline in front of it never
|
||||
notices the cancellation, and one idle connection is enough to spend the whole
|
||||
grace period and then report a context error. Reach those connections with a
|
||||
read deadline in the past, **not** a close — a query already inside the
|
||||
pipeline is past its read, and its answer still has to go out.
|
||||
- **Every query goroutine is registered with `s.wait`, UDP included**, so `Stop`
|
||||
returning means nothing is still inside the engine: the app snapshots the
|
||||
cache and closes the store immediately afterwards, and a query still in flight
|
||||
would be writing to a store on its way out. A UDP answer has nowhere else to
|
||||
go, so that transport also gets a short bounded drain before its socket
|
||||
closes — see `udpDrainGrace` in `internal/server/server.go`.
|
||||
- **Only the Valkey lease holder fetches blocklists**, so N instances do not hit
|
||||
a mirror N times. Followers serve their compiled copy.
|
||||
- **Blocklist changes flush the cache** rather than guessing which subtrees are
|
||||
@@ -149,6 +164,16 @@ partial result described as done is not.
|
||||
- Assert on invariants, not on what a concurrent goroutine happened to observe.
|
||||
"A reader must have seen address X" is a flake; "every list a reader sees is one
|
||||
of the published ones" is a real property and cannot flake.
|
||||
- **A test that only dials a stream transport races the accept loop.** A
|
||||
shutdown can close the listener while the connection is still in the backlog,
|
||||
so the server never accepts it and the test passes for the wrong reason.
|
||||
Exchange one query first, then leave the connection idle
|
||||
(`harness.idleStream`).
|
||||
- **Count leaked goroutines by stack, not by `runtime.NumGoroutine()`.** The
|
||||
test binary shares its process with net/http, crypto/tls and the race
|
||||
detector, so a baseline count is noise. "Is any goroutine still running code
|
||||
in this package?" is a property of the product — and it needs the caller's own
|
||||
goroutine filtered out by id, or every run finds exactly one.
|
||||
- Spin up `dns.Server` fakes on `127.0.0.1:0`: `net.Listen`/`net.ListenPacket` to
|
||||
find the port, `ActivateAndServe` in a goroutine, stop in `t.Cleanup`.
|
||||
- A `dns.Msg` carrying a `TXT` string over 255 bytes **does not pack** in
|
||||
|
||||
Reference in new issue
Block a user