internal/server was the last package whose shutdown was tested by assumption:
four transports, three shutdown shapes, and every one of them can strand a
goroutine that a client is holding open. Both bugs below were live in the tree
as handed over and neither had a test.
An idle client could hold the whole shutdown open. The stream accept loop waits
for its connections before Stop returns, and a connection parked in ReadMsg with
no deadline in front of it only ends when the client goes away — ctx.Err() is
checked between reads, so cancelling the context cannot reach a read that is
already blocked. One connection that connected and said nothing was enough to
make Stop sit there for the caller's entire grace period (15s in main) and then
return context.DeadlineExceeded, which the app reports as "shutdown completed
with errors"; and the cache snapshot the app takes right after Stop never
happened, because the orchestrator had given up first. It is a shutdown DoS that
costs the attacker one connection. The fix is a read deadline in the past on
every accepted connection when the context is cancelled, and deliberately not a
close: a query already inside the pipeline is past its read, so its answer still
goes out, which is the invariant TestInFlightQueryIsAnsweredAcrossStop pins. The
accept loop keeps a small registry of live connections for this — the accept
path, not the query path, and no query ever takes that mutex.
A UDP query in flight at Stop was lost and nothing waited for it. The stream
paths register their connections with s.wait; servePacket did not register at
all, so a UDP query already inside the pipeline was gone twice over: nothing
waited for it, and the answer had nowhere to go once it was ready, because a
datagram is written to the very socket the query arrived on. The visible damage
was not the lost answer (the client retries) but the app, which snapshots the
cache and closes the store the moment Stop returns — a query still in the
pipeline could be writing to a store on its way out. Packets are now counted in
s.wait, and the UDP listener's close wakes the read loop with an expired
deadline and then waits, bounded by udpDrainGrace (250ms), for the loop and the
packets it spawned before closing the socket. The bound is deliberate: a
shutdown may not spend its grace period on one slow upstream, and past the bound
the answer is dropped, which is what the client would see anyway.
Stop returning did not mean the process was done, so nothing could tell a
leaking shutdown from a clean one.
TestStopLeavesNoGoroutineRunningServerCode counts goroutines whose stack still
has a frame in the package — not runtime.NumGoroutine(), which in a test binary
is net/http, crypto/tls and the race detector talking over each other — and it
found the idle-connection bug above. TestStopIsNotBlockedByAnIdleStreamConnection
sets a 60s DoT idle timeout on purpose so a config value cannot hide the
regression behind a lucky tick, and TestInFlightQueryIsAnsweredAcrossStop now
covers all four transports. harness.idleStream exists because a test that only
dials a stream transport races the accept loop: Stop can close the listener
while the connection is still in the backlog, the server never accepts it, and a
shutdown assertion that needs a live server-side goroutine passes for the wrong
reason.
AGENTS.md gains the two durable invariants (no step in the shutdown may wait for
a client; every query goroutine is registered with s.wait, UDP included) and the
two test-hygiene traps. STATUS.md drops the UDP query lost at Stop from its
pinned list, since it is now a test rather than an accident, and the plan starts
at the pinned decisions that are actually left.
A cgroup OOM kill is the worst outcome available to a resolver: an outage
rather than a cache miss, with no chance to serve stale on the way out. So:
- memory.limit / memory.reserve / memory.cache_max_bytes, a new config section.
limit becomes a GOMEMLIMIT through debug.SetMemoryLimit, reduced by the
reserve. The reserve exists because the Go heap does not account for
goroutine stacks, runtime metadata, or the bbolt mmap backing the cache
snapshot and the query log — handing the runtime the full cgroup limit is how
you get killed by a mechanism that believed it had headroom.
- config.Size, so a budget reads as "512MiB" rather than a byte count converted
by hand. Strict about junk: "512MiBx" is an error.
- The cache byte budget is now actually fed to the LRU. It already enforced
maxBytes per shard on insert, but had no setter and nothing ever called one,
so the footprint of cache.size was unbounded: one cached ANY or TXT answer
can be tens of kilobytes, so 500k entries is anywhere from 150MB to several GB
depending on what the traffic asked for.
- api.pprof mounts net/http/pprof on the admin listener. The pprof handlers sit
on the app-level mux, not inside the API's own handler, so they would bypass
the auth middleware entirely; the option is therefore refused unless every
admin address is loopback. Profiling means port-forwarding, so this is safe
by construction rather than by policy, and cmdline cannot leak an argument
that happens to be a key.
- serveUDP was calling net.ResolveUDPAddr per packet. The peer already is a
*net.UDPAddr; that call builds a string, re-parses the port and allocates on
the busiest path in the process.
Also fixes a flaky test: TestParallelPolicyAnswersFromTheFastestUpstream
asserted the slow upstreams were hit, but the fan-out cancels on the first
answer, so whether they were dispatched before cancellation is a real race. It
now asserts the answer came from the fast upstream and that nothing waited.
developer