Concurrent cache misses for one name each went upstream. After a restart, a
cache flush or a cluster-object change that is a burst of identical queries
exactly when the cache is empty. Identical exchanges (same name, type, class,
DO bit, pinned upstream set and forwarded client subnet) now share one: the
first query runs it on its own goroutine and context, the rest wait and get
private copies, and the leader alone writes the cache. If the leader's client
leaves, waiters take over instead of failing with its cancellation. Shared
answers are counted in fancydns_upstream_coalesced_total.
validate() accepted OpcodeUpdate, so a dynamic update was forwarded upstream as
an ordinary lookup, and every other opcode got FORMERR. A recursive resolver
implements QUERY only: anything else is now NOTIMP (RFC 1035 4.1.1), checked
before the question count since STATUS carries no question.
Prefetch only fired when a stale entry was served, so the first client after a
hot name expired got a TTL-1 stale answer. A fresh entry is now refreshed in
the background when it has been served at least twice and under a tenth of its
lifetime remains; the hit counter restarts with each stored entry, so a name
nobody comes back for stops being refreshed. Counted in
fancydns_cache_prefetch_total.
miekg/dns matches only the message ID of an upstream reply, so a forged reply
with the right ID for a different question was accepted and cached under the
name that was asked. exchangeOne now requires the reply to carry the question
that was sent (an empty section only for FORMERR, SERVFAIL, NOTIMP and
REFUSED). A mismatch fails the attempt without counting against the upstream,
so a spoofer cannot trip the circuit breaker.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The initial upstream exchange ignored its supplied timeout and used hardcoded client limits. Bound each attempt with a context, copy the shared client for its timeout, and share the attempt budget with UDP-to-TCP fallback. Close private connections on cancellation to interrupt reads, and exclude caller cancellation from upstream failures so parallel losers remain healthy.
The normal engine pipeline never applied server.timeouts.query, allowing failover to outlive the configured query budget. Apply a hot-reloaded overall context deadline at Handle while retaining earlier caller deadlines. Add silent UDP/TCP/TLS, cancellation, fallback, failover, and reload regressions.
The server inflight test assumed receiving a TCP reply meant its sent counter had already been incremented. Wait for both completion counters before asserting them, preserving the existing assertions while removing the scheduling race.
internal/engine, internal/config and internal/metrics had no test file at all,
and writing them turned up five real bugs.
internal/engine (23 tests, new harness_test.go): a wired engine over a fake
upstream on loopback, a local zone, a blocklist, a cache, a query log and an
injectable clock. It pins the pipeline order end to end -- ACL deny before the
local zone, local zone before blocklist, blocklist before cache, drop sends
nothing, the concurrency and rate-limit refusals, serve-stale served with TTL 1
and refreshed once per key, truncation against a 512-byte client, and exactly
one metrics record and one query-log entry per query.
Five fixes:
- config.ParseSize ignored the unit multiplier, so 512MiB parsed as 512 bytes
and the whole memory envelope was inert for anyone who wrote a unit.
- finalize (EDNS0 and the UDP size) ran only on the cache-hit path, so a large
uncached answer left oversized for a 512-byte client. It now runs on the
upstream, local-zone and blocklist paths too.
- The upstream UDP client read buffer was hard-coded to 1232 while we advertise
dns.max_udp_size, so with a larger buffer configured every answer above 1232
bytes was read truncated and became SERVFAIL. The pool takes the size now.
- LRU.prefetchGate was a 64-slot global cap, not the single flight its comment
promised: a burst of stale hits launched up to 64 duplicate refreshes of one
name. It is a per-key latch now; distinct names still refresh in parallel.
- metrics.RecordMiss and RecordStaleHit moved only the Prometheus collector, so
the Cache page's misses and stale hits read 0 for the life of the process.
AGENTS.md gains the trap that cost the most time: this version of miekg/dns
reports a character-string over 255 bytes as zero length, so such a message
silently fails to pack, and the write failure looks like a client timeout.
It asserted that a reader had to observe both of the swapped-in upstream
addresses, which is a demand about *which* list a reader happens to see. A
reader that finished its sampling before the rebind loop got there satisfied
every real property and still failed, so this failed roughly 2 runs in 10 and
would have turned CI red on nothing.
What the test should pin is the copy-on-write invariant itself: a reader must
only ever see a whole, published list. Each reader now compares every observed
snapshot against the three lists this test publishes, and fails immediately on
a half-applied one. That is checked on every iteration rather than inferred
afterwards, and it makes no demand about which list was seen, so it cannot
flake. 15 consecutive -race runs, no failure.
A cgroup OOM kill is the worst outcome available to a resolver: an outage
rather than a cache miss, with no chance to serve stale on the way out. So:
- memory.limit / memory.reserve / memory.cache_max_bytes, a new config section.
limit becomes a GOMEMLIMIT through debug.SetMemoryLimit, reduced by the
reserve. The reserve exists because the Go heap does not account for
goroutine stacks, runtime metadata, or the bbolt mmap backing the cache
snapshot and the query log — handing the runtime the full cgroup limit is how
you get killed by a mechanism that believed it had headroom.
- config.Size, so a budget reads as "512MiB" rather than a byte count converted
by hand. Strict about junk: "512MiBx" is an error.
- The cache byte budget is now actually fed to the LRU. It already enforced
maxBytes per shard on insert, but had no setter and nothing ever called one,
so the footprint of cache.size was unbounded: one cached ANY or TXT answer
can be tens of kilobytes, so 500k entries is anywhere from 150MB to several GB
depending on what the traffic asked for.
- api.pprof mounts net/http/pprof on the admin listener. The pprof handlers sit
on the app-level mux, not inside the API's own handler, so they would bypass
the auth middleware entirely; the option is therefore refused unless every
admin address is loopback. Profiling means port-forwarding, so this is safe
by construction rather than by policy, and cmdline cannot leak an argument
that happens to be a key.
- serveUDP was calling net.ResolveUDPAddr per packet. The peer already is a
*net.UDPAddr; that call builds a string, re-parses the port and allocates on
the busiest path in the process.
Also fixes a flaky test: TestParallelPolicyAnswersFromTheFastestUpstream
asserted the slow upstreams were hit, but the fan-out cancels on the first
answer, so whether they were dispatched before cancellation is a real race. It
now asserts the answer came from the fast upstream and that nothing waited.
The cache snapshot never loaded, so every restart was a cold cache. Snapshot
packed the whole key while Load decoded the record header out of the key's
remainder instead of the value, so every entry was skipped. The two sides are
now an explicit encodeRecord/decodeRecord pair with a version byte and a CRC-32C
trailer, so a future format change is detected instead of silently misparsed.
Load also drops entries that are already past ExpiresAt and rewrites the TTL to
the time actually left, so a served TTL never jumps back up across a restart.
Blocklist, all six gaps the failing tests were pointing at: a leading UTF-8 BOM
is stripped, "!allow example.com" works, SourceMeta exists for a source that
only ever contributed a rule, Pattern() reports the bare name again (the new
WireName() produces the wildcard form for the API), an RPZ record with no
explicit TTL inherits the zone default instead of whatever the reader left in
the RR, a hosts line with no address in the first field says so, and the
adblock parser handles CRLF and adblock-flavoured lines pasted into a plain
list. TestHolderSwapIsAtomic was asserting the wrong invariant — the swap was
always atomic, a name blocked by the old set is legitimately unblocked by the
new one.
Tests added for acl and resolver, both of which turned up real bugs:
- a PTR suffix matched across label boundaries, so a group scoped to
example.com. also swallowed ample.com. and notexample.com. — a client could
land in a group it does not belong to;
- the "*" catch-all was dropped from the rendered domain patterns, so the
engine's per-query recompile lost the "everything" case;
- Upstream.Success never closed an open breaker, so a recovered upstream stayed
excluded for the full cooldown;
- Exchange fanned out in parallel even under policy: failover.
Deploy set: go.mod requires Go 1.27.1 but CI pinned 1.23 and the Dockerfile
pinned golang:1.23-alpine, so neither could have built. The Dockerfile also
copied the source tree after the console bundle, which clobbered it.
The k8s set asked for hostNetwork with two replicas, but the server never sets
SO_REUSEPORT, so the second pod could not bind :53 and would crash-loop. It now
runs without hostNetwork behind a LoadBalancer Service, with a separate
ClusterIP Service for the Ingress (a headless Service cannot back one), a
ServiceAccount, an API-key Secret, per-pod PVCs, and a kustomization. The HPA
is gone: it cannot drive a StatefulSet. To let the API key come from that
Secret, config.Load now expands ${VAR} / ${VAR:-default} / ${NAME-default} and
fails to start on an unset variable with no default, so a missing secret is a
crash loop rather than an unauthenticated console. Expansion skips YAML
comments, so documenting ${VAR} in the ConfigMap does not break the file.
Verified: gofmt, go build, go vet clean; go test -race ./internal/... green
(acl, app, blocklist, cache, config, resolver, zonefile).
matsandClaude Sonnet 5.5