fancy-dns/STATUS.md

fancy-dns — development status and continuation plan

Last updated when the whole tree went green. AGENTS.md holds the decisions, invariants and working rules; this file holds state and the plan.

Goal

A lightweight, high-performance recursive DNS server with:

• Go backend — sharded LRU cache backed by bbolt, blocklist replication from multiple formats, per-client ACLs, query logging, Prometheus metrics.
• Secure DNS — DNS-over-TLS (RFC 7858) and DNS-over-HTTPS (RFC 8484). DNSSEC validation was explicitly scoped out; AD/CD/extended-RCODE are passed through only.
• React + Tailwind admin console embedded in the binary.
• Config as zone files + YAML, hot-reloadable, with Valkey/Redis for cross-instance sync (chosen over AMQP).

Architecture decisions (settled with the user)

Decision	Choice	Why
Scope	Full production	clustering, metrics, ACLs, query logs, hot reload, Docker, k8s, CI
Config store	YAML + DNS master files	reviewable/diffable in git
Cluster sync	Valkey (Redis protocol)	simple, mature Go client, easy local dev
Secure DNS	DoT + DoH, no DNSSEC validation	avoids signer/anchor machinery; pass-through only
Hot cache	Sharded LRU in memory, bbolt for snapshots	no I/O on the hot path; warm restart
k8s networking	No hostNetwork, LoadBalancer Service	the server never sets SO_REUSEPORT, so two host-network pods cannot share :53
Memory	Soft GOMEMLIMIT, cache byte budget	a cgroup kill is an outage; a cache miss is not
Scale-out	StatefulSet, no HPA	an HPA cannot drive a StatefulSet

Package layout

cmd/fancy-dns/main.go          entrypoint: flags, signals, SIGHUP reload
internal/
  config/       Config struct, Duration, Size, Validate, atomic Store + hooks,
                ${VAR} environment expansion in Load
  zonefile/     RFC 1035 master-file parser ($TTL/$ORIGIN, parens, serial math)
  blocklist/    Set (exact/sub/root maps), 4 parsers, Manager (fetch+compile+swap)
  cache/        Key, Entry, sharded LRU, TTL policy, serve-stale, bbolt Store
  resolver/     Upstream pool: health, circuit breakers, parallel/failover/random
  acl/          compiled groups, CIDR/PTR/tag matching, per-group domain policy
  localzone/    authoritative data from master files, wildcards, delegations
  engine/       THE QUERY PIPELINE (see below), reverse-PTR cache
  server/       UDP, TCP, DoT, DoH listeners; all funnel into one handler
  qlog/         bbolt query-log ring + live in-memory ring + SSE fanout
  metrics/      Prometheus collectors + rolling time-series rings
  cluster/      Valkey pub/sub, leader lease, divergence detection
  api/          admin REST + SSE, auth, CORS, rate limiting
  app/          wiring, lifecycle, API hook implementations, memory envelope
  web/          go:embed of the built console
web/            Vite + React + TS + Tailwind console
deploy/         example config, example zones, k8s manifests

Query pipeline order (engine.Handle)

validate → ACL deny → per-client concurrency → global concurrency → rate limit
  → group allow/deny domains → local zone → blocklist → cache → upstream
  → cache put → finalize (EDNS0, UDP truncation) → metrics + qlog

Cache is a single shard-local lookup; only a miss touches the network.

Status

Green — the whole tree passes

gofmt clean, go build ./... clean, go vet clean.

go test -race ./internal/... -count=1 -timeout 300s is green across every package that has a suite: acl, app, blocklist, cache, config, engine, localzone, metrics, qlog, resolver, zonefile — roughly 230 tests — and api joined the list with 78 cases of its own.

make check-go is the inner loop (1.3 s: gofmt check, vet, compile). It uses go-build, not build, because build depends on web and shells out to npm. make check = check-go test.

No test suite yet

cluster, server, web.

internal/engine is now covered — 23 tests over a harness (harness_test.go) that builds a wired engine with a fake upstream on loopback, a local zone, a blocklist, a cache and a query log — and internal/config has a Size suite. internal/metrics has a counter/gauge/exposition suite (9 tests) that pins the metric names, which are the contract with the Grafana dashboards and the ServiceMonitor. internal/api is covered too: every route in the table (each asserted to reach its hook, so a new route without a case is a route nothing tests), method/path rejection, a read-route sweep that asserts nothing mutated, panic recovery, the base path, all three auth modes, read-only keys and allow_config_write: false, loopback trust, and the hashed API keys. internal/server is the one in progress.

Fixed in the last round (the engine and config.Size work)

• config.ParseSize ignored the unit multiplier entirely. scaleSize took the number and returned Size(n); mult never entered the arithmetic, so 512MiB parsed as 512 bytes and 1GiB as 1 byte. Every memory budget written in units — memory.limit, memory.reserve, memory.cache_max_bytes, the cache byte budget in the app wiring — was off by a factor of the unit, and EffectiveLimit's 64 MiB floor then quietly replaced the operator's number. The memory envelope was inert for anyone who wrote a unit, which is everyone: a bare number is bytes. The fix is one multiplication; size_test.go now pins 1.5GiB, 0.5KiB, the unit shadowing cases (MiB must not match M), the out-of-range error and the truncating rounding.
• A cache miss and a stale hit moved only the Prometheus counter. metrics.RecordMiss and metrics.RecordStaleHit bumped the collector and not the atomics that Totals reads, so the Cache page's "misses" and "stale hits" read 0 for the life of the process while the Grafana panel was right — exactly the disagreement Record's doc comment says cannot happen.
• The uncached path never truncated. finalize — the EDNS0/UDP-size step that turns an oversized answer into a question-only TC reply — was called only on the cache hit path. A multi-kilobyte answer from upstream, from the local zone or from a sinkhole left here as a datagram no 512-byte UDP client can receive, and the client silently dropped it. All four reply paths now finalize.
• The serve-stale prefetch gate was not single-flight. LRU.prefetchGate was make(chan struct{}, 64), so a burst of stale hits on one hot name launched up to 64 duplicate upstream exchanges instead of the one its own doc comment promised. It is a per-key latch now (a map[Key]struct{} under a mutex): distinct stale names still refresh in parallel, one name does not. TryPrefetch/ReleasePrefetch take the key.
• The upstream UDP client read buffer was hard-coded to 1232 while the engine advertises dns.max_udp_size upstream. An operator who raised max_udp_size above 1232 got a SERVFAIL for every answer over 1232 bytes: the datagram was read truncated and failed to unpack with "dns: overflowing header size". resolver.NewPoolWithUDPSize takes the advertised size and the app passes cfg.DNS.MaxUDPSize; NewPool keeps the 1232 default.

Fixed recently and worth not re-breaking

• Every relative RDATA name in a master file pointed at the DNS root. The qualification pass ran after dns.NewRR, and the library resolves RDATA names against the root, so www CNAME host became CNAME host. — the alias pointed at the root, and so did every relative NS, MX, PTR, SRV, NAPTR and SOA target in every zone file. It is now a token-level pass driven by a table of which RDATA positions hold a name; dotted names stay absolute, so the RPZ targets . and rpz-drop. and the null MX/SRV targets are untouched.
• The cache snapshot never round-tripped. Snapshot packed the whole key while Load decoded the record header out of the key's remainder instead of the value, so all 25 entries were skipped — every restart was a cold cache. There is now an explicit encodeRecord/decodeRecord pair with a version byte and a CRC-32C trailer. Load drops entries already past ExpiresAt and rewrites the TTL to the time actually left, so a served TTL never jumps back up across a restart.
• ${VAR} expansion never worked inside quotes — which is the only form anyone writes. The reference check sat in the same switch as quote tracking, below the inDouble/inSingle branches, so api_keys: ["${KEY}"] came out of Load unchanged and the admin API key was the literal string "${FANCYDNS_API_KEY}". The k8s manifest looked correct and was not. A variable that is set but empty now expands to empty rather than erroring (the ${X} / ${X-y} / ${X:-y} distinction); \${FOO} is literal.
• ACL PTR suffix matched across label boundaries, so a group scoped to example.com. also swallowed ample.com. — a client could land in a group it does not belong to.
• The ACL * catch-all was dropped from the rendered domain patterns, so the engine's per-query recompile lost the "everything" case.
• Upstream.Success never closed an open breaker. A recovered upstream stayed excluded for the full cooldown — the mirror of the older bug where a failed exchange marked a dead upstream healthy.
• Exchange fanned out in parallel even under policy: failover, because the parallel path was only taken when exactly one upstream was chosen.
• The cache byte budget was never fed to the LRU. maxBytes was enforced per shard on insert but had no setter and nothing called one, so cache.size had no predictable memory cost.
• The memory envelope (memory.limit/reserve/cache_max_bytes) is new, wired to debug.SetMemoryLimit. The reserve covers what the Go heap does not account for: goroutine stacks, runtime metadata, and the bbolt mmap.
• Blocklist: leading BOM stripped, !allow works, SourceMeta exists for a rule-only source, Pattern() reports the bare name (WireName() gives the wildcard form for the API), RPZ inherits the zone default TTL, hosts files complain about a missing address, adblock handles CRLF and over-long lines.
• $TTL was ignored by the zone parser; every record got the SOA minimum.
• TestRebindUnderConcurrentQueries was a flake (2 runs in 10) — it asserted a reader had to observe two specific addresses, which is a claim about what a goroutine happened to see. It now pins the copy-on-write invariant itself: every observed snapshot must equal one of the published lists.
• TestHolderSwapIsAtomic had the wrong invariant, not the Holder.
• CI could not have passed: go.mod requires Go 1.27.1, CI pinned 1.23 and the Dockerfile pinned golang:1.23-alpine.
• The Dockerfile clobbered the console — it copied the source tree after the console bundle.
• The k8s set could not have worked: hostNetwork: true with replicas: 2 and no SO_REUSEPORT. Now a LoadBalancer Service, a separate ClusterIP Service for the Ingress, a ServiceAccount, an API-key Secret, per-pod PVCs, a kustomization. The HPA is gone.
• serveUDP called net.ResolveUDPAddr per packet; the peer already is a *net.UDPAddr.

New findings from the api suite, unfixed

• The login rate limiter never applies to the login endpoint. handleLogin is registered through public, which bypasses authed and therefore allowRequest — the only reader of lockedTill, and handleLogin the only writer of failures. So api.rate_limit.per_minute, documented as "requests per minute per IP for the login endpoint", applies to every other route instead: posting 100 bad keys to /api/auth/login returns 100 × 401 and never a 429, and the lockout only ever locks the caller out of the rest of the API. Brute-force protection is the one thing the auth surface most needs. The suite asserts nothing about this, so the fix is a free choice: register login through the limiter, or move the lockout check into the login handler.
• Auth configuration is frozen at New. api.base_path, api.trusted_proxies and the authenticator (auth.mode, api_keys) are captured once in New; a reload re-reads CORS only. Writing a new API key through the settings endpoint therefore answers ok while the running server keeps the old key, and a trusted_proxies change needs a restart. A config write endpoint that cannot apply its own auth change is an operational trap. Pin the current behaviour with a test before changing it.
• api.constantTimeEqual has no callers, and the authenticator doc comment describes a constant-time "API key scan" that does not exist. The real design is better than the comment: API keys are compared as SHA-256 digests through a hash-keyed map and the key itself is never stored, which satisfies the AGENTS.md invariant. The comment is wrong, not the code; delete the dead function and fix the comment.

Open decisions

• Unknown config keys are silently ignored. Writing block: where the struct wants blocklist: produced a plausible-looking server, and two test fixtures were caught by this rather than by what they were testing. KnownFields(true) would reject it, but that is a behaviour change for anyone carrying extra keys.
• DoH clients behind an Ingress all present as the proxy. httpRemoteAddr uses the raw RemoteAddr and never consults trusted_proxies, so every DoH client lands in the Ingress's ACL group. Fail-closed if that group is restrictive, fail-open if it is permissive. Pin the behaviour with a test before changing it.
• A character-string longer than 255 bytes cannot be packed by this version of miekg/dns (v1.1.73). TXT.len() reports a >255-byte string as zero length and EDNS0_NSID.len() is likewise wrong, so Msg.Pack fails with "dns: overflow packing opt" — or, on the server write path, silently fails and the query times out. Two consequences: a fixture that wants a multi-kilobyte answer has to build many short strings (the engine harness uses 15 x 200), and a real DoT/DoH/UDP client that sends an NSID option gets a SERVFAIL, because the engine forwards its own NSID for such a query and the message will not pack. Not fixed: it needs a library bump or a deliberate decision to answer NSID locally rather than forward it.
• "1Gi" is not a valid size — only GiB, G, GB. The parser is deliberately strict, but k8s spells it 1Gi and the k8s manifests are the likeliest place an operator writes a memory budget. Left as is and pinned by a test; worth a decision.
• metrics.OutcomeLocal has no producer. The engine reports a local-zone answer as ActionAllow, which finish maps to the upstream outcome, so locally answered queries inflate the upstream counter and the dashboard's local column is always empty. Not changed: fixing it moves published counters, so it wants a decision and a changelog note.

Continuation plan, in order

1. internal/engine suite — done, 23 tests. The original ask was the largest uncovered surface and the one every packet touches. Pin the pipeline order end to end: ACL deny before anything else, local zone before blocklist, blocklist before cache, a blocklist hit served and cached without touching the upstream, the concurrency and rate-limit refusals, serve-stale with its single background refresh, truncation against a 512-byte client buffer, and that the metrics and qlog hooks fire exactly once per query and not at all for a query rejected before the pipeline reached them. Reuse the harness in internal/app/integration_test.go.
2. internal/api suite — done, partly (78 cases): the route table, the three auth modes, the hashed-key invariant, trusted_proxies and loopback trust, read-only keys, allow_config_write: false, panic recovery. Not covered, and the harness already carries what they need (fake.stream/unsub for SSE, fake.fail for error paths, harness.srv.now as an advanceable clock): CORS, the rate limiter, the SSE open/unsubscribe/no-leak case, /metrics (enabled/protected/501), and the settings PUT/validate/reload happy paths.
3. internal/server suite — in progress. All four listeners behind one handler, per-listener config, the UDP read-deadline reset, shutdown ordering.
4. internal/metrics + internal/cluster — the collectors are done (9 tests, the metric names pinned as the contract with the dashboards); what is left is cluster: the rolling rings, the Valkey lease, pub/sub sync, divergence detection. miniredis is the obvious test server.
5. Tests for config.Size — done, and the unit-multiplier bug it was supposed to catch was real.
6. Wire memory: into deploy/k8s/00-config.yaml, matched to the StatefulSet's 1Gi limit. — done: 1GiB limit, 256MiB reserve, 384MiB cache budget.
7. Decide on KnownFields(true).
8. Re-run the console end to end; the API surface has changed several times.

Design notes worth preserving

• Cache stores packed wire bytes, not *dns.Msg. A hit costs one Unpack and yields a private copy, so serving never mutates shared state and TTL ageing is free. Same trade dnsmasq makes.
• Everything the query path reads is behind an atomic.Pointer holder — ACL, blocklist, localzone, and now the upstream pool. Readers load the pointer once and work against a value that cannot change under them. Prefer a new immutable snapshot over a lock.
• Cache invalidation on rule changes flushes the cache rather than guessing which subtrees are affected. invalidateBlocklistCache() in internal/app/hooks.go.
• API responses share the field names in internal/app/hooks.go. If you rename one, grep the frontend too.
• API keys are compared by SHA-256 hash, constant-time, and X-Forwarded-For is honoured only from trusted_proxies.
• The UDP listener resets its read deadline every iteration. A deadline set once expires for good and turns the loop into a spin that never reads another packet (caught by TestBlocklistToggle).
• Only the Valkey lease holder fetches lists, so N instances do not hit a mirror N times.
• Serve-stale answers get TTL 1; an expired upstream answer is never cached; a truncated answer is never cached (it would be served truncated forever).
• PatternKind.Pattern is the display form; WireName is the wildcard form. The internal *. / **. prefixes must not leak into the UI, but the API needs them to re-add a rule unchanged.
• pprof is loopback-only by construction, not by policy, so no config flag can turn it into a network-reachable unauthenticated endpoint.

Test-hygiene traps

• dns.NewRR needs a TTL column: dns.NewRR("host.example.com") does not parse. That is a fixture bug, not a product bug.
• A test that closes a stop channel after wg.Wait() deadlocks the whole package and looks like "the suite hangs". Close first, then wait.
• Assert on invariants, not on what a concurrent goroutine happened to observe. "A reader must have seen address X" is a flake; "every list a reader sees is one of the published ones" is a real property and cannot flake.
• tsc -b replays cached diagnostics from node_modules/.tmp/*.tsbuildinfo. After a large edit, rm node_modules/.tmp/*.tsbuildinfo.
• vite build empties outDir and deletes internal/web/dist/.gitkeep. Restore it or the directory disappears from git.
• dns.Msg with a character-string over 255 bytes does not pack in miekg/dns v1.1.73 (TXT of 4 KiB, any EDNS0_NSID). The failure is silent on a server write path, so it presents as a client-side timeout; build a big answer out of many strings under 255 bytes instead.
• gofmt -l internal/ is the fastest way to see that a file does not parse.