fancy-dns/STATUS.md

fancy-dns — development status and continuation plan

Last updated when the whole tree went green. AGENTS.md holds the decisions, invariants and working rules; this file holds state and the plan.

Goal

A lightweight, high-performance recursive DNS server with:

• Go backend — sharded LRU cache backed by bbolt, blocklist replication from multiple formats, per-client ACLs, query logging, Prometheus metrics.
• Secure DNS — DNS-over-TLS (RFC 7858) and DNS-over-HTTPS (RFC 8484). DNSSEC validation was explicitly scoped out; AD/CD/extended-RCODE are passed through only.
• React + Tailwind admin console embedded in the binary.
• Config as zone files + YAML, hot-reloadable, with Valkey/Redis for cross-instance sync (chosen over AMQP).

Architecture decisions (settled with the user)

Decision	Choice	Why
Scope	Full production	clustering, metrics, ACLs, query logs, hot reload, Docker, k8s, CI
Config store	YAML + DNS master files	reviewable/diffable in git
Cluster sync	Valkey (Redis protocol)	simple, mature Go client, easy local dev
Secure DNS	DoT + DoH, no DNSSEC validation	avoids signer/anchor machinery; pass-through only
Hot cache	Sharded LRU in memory, bbolt for snapshots	no I/O on the hot path; warm restart
k8s networking	No hostNetwork, LoadBalancer Service	the server never sets SO_REUSEPORT, so two host-network pods cannot share :53
Memory	Soft GOMEMLIMIT, cache byte budget	a cgroup kill is an outage; a cache miss is not
Scale-out	StatefulSet, no HPA	an HPA cannot drive a StatefulSet

Package layout

cmd/fancy-dns/main.go          entrypoint: flags, signals, SIGHUP reload
internal/
  config/       Config struct, Duration, Size, Validate, atomic Store + hooks,
                ${VAR} environment expansion in Load
  zonefile/     RFC 1035 master-file parser ($TTL/$ORIGIN, parens, serial math)
  blocklist/    Set (exact/sub/root maps), 4 parsers, Manager (fetch+compile+swap)
  cache/        Key, Entry, sharded LRU, TTL policy, serve-stale, bbolt Store
  resolver/     Upstream pool: health, circuit breakers, parallel/failover/random
  acl/          compiled groups, CIDR/PTR/tag matching, per-group domain policy
  localzone/    authoritative data from master files, wildcards, delegations
  engine/       THE QUERY PIPELINE (see below), reverse-PTR cache
  server/       UDP, TCP, DoT, DoH listeners; all funnel into one handler
  qlog/         bbolt query-log ring + live in-memory ring + SSE fanout
  metrics/      Prometheus collectors + rolling time-series rings
  cluster/      Valkey pub/sub, leader lease, divergence detection
  api/          admin REST + SSE, auth, CORS, rate limiting
  app/          wiring, lifecycle, API hook implementations, memory envelope
  web/          go:embed of the built console
web/            Vite + React + TS + Tailwind console
deploy/         example config, example zones, k8s manifests

Query pipeline order (engine.Handle)

validate → ACL deny → per-client concurrency → global concurrency → rate limit
  → group allow/deny domains → local zone → blocklist → cache → upstream
  → cache put → finalize (EDNS0, UDP truncation) → metrics + qlog

Cache is a single shard-local lookup; only a miss touches the network.

Status

Green — the whole tree passes

gofmt clean, go build ./... clean, go vet clean.

go test -race ./internal/... -count=1 -timeout 300s is green across every package that has a suite: acl, app, blocklist, cache, config, localzone, qlog, resolver, zonefile — roughly 190 tests.

make check-go is the inner loop (1.3 s: gofmt check, vet, compile). It uses go-build, not build, because build depends on web and shells out to npm. make check = check-go test.

No test suite yet

api, cluster, engine, metrics, server, web.

engine is the notable one: it is the query pipeline every packet goes through and it has no test file at all. An attempt at one did not land, so treat it as greenfield rather than an extension of an existing suite.

Fixed recently and worth not re-breaking

• Every relative RDATA name in a master file pointed at the DNS root. The qualification pass ran after dns.NewRR, and the library resolves RDATA names against the root, so www CNAME host became CNAME host. — the alias pointed at the root, and so did every relative NS, MX, PTR, SRV, NAPTR and SOA target in every zone file. It is now a token-level pass driven by a table of which RDATA positions hold a name; dotted names stay absolute, so the RPZ targets . and rpz-drop. and the null MX/SRV targets are untouched.
• The cache snapshot never round-tripped. Snapshot packed the whole key while Load decoded the record header out of the key's remainder instead of the value, so all 25 entries were skipped — every restart was a cold cache. There is now an explicit encodeRecord/decodeRecord pair with a version byte and a CRC-32C trailer. Load drops entries already past ExpiresAt and rewrites the TTL to the time actually left, so a served TTL never jumps back up across a restart.
• ${VAR} expansion never worked inside quotes — which is the only form anyone writes. The reference check sat in the same switch as quote tracking, below the inDouble/inSingle branches, so api_keys: ["${KEY}"] came out of Load unchanged and the admin API key was the literal string "${FANCYDNS_API_KEY}". The k8s manifest looked correct and was not. A variable that is set but empty now expands to empty rather than erroring (the ${X} / ${X-y} / ${X:-y} distinction); \${FOO} is literal.
• ACL PTR suffix matched across label boundaries, so a group scoped to example.com. also swallowed ample.com. — a client could land in a group it does not belong to.
• The ACL * catch-all was dropped from the rendered domain patterns, so the engine's per-query recompile lost the "everything" case.
• Upstream.Success never closed an open breaker. A recovered upstream stayed excluded for the full cooldown — the mirror of the older bug where a failed exchange marked a dead upstream healthy.
• Exchange fanned out in parallel even under policy: failover, because the parallel path was only taken when exactly one upstream was chosen.
• The cache byte budget was never fed to the LRU. maxBytes was enforced per shard on insert but had no setter and nothing called one, so cache.size had no predictable memory cost.
• The memory envelope (memory.limit/reserve/cache_max_bytes) is new, wired to debug.SetMemoryLimit. The reserve covers what the Go heap does not account for: goroutine stacks, runtime metadata, and the bbolt mmap.
• Blocklist: leading BOM stripped, !allow works, SourceMeta exists for a rule-only source, Pattern() reports the bare name (WireName() gives the wildcard form for the API), RPZ inherits the zone default TTL, hosts files complain about a missing address, adblock handles CRLF and over-long lines.
• $TTL was ignored by the zone parser; every record got the SOA minimum.
• TestRebindUnderConcurrentQueries was a flake (2 runs in 10) — it asserted a reader had to observe two specific addresses, which is a claim about what a goroutine happened to see. It now pins the copy-on-write invariant itself: every observed snapshot must equal one of the published lists.
• TestHolderSwapIsAtomic had the wrong invariant, not the Holder.
• CI could not have passed: go.mod requires Go 1.27.1, CI pinned 1.23 and the Dockerfile pinned golang:1.23-alpine.
• The Dockerfile clobbered the console — it copied the source tree after the console bundle.
• The k8s set could not have worked: hostNetwork: true with replicas: 2 and no SO_REUSEPORT. Now a LoadBalancer Service, a separate ClusterIP Service for the Ingress, a ServiceAccount, an API-key Secret, per-pod PVCs, a kustomization. The HPA is gone.
• serveUDP called net.ResolveUDPAddr per packet; the peer already is a *net.UDPAddr.

Open decisions

• Unknown config keys are silently ignored. Writing block: where the struct wants blocklist: produced a plausible-looking server, and two test fixtures were caught by this rather than by what they were testing. KnownFields(true) would reject it, but that is a behaviour change for anyone carrying extra keys.
• DoH clients behind an Ingress all present as the proxy. httpRemoteAddr uses the raw RemoteAddr and never consults trusted_proxies, so every DoH client lands in the Ingress's ACL group. Fail-closed if that group is restrictive, fail-open if it is permissive. Pin the behaviour with a test before changing it.

Continuation plan, in order

1. internal/engine suite — the largest uncovered surface and the one every packet touches. Pin the pipeline order end to end: ACL deny before anything else, local zone before blocklist, blocklist before cache, a blocklist hit served and cached without touching the upstream, the concurrency and rate-limit refusals, serve-stale with its single background refresh, truncation against a 512-byte client buffer, and that the metrics and qlog hooks fire exactly once per query and not at all for a query rejected before the pipeline reached them. Reuse the harness in internal/app/integration_test.go.
2. internal/api suite — every route, the three auth modes, the constant-time API-key comparison, X-Forwarded-For honoured only from trusted_proxies, CORS, rate limiting, and the SSE stream (open a subscriber, assert delivery, unsubscribe, assert no goroutine leak).
3. internal/server suite — all four listeners behind one handler, per-listener config, the UDP read-deadline reset, shutdown ordering.
4. internal/metrics + internal/cluster — the collectors, the rolling rings, the Valkey lease, pub/sub sync, divergence detection. miniredis is the obvious test server.
5. Tests for config.Size — the newest parsing code in the repo and the most likely to be wrong at a boundary like 1.5GiB.
6. Wire memory: into deploy/k8s/00-config.yaml, matched to the StatefulSet's 1Gi limit.
7. Decide on KnownFields(true).
8. Re-run the console end to end; the API surface has changed several times.

Design notes worth preserving

• Cache stores packed wire bytes, not *dns.Msg. A hit costs one Unpack and yields a private copy, so serving never mutates shared state and TTL ageing is free. Same trade dnsmasq makes.
• Everything the query path reads is behind an atomic.Pointer holder — ACL, blocklist, localzone, and now the upstream pool. Readers load the pointer once and work against a value that cannot change under them. Prefer a new immutable snapshot over a lock.
• Cache invalidation on rule changes flushes the cache rather than guessing which subtrees are affected. invalidateBlocklistCache() in internal/app/hooks.go.
• API responses share the field names in internal/app/hooks.go. If you rename one, grep the frontend too.
• API keys are compared by SHA-256 hash, constant-time, and X-Forwarded-For is honoured only from trusted_proxies.
• The UDP listener resets its read deadline every iteration. A deadline set once expires for good and turns the loop into a spin that never reads another packet (caught by TestBlocklistToggle).
• Only the Valkey lease holder fetches lists, so N instances do not hit a mirror N times.
• Serve-stale answers get TTL 1; an expired upstream answer is never cached; a truncated answer is never cached (it would be served truncated forever).
• PatternKind.Pattern is the display form; WireName is the wildcard form. The internal *. / **. prefixes must not leak into the UI, but the API needs them to re-add a rule unchanged.
• pprof is loopback-only by construction, not by policy, so no config flag can turn it into a network-reachable unauthenticated endpoint.

Test-hygiene traps

• dns.NewRR needs a TTL column: dns.NewRR("host.example.com") does not parse. That is a fixture bug, not a product bug.
• A test that closes a stop channel after wg.Wait() deadlocks the whole package and looks like "the suite hangs". Close first, then wait.
• Assert on invariants, not on what a concurrent goroutine happened to observe. "A reader must have seen address X" is a flake; "every list a reader sees is one of the published ones" is a real property and cannot flake.
• tsc -b replays cached diagnostics from node_modules/.tmp/*.tsbuildinfo. After a large edit, rm node_modules/.tmp/*.tsbuildinfo.
• vite build empties outDir and deletes internal/web/dist/.gitkeep. Restore it or the directory disappears from git.
• gofmt -l internal/ is the fastest way to see that a file does not parse.