fancy-dns/STATUS.md

# fancy-dns — development status and continuation plan

Last updated when the whole tree went green. `AGENTS.md` holds the decisions,
invariants and working rules; this file holds state and the plan.

## Goal

A lightweight, high-performance recursive DNS server with:

- **Go backend** — sharded LRU cache backed by bbolt, blocklist replication from
  multiple formats, per-client ACLs, query logging, Prometheus metrics.
- **Secure DNS** — DNS-over-TLS (RFC 7858) and DNS-over-HTTPS (RFC 8484).
  DNSSEC validation was explicitly scoped **out**; AD/CD/extended-RCODE are
  passed through only.
- **React + Tailwind admin console** embedded in the binary.
- **Config as zone files + YAML**, hot-reloadable, with **Valkey/Redis** for
  cross-instance sync (chosen over AMQP).

## Architecture decisions (settled with the user)

| Decision | Choice | Why |
|---|---|---|
| Scope | Full production | clustering, metrics, ACLs, query logs, hot reload, Docker, k8s, CI |
| Config store | YAML + DNS master files | reviewable/diffable in git |
| Cluster sync | Valkey (Redis protocol) | simple, mature Go client, easy local dev |
| Secure DNS | DoT + DoH, no DNSSEC validation | avoids signer/anchor machinery; pass-through only |
| Hot cache | Sharded LRU in memory, bbolt for snapshots | no I/O on the hot path; warm restart |
| k8s networking | No hostNetwork, LoadBalancer Service | the server never sets SO_REUSEPORT, so two host-network pods cannot share :53 |
| Memory | Soft GOMEMLIMIT, cache byte budget | a cgroup kill is an outage; a cache miss is not |
| Scale-out | StatefulSet, no HPA | an HPA cannot drive a StatefulSet |

## Package layout

```
cmd/fancy-dns/main.go          entrypoint: flags, signals, SIGHUP reload
internal/
  config/       Config struct, Duration, Size, Validate, atomic Store + hooks,
                ${VAR} environment expansion in Load
  zonefile/     RFC 1035 master-file parser ($TTL/$ORIGIN, parens, serial math)
  blocklist/    Set (exact/sub/root maps), 4 parsers, Manager (fetch+compile+swap)
  cache/        Key, Entry, sharded LRU, TTL policy, serve-stale, bbolt Store
  resolver/     Upstream pool: health, circuit breakers, parallel/failover/random
  acl/          compiled groups, CIDR/PTR/tag matching, per-group domain policy
  localzone/    authoritative data from master files, wildcards, delegations
  engine/       THE QUERY PIPELINE (see below), reverse-PTR cache
  server/       UDP, TCP, DoT, DoH listeners; all funnel into one handler
  qlog/         bbolt query-log ring + live in-memory ring + SSE fanout
  metrics/      Prometheus collectors + rolling time-series rings
  cluster/      Valkey pub/sub, leader lease, divergence detection
  api/          admin REST + SSE, auth, CORS, rate limiting
  app/          wiring, lifecycle, API hook implementations, memory envelope
  web/          go:embed of the built console
web/            Vite + React + TS + Tailwind console
deploy/         example config, example zones, k8s manifests
```

### Query pipeline order (engine.Handle)

```
validate → ACL deny → per-client concurrency → global concurrency → rate limit
  → group allow/deny domains → local zone → blocklist → cache → upstream
  → cache put → finalize (EDNS0, UDP truncation) → metrics + qlog
```

Cache is a single shard-local lookup; only a miss touches the network.

## Status

### Green — the whole tree passes

`gofmt` clean, `go build ./...` clean, `go vet` clean.

`go test -race ./internal/... -count=1 -timeout 300s` is **green across every
package that has a suite**: `acl`, `app`, `blocklist`, `cache`, `config`,
`engine`, `localzone`, `metrics`, `qlog`, `resolver`, `zonefile` — roughly 230
tests — and `api` joined the list with 78 cases of its own.

`make check-go` is the inner loop (1.3 s: gofmt check, vet, compile). It uses
`go-build`, not `build`, because `build` depends on `web` and shells out to npm.
`make check` = `check-go test`.

### No test suite yet

`cluster`, `server`, `web`.

`internal/engine` is now covered — 23 tests over a harness (`harness_test.go`)
that builds a wired engine with a fake upstream on loopback, a local zone, a
blocklist, a cache and a query log — and `internal/config` has a `Size` suite.
`internal/metrics` has a counter/gauge/exposition suite (9 tests) that pins the
metric names, which are the contract with the Grafana dashboards and the
ServiceMonitor. `internal/api` is covered too: every route in the table (each
asserted to reach its hook, so a new route without a case is a route nothing
tests), method/path rejection, a read-route sweep that asserts nothing mutated,
panic recovery, the base path, all three auth modes, read-only keys and
`allow_config_write: false`, loopback trust, and the hashed API keys.
`internal/server` is the one in progress.

## Fixed in the last round (the engine and `config.Size` work)

- **`config.ParseSize` ignored the unit multiplier entirely.** `scaleSize` took
  the number and returned `Size(n)`; `mult` never entered the arithmetic, so
  `512MiB` parsed as **512 bytes** and `1GiB` as 1 byte. Every memory budget
  written in units — `memory.limit`, `memory.reserve`, `memory.cache_max_bytes`,
  the cache byte budget in the app wiring — was off by a factor of the unit,
  and `EffectiveLimit`'s 64 MiB floor then quietly replaced the operator's
  number. The memory envelope was inert for anyone who wrote a unit, which is
  everyone: a bare number is bytes. The fix is one multiplication;
  `size_test.go` now pins `1.5GiB`, `0.5KiB`, the unit shadowing cases (`MiB`
  must not match `M`), the out-of-range error and the truncating rounding.
- **A cache miss and a stale hit moved only the Prometheus counter.**
  `metrics.RecordMiss` and `metrics.RecordStaleHit` bumped the collector and
  not the atomics that `Totals` reads, so the Cache page's "misses" and "stale
  hits" read 0 for the life of the process while the Grafana panel was right —
  exactly the disagreement `Record`'s doc comment says cannot happen.
- **The uncached path never truncated.** `finalize` — the EDNS0/UDP-size step
  that turns an oversized answer into a question-only TC reply — was called
  only on the *cache hit* path. A multi-kilobyte answer from upstream, from the
  local zone or from a sinkhole left here as a datagram no 512-byte UDP client
  can receive, and the client silently dropped it. All four reply paths now
  finalize.
- **The serve-stale prefetch gate was not single-flight.**
  `LRU.prefetchGate` was `make(chan struct{}, 64)`, so a burst of stale hits on
  one hot name launched up to 64 duplicate upstream exchanges instead of the
  one its own doc comment promised. It is a per-key latch now (a
  `map[Key]struct{}` under a mutex): distinct stale names still refresh in
  parallel, one name does not. `TryPrefetch`/`ReleasePrefetch` take the key.
- **The upstream UDP client read buffer was hard-coded to 1232** while the
  engine advertises `dns.max_udp_size` upstream. An operator who raised
  `max_udp_size` above 1232 got a SERVFAIL for *every* answer over 1232 bytes:
  the datagram was read truncated and failed to unpack with "dns: overflowing
  header size". `resolver.NewPoolWithUDPSize` takes the advertised size and the
  app passes `cfg.DNS.MaxUDPSize`; `NewPool` keeps the 1232 default.

### Fixed recently and worth not re-breaking

- **Every relative RDATA name in a master file pointed at the DNS root.** The
  qualification pass ran *after* `dns.NewRR`, and the library resolves RDATA
  names against the root, so `www CNAME host` became `CNAME host.` — the alias
  pointed at the root, and so did every relative NS, MX, PTR, SRV, NAPTR and SOA
  target in every zone file. It is now a token-level pass driven by a table of
  which RDATA positions hold a name; dotted names stay absolute, so the RPZ
  targets `.` and `rpz-drop.` and the null MX/SRV targets are untouched.
- **The cache snapshot never round-tripped.** `Snapshot` packed the whole key
  while `Load` decoded the record header out of the *key's* remainder instead of
  the value, so all 25 entries were skipped — every restart was a cold cache.
  There is now an explicit `encodeRecord`/`decodeRecord` pair with a version byte
  and a CRC-32C trailer. `Load` drops entries already past `ExpiresAt` and
  rewrites the TTL to the time actually left, so a served TTL never jumps back up
  across a restart.
- **`${VAR}` expansion never worked inside quotes** — which is the only form
  anyone writes. The reference check sat in the same switch as quote tracking,
  below the `inDouble`/`inSingle` branches, so `api_keys: ["${KEY}"]` came out of
  `Load` unchanged and the admin API key was the literal string
  `"${FANCYDNS_API_KEY}"`. The k8s manifest looked correct and was not. A variable
  that is set but empty now expands to empty rather than erroring (the
  `${X}` / `${X-y}` / `${X:-y}` distinction); `\${FOO}` is literal.
- **ACL PTR suffix matched across label boundaries**, so a group scoped to
  `example.com.` also swallowed `ample.com.` — a client could land in a group it
  does not belong to.
- **The ACL `*` catch-all was dropped** from the rendered domain patterns, so the
  engine's per-query recompile lost the "everything" case.
- **`Upstream.Success` never closed an open breaker.** A recovered upstream stayed
  excluded for the full cooldown — the mirror of the older bug where a failed
  exchange marked a dead upstream healthy.
- **`Exchange` fanned out in parallel even under `policy: failover`**, because the
  parallel path was only taken when exactly one upstream was chosen.
- **The cache byte budget was never fed to the LRU.** `maxBytes` was enforced per
  shard on insert but had no setter and nothing called one, so `cache.size` had no
  predictable memory cost.
- **The memory envelope** (`memory.limit/reserve/cache_max_bytes`) is new, wired to
  `debug.SetMemoryLimit`. The reserve covers what the Go heap does not account
  for: goroutine stacks, runtime metadata, and the bbolt mmap.
- **Blocklist:** leading BOM stripped, `!allow` works, `SourceMeta` exists for a
  rule-only source, `Pattern()` reports the bare name (`WireName()` gives the
  wildcard form for the API), RPZ inherits the zone default TTL, hosts files
  complain about a missing address, adblock handles CRLF and over-long lines.
- **`$TTL` was ignored** by the zone parser; every record got the SOA minimum.
- **`TestRebindUnderConcurrentQueries` was a flake** (2 runs in 10) — it asserted
  a reader had to observe two specific addresses, which is a claim about what a
  goroutine happened to see. It now pins the copy-on-write invariant itself:
  every observed snapshot must equal one of the published lists.
- **`TestHolderSwapIsAtomic` had the wrong invariant**, not the Holder.
- **CI could not have passed:** `go.mod` requires Go 1.27.1, CI pinned 1.23 and
  the Dockerfile pinned `golang:1.23-alpine`.
- **The Dockerfile clobbered the console** — it copied the source tree after the
  console bundle.
- **The k8s set could not have worked:** `hostNetwork: true` with `replicas: 2`
  and no SO_REUSEPORT. Now a LoadBalancer Service, a separate ClusterIP Service
  for the Ingress, a ServiceAccount, an API-key Secret, per-pod PVCs, a
  kustomization. The HPA is gone.
- **`serveUDP` called `net.ResolveUDPAddr` per packet**; the peer already is a
  `*net.UDPAddr`.

## New findings from the `api` suite, unfixed

- **The login rate limiter never applies to the login endpoint.** `handleLogin`
  is registered through `public`, which bypasses `authed` and therefore
  `allowRequest` — the only reader of `lockedTill`, and `handleLogin` the only
  writer of `failures`. So `api.rate_limit.per_minute`, documented as "requests
  per minute per IP **for the login endpoint**", applies to every *other* route
  instead: posting 100 bad keys to `/api/auth/login` returns 100 × 401 and never
  a 429, and the lockout only ever locks the caller out of the rest of the API.
  Brute-force protection is the one thing the auth surface most needs. The suite
  asserts nothing about this, so the fix is a free choice: register login
  through the limiter, or move the lockout check into the login handler.
- **Auth configuration is frozen at `New`.** `api.base_path`,
  `api.trusted_proxies` and the authenticator (`auth.mode`, `api_keys`) are
  captured once in `New`; a reload re-reads CORS only. Writing a new API key
  through the settings endpoint therefore answers `ok` while the running server
  keeps the old key, and a `trusted_proxies` change needs a restart. A config
  write endpoint that cannot apply its own auth change is an operational trap.
  Pin the current behaviour with a test before changing it.
- **`api.constantTimeEqual` has no callers**, and the `authenticator` doc comment
  describes a constant-time "API key scan" that does not exist. The real design
  is better than the comment: API keys are compared as SHA-256 digests through a
  hash-keyed map and the key itself is never stored, which satisfies the
  AGENTS.md invariant. The comment is wrong, not the code; delete the dead
  function and fix the comment.

### Open decisions

- **Unknown config keys are silently ignored.** Writing `block:` where the struct
  wants `blocklist:` produced a plausible-looking server, and two test fixtures
  were caught by this rather than by what they were testing. `KnownFields(true)`
  would reject it, but that is a behaviour change for anyone carrying extra keys.
- **DoH clients behind an Ingress all present as the proxy.** `httpRemoteAddr`
  uses the raw `RemoteAddr` and never consults `trusted_proxies`, so every DoH
  client lands in the Ingress's ACL group. Fail-closed if that group is
  restrictive, fail-open if it is permissive. Pin the behaviour with a test
  before changing it.
- **A character-string longer than 255 bytes cannot be packed by this version
  of `miekg/dns` (v1.1.73).** `TXT.len()` reports a >255-byte string as zero
  length and `EDNS0_NSID.len()` is likewise wrong, so `Msg.Pack` fails with
  "dns: overflow packing opt" — or, on the server write path, silently fails
  and the query times out. Two consequences: a fixture that wants a
  multi-kilobyte answer has to build many short strings (the engine harness
  uses 15 x 200), and a **real DoT/DoH/UDP client that sends an NSID option
  gets a SERVFAIL**, because the engine forwards its own NSID for such a query
  and the message will not pack. Not fixed: it needs a library bump or a
  deliberate decision to answer NSID locally rather than forward it.
- **`"1Gi"` is not a valid size** — only `GiB`, `G`, `GB`. The parser is
  deliberately strict, but k8s spells it `1Gi` and the k8s manifests are the
  likeliest place an operator writes a memory budget. Left as is and pinned by
  a test; worth a decision.
- **`metrics.OutcomeLocal` has no producer.** The engine reports a local-zone
  answer as `ActionAllow`, which `finish` maps to the *upstream* outcome, so
  locally answered queries inflate the upstream counter and the dashboard's
  local column is always empty. Not changed: fixing it moves published
  counters, so it wants a decision and a changelog note.

## Continuation plan, in order

1. ~~**`internal/engine` suite**~~ — **done**, 23 tests. The original ask was
   the largest uncovered surface and the one every
   packet touches. Pin the pipeline order end to end: ACL deny before anything
   else, local zone before blocklist, blocklist before cache, a blocklist hit
   served and cached without touching the upstream, the concurrency and
   rate-limit refusals, serve-stale with its single background refresh,
   truncation against a 512-byte client buffer, and that the metrics and qlog
   hooks fire exactly once per query and not at all for a query rejected before
   the pipeline reached them. Reuse the harness in
   `internal/app/integration_test.go`.
2. ~~**`internal/api` suite**~~ — **done, partly** (78 cases): the route table,
   the three auth modes, the hashed-key invariant, `trusted_proxies` and
   loopback trust, read-only keys, `allow_config_write: false`, panic recovery.
   **Not covered**, and the harness already carries what they need
   (`fake.stream`/`unsub` for SSE, `fake.fail` for error paths, `harness.srv.now`
   as an advanceable clock): CORS, the rate limiter, the SSE
   open/unsubscribe/no-leak case, `/metrics` (enabled/protected/501), and the
   settings PUT/validate/reload happy paths.
3. **`internal/server` suite** — in progress. All four listeners behind one handler,
   per-listener config, the UDP read-deadline reset, shutdown ordering.
4. **`internal/metrics` + `internal/cluster`** — the collectors are **done** (9
   tests, the metric names pinned as the contract with the dashboards); what is
   left is `cluster`: the rolling rings, the Valkey lease, pub/sub sync,
   divergence detection. `miniredis` is the obvious test server.
5. ~~Tests for `config.Size`~~ — **done**, and the unit-multiplier bug it was
   supposed to catch was real.
6. ~~Wire `memory:` into `deploy/k8s/00-config.yaml`, matched to the
   StatefulSet's 1Gi limit.~~ — **done**: `1GiB` limit, `256MiB` reserve,
   `384MiB` cache budget.
7. Decide on `KnownFields(true)`.
8. Re-run the console end to end; the API surface has changed several times.

## Design notes worth preserving

- **Cache stores packed wire bytes, not `*dns.Msg`.** A hit costs one `Unpack`
  and yields a private copy, so serving never mutates shared state and TTL
  ageing is free. Same trade dnsmasq makes.
- **Everything the query path reads is behind an `atomic.Pointer` holder** — ACL,
  blocklist, localzone, and now the upstream pool. Readers load the pointer once
  and work against a value that cannot change under them. Prefer a new immutable
  snapshot over a lock.
- **Cache invalidation on rule changes flushes the cache** rather than guessing
  which subtrees are affected. `invalidateBlocklistCache()` in
  `internal/app/hooks.go`.
- **API responses share the field names in `internal/app/hooks.go`.** If you
  rename one, grep the frontend too.
- **API keys are compared by SHA-256 hash, constant-time**, and
  `X-Forwarded-For` is honoured *only* from `trusted_proxies`.
- **The UDP listener resets its read deadline every iteration.** A deadline set
  once expires for good and turns the loop into a spin that never reads another
  packet (caught by `TestBlocklistToggle`).
- **Only the Valkey lease holder fetches lists**, so N instances do not hit a
  mirror N times.
- **Serve-stale** answers get TTL 1; an expired upstream answer is never cached;
  a truncated answer is never cached (it would be served truncated forever).
- **`PatternKind.Pattern` is the display form; `WireName` is the wildcard form.**
  The internal `*.` / `**.` prefixes must not leak into the UI, but the API
  needs them to re-add a rule unchanged.
- **pprof is loopback-only by construction**, not by policy, so no config flag
  can turn it into a network-reachable unauthenticated endpoint.

## Test-hygiene traps

- `dns.NewRR` needs a TTL column: `dns.NewRR("host.example.com")` does not
  parse. That is a fixture bug, not a product bug.
- A test that closes a stop channel *after* `wg.Wait()` deadlocks the whole
  package and looks like "the suite hangs". Close first, then wait.
- Assert on invariants, not on what a concurrent goroutine happened to observe.
  "A reader must have seen address X" is a flake; "every list a reader sees is one
  of the published ones" is a real property and cannot flake.
- `tsc -b` replays **cached diagnostics** from `node_modules/.tmp/*.tsbuildinfo`.
  After a large edit, `rm node_modules/.tmp/*.tsbuildinfo`.
- `vite build` empties `outDir` and **deletes `internal/web/dist/.gitkeep`**.
  Restore it or the directory disappears from git.
- **`dns.Msg` with a character-string over 255 bytes does not pack** in
  miekg/dns v1.1.73 (`TXT` of 4 KiB, any `EDNS0_NSID`). The failure is silent on
  a server write path, so it presents as a client-side timeout; build a big
  answer out of many strings under 255 bytes instead.
- `gofmt -l internal/` is the fastest way to see that a file does not parse.