fancy-dns/STATUS.md

# fancy-dns — development status and continuation plan

Last updated when the whole tree went green. `AGENTS.md` holds the decisions,
invariants and working rules; this file holds state and the plan.

## Goal

A lightweight, high-performance recursive DNS server with:

- **Go backend** — sharded LRU cache backed by bbolt, blocklist replication from
  multiple formats, per-client ACLs, query logging, Prometheus metrics.
- **Secure DNS** — DNS-over-TLS (RFC 7858) and DNS-over-HTTPS (RFC 8484).
  DNSSEC validation was explicitly scoped **out**; AD/CD/extended-RCODE are
  passed through only.
- **React + Tailwind admin console** embedded in the binary.
- **Config as zone files + YAML**, hot-reloadable, with **Valkey/Redis** for
  cross-instance sync (chosen over AMQP).

## Architecture decisions (settled with the user)

| Decision | Choice | Why |
|---|---|---|
| Scope | Full production | clustering, metrics, ACLs, query logs, hot reload, Docker, k8s, CI |
| Config store | YAML + DNS master files | reviewable/diffable in git |
| Cluster sync | Valkey (Redis protocol) | simple, mature Go client, easy local dev |
| Secure DNS | DoT + DoH, no DNSSEC validation | avoids signer/anchor machinery; pass-through only |
| Hot cache | Sharded LRU in memory, bbolt for snapshots | no I/O on the hot path; warm restart |
| k8s networking | No hostNetwork, LoadBalancer Service | the server never sets SO_REUSEPORT, so two host-network pods cannot share :53 |
| Memory | Soft GOMEMLIMIT, cache byte budget | a cgroup kill is an outage; a cache miss is not |
| Scale-out | StatefulSet, no HPA | an HPA cannot drive a StatefulSet |

## Package layout

```
cmd/fancy-dns/main.go          entrypoint: flags, signals, SIGHUP reload
internal/
  config/       Config struct, Duration, Size, Validate, atomic Store + hooks,
                ${VAR} environment expansion in Load
  zonefile/     RFC 1035 master-file parser ($TTL/$ORIGIN, parens, serial math)
  blocklist/    Set (exact/sub/root maps), 4 parsers, Manager (fetch+compile+swap)
  cache/        Key, Entry, sharded LRU, TTL policy, serve-stale, bbolt Store
  resolver/     Upstream pool: health, circuit breakers, parallel/failover/random
  acl/          compiled groups, CIDR/PTR/tag matching, per-group domain policy
  localzone/    authoritative data from master files, wildcards, delegations
  engine/       THE QUERY PIPELINE (see below), reverse-PTR cache
  server/       UDP, TCP, DoT, DoH listeners; all funnel into one handler
  qlog/         bbolt query-log ring + live in-memory ring + SSE fanout
  metrics/      Prometheus collectors + rolling time-series rings
  cluster/      Valkey pub/sub, leader lease, divergence detection
  api/          admin REST + SSE, auth, CORS, rate limiting
  app/          wiring, lifecycle, API hook implementations, memory envelope
  web/          go:embed of the built console
web/            Vite + React + TS + Tailwind console
deploy/         example config, example zones, k8s manifests
```

### Query pipeline order (engine.Handle)

```
validate → ACL deny → per-client concurrency → global concurrency → rate limit
  → group allow/deny domains → local zone → blocklist → cache → upstream
  → cache put → finalize (EDNS0, UDP truncation) → metrics + qlog
```

Cache is a single shard-local lookup; only a miss touches the network.

## Status

### Green — the whole tree passes

`gofmt` clean, `go build ./...` clean, `go vet` clean.

`go test -race ./internal/... -count=1 -timeout 300s` is **green across every
package that has a suite**: `acl`, `app`, `blocklist`, `cache`, `config`,
`localzone`, `qlog`, `resolver`, `zonefile` — roughly 190 tests.

`make check-go` is the inner loop (1.3 s: gofmt check, vet, compile). It uses
`go-build`, not `build`, because `build` depends on `web` and shells out to npm.
`make check` = `check-go test`.

### No test suite yet

`api`, `cluster`, **`engine`**, `metrics`, `server`, `web`.

`engine` is the notable one: it is the query pipeline every packet goes through
and it has no test file at all. An attempt at one did not land, so treat it as
greenfield rather than an extension of an existing suite.

### Fixed recently and worth not re-breaking

- **Every relative RDATA name in a master file pointed at the DNS root.** The
  qualification pass ran *after* `dns.NewRR`, and the library resolves RDATA
  names against the root, so `www CNAME host` became `CNAME host.` — the alias
  pointed at the root, and so did every relative NS, MX, PTR, SRV, NAPTR and SOA
  target in every zone file. It is now a token-level pass driven by a table of
  which RDATA positions hold a name; dotted names stay absolute, so the RPZ
  targets `.` and `rpz-drop.` and the null MX/SRV targets are untouched.
- **The cache snapshot never round-tripped.** `Snapshot` packed the whole key
  while `Load` decoded the record header out of the *key's* remainder instead of
  the value, so all 25 entries were skipped — every restart was a cold cache.
  There is now an explicit `encodeRecord`/`decodeRecord` pair with a version byte
  and a CRC-32C trailer. `Load` drops entries already past `ExpiresAt` and
  rewrites the TTL to the time actually left, so a served TTL never jumps back up
  across a restart.
- **`${VAR}` expansion never worked inside quotes** — which is the only form
  anyone writes. The reference check sat in the same switch as quote tracking,
  below the `inDouble`/`inSingle` branches, so `api_keys: ["${KEY}"]` came out of
  `Load` unchanged and the admin API key was the literal string
  `"${FANCYDNS_API_KEY}"`. The k8s manifest looked correct and was not. A variable
  that is set but empty now expands to empty rather than erroring (the
  `${X}` / `${X-y}` / `${X:-y}` distinction); `\${FOO}` is literal.
- **ACL PTR suffix matched across label boundaries**, so a group scoped to
  `example.com.` also swallowed `ample.com.` — a client could land in a group it
  does not belong to.
- **The ACL `*` catch-all was dropped** from the rendered domain patterns, so the
  engine's per-query recompile lost the "everything" case.
- **`Upstream.Success` never closed an open breaker.** A recovered upstream stayed
  excluded for the full cooldown — the mirror of the older bug where a failed
  exchange marked a dead upstream healthy.
- **`Exchange` fanned out in parallel even under `policy: failover`**, because the
  parallel path was only taken when exactly one upstream was chosen.
- **The cache byte budget was never fed to the LRU.** `maxBytes` was enforced per
  shard on insert but had no setter and nothing called one, so `cache.size` had no
  predictable memory cost.
- **The memory envelope** (`memory.limit/reserve/cache_max_bytes`) is new, wired to
  `debug.SetMemoryLimit`. The reserve covers what the Go heap does not account
  for: goroutine stacks, runtime metadata, and the bbolt mmap.
- **Blocklist:** leading BOM stripped, `!allow` works, `SourceMeta` exists for a
  rule-only source, `Pattern()` reports the bare name (`WireName()` gives the
  wildcard form for the API), RPZ inherits the zone default TTL, hosts files
  complain about a missing address, adblock handles CRLF and over-long lines.
- **`$TTL` was ignored** by the zone parser; every record got the SOA minimum.
- **`TestRebindUnderConcurrentQueries` was a flake** (2 runs in 10) — it asserted
  a reader had to observe two specific addresses, which is a claim about what a
  goroutine happened to see. It now pins the copy-on-write invariant itself:
  every observed snapshot must equal one of the published lists.
- **`TestHolderSwapIsAtomic` had the wrong invariant**, not the Holder.
- **CI could not have passed:** `go.mod` requires Go 1.27.1, CI pinned 1.23 and
  the Dockerfile pinned `golang:1.23-alpine`.
- **The Dockerfile clobbered the console** — it copied the source tree after the
  console bundle.
- **The k8s set could not have worked:** `hostNetwork: true` with `replicas: 2`
  and no SO_REUSEPORT. Now a LoadBalancer Service, a separate ClusterIP Service
  for the Ingress, a ServiceAccount, an API-key Secret, per-pod PVCs, a
  kustomization. The HPA is gone.
- **`serveUDP` called `net.ResolveUDPAddr` per packet**; the peer already is a
  `*net.UDPAddr`.

### Open decisions

- **Unknown config keys are silently ignored.** Writing `block:` where the struct
  wants `blocklist:` produced a plausible-looking server, and two test fixtures
  were caught by this rather than by what they were testing. `KnownFields(true)`
  would reject it, but that is a behaviour change for anyone carrying extra keys.
- **DoH clients behind an Ingress all present as the proxy.** `httpRemoteAddr`
  uses the raw `RemoteAddr` and never consults `trusted_proxies`, so every DoH
  client lands in the Ingress's ACL group. Fail-closed if that group is
  restrictive, fail-open if it is permissive. Pin the behaviour with a test
  before changing it.

## Continuation plan, in order

1. **`internal/engine` suite** — the largest uncovered surface and the one every
   packet touches. Pin the pipeline order end to end: ACL deny before anything
   else, local zone before blocklist, blocklist before cache, a blocklist hit
   served and cached without touching the upstream, the concurrency and
   rate-limit refusals, serve-stale with its single background refresh,
   truncation against a 512-byte client buffer, and that the metrics and qlog
   hooks fire exactly once per query and not at all for a query rejected before
   the pipeline reached them. Reuse the harness in
   `internal/app/integration_test.go`.
2. **`internal/api` suite** — every route, the three auth modes, the constant-time
   API-key comparison, `X-Forwarded-For` honoured *only* from `trusted_proxies`,
   CORS, rate limiting, and the SSE stream (open a subscriber, assert delivery,
   unsubscribe, assert no goroutine leak).
3. **`internal/server` suite** — all four listeners behind one handler,
   per-listener config, the UDP read-deadline reset, shutdown ordering.
4. **`internal/metrics` + `internal/cluster`** — the collectors, the rolling
   rings, the Valkey lease, pub/sub sync, divergence detection. `miniredis` is the
   obvious test server.
5. Tests for `config.Size` — the newest parsing code in the repo and the most
   likely to be wrong at a boundary like `1.5GiB`.
6. Wire `memory:` into `deploy/k8s/00-config.yaml`, matched to the StatefulSet's
   1Gi limit.
7. Decide on `KnownFields(true)`.
8. Re-run the console end to end; the API surface has changed several times.

## Design notes worth preserving

- **Cache stores packed wire bytes, not `*dns.Msg`.** A hit costs one `Unpack`
  and yields a private copy, so serving never mutates shared state and TTL
  ageing is free. Same trade dnsmasq makes.
- **Everything the query path reads is behind an `atomic.Pointer` holder** — ACL,
  blocklist, localzone, and now the upstream pool. Readers load the pointer once
  and work against a value that cannot change under them. Prefer a new immutable
  snapshot over a lock.
- **Cache invalidation on rule changes flushes the cache** rather than guessing
  which subtrees are affected. `invalidateBlocklistCache()` in
  `internal/app/hooks.go`.
- **API responses share the field names in `internal/app/hooks.go`.** If you
  rename one, grep the frontend too.
- **API keys are compared by SHA-256 hash, constant-time**, and
  `X-Forwarded-For` is honoured *only* from `trusted_proxies`.
- **The UDP listener resets its read deadline every iteration.** A deadline set
  once expires for good and turns the loop into a spin that never reads another
  packet (caught by `TestBlocklistToggle`).
- **Only the Valkey lease holder fetches lists**, so N instances do not hit a
  mirror N times.
- **Serve-stale** answers get TTL 1; an expired upstream answer is never cached;
  a truncated answer is never cached (it would be served truncated forever).
- **`PatternKind.Pattern` is the display form; `WireName` is the wildcard form.**
  The internal `*.` / `**.` prefixes must not leak into the UI, but the API
  needs them to re-add a rule unchanged.
- **pprof is loopback-only by construction**, not by policy, so no config flag
  can turn it into a network-reachable unauthenticated endpoint.

## Test-hygiene traps

- `dns.NewRR` needs a TTL column: `dns.NewRR("host.example.com")` does not
  parse. That is a fixture bug, not a product bug.
- A test that closes a stop channel *after* `wg.Wait()` deadlocks the whole
  package and looks like "the suite hangs". Close first, then wait.
- Assert on invariants, not on what a concurrent goroutine happened to observe.
  "A reader must have seen address X" is a flake; "every list a reader sees is one
  of the published ones" is a real property and cannot flake.
- `tsc -b` replays **cached diagnostics** from `node_modules/.tmp/*.tsbuildinfo`.
  After a large edit, `rm node_modules/.tmp/*.tsbuildinfo`.
- `vite build` empties `outDir` and **deletes `internal/web/dist/.gitkeep`**.
  Restore it or the directory disappears from git.
- `gofmt -l internal/` is the fastest way to see that a file does not parse.