fancy-dns/AGENTS.md

# AGENTS.md — working on fancy-dns

Project-specific knowledge for anyone (human or agent) changing this repo.
General state and the continuation plan live in `STATUS.md`; read that first for
what is done and what is next. This file is the part that should *not* change
every week: the decisions, the invariants, and the rules that have been learned
the hard way.

## What this is

A recursive DNS server in Go with an embedded React console. Config is YAML plus
RFC 1035 master files, hot-reloaded over SIGHUP. Blocklists in four formats, a
sharded LRU cache with a bbolt snapshot for warm restart, per-client ACLs,
query logging, Prometheus metrics, and optional Valkey/Redis for cross-instance
sync.

## Layout

```
cmd/fancy-dns/main.go          entrypoint: flags, signals, SIGHUP reload
internal/
  config/       Config struct, Duration, Size, Validate, atomic Store + hooks,
                ${VAR} environment expansion in Load
  zonefile/     RFC 1035 master-file parser ($TTL/$ORIGIN, parens, serial math)
  blocklist/    Set (exact/sub/root maps), 4 parsers, Manager (fetch+compile+swap)
  cache/        Key, Entry, sharded LRU, TTL policy, serve-stale, bbolt Store
  resolver/     Upstream pool: health, circuit breakers, parallel/failover/random
  acl/          compiled groups, CIDR/PTR/tag matching, per-group domain policy
  localzone/    authoritative data from master files, wildcards, delegations
  engine/       THE QUERY PIPELINE, reverse-PTR cache
  server/       UDP, TCP, DoT, DoH listeners; all funnel into one handler
  qlog/         bbolt query-log ring + live in-memory ring + SSE fanout
  metrics/      Prometheus collectors + rolling time-series rings
  cluster/      Valkey pub/sub, leader lease, divergence detection
  kubeedit/     the console's writes to ClientGroup/BlockList/DNSZone objects
  console/      the UI/edit half: auth, cluster writes, proxy to the resolver
  kubesource/   read-only watch of the fancydns.k6n.net CRDs (DNSZone,
                ClientGroup, BlockList) -> Snapshot -> config overlay
  api/          admin REST + SSE, auth, CORS, rate limiting
  app/          wiring, lifecycle, API hook implementations, memory envelope
  web/          go:embed of the built console
web/            Vite + React + TS + Tailwind console
deploy/         example config, example zones, k8s manifests
```

### Query pipeline order (engine.Handle)

```
validate → ACL deny → per-client concurrency → global concurrency → rate limit
  → group allow/deny domains → local zone → blocklist → cache → upstream
  → cache put → finalize (EDNS0, UDP truncation) → metrics + qlog
```

Cache is a single shard-local lookup; only a miss touches the network.

## Decisions already made — do not relitigate

| Decision | Choice | Why |
|---|---|---|
| Scope | Full production | clustering, metrics, ACLs, query logs, hot reload, Docker, k8s, CI |
| Config store | YAML + DNS master files | reviewable/diffable in git |
| Console-written zone data | Its own directory, `blocklist.managed_zones`, one file per apex | the console is a real operator surface, so it needs somewhere to write; a separate directory keeps the operator's master files read-only to it, and an unset key keeps the console read-only by default |
| Cluster sync | Valkey (Redis protocol) | simple, mature Go client, easy local dev |
| Secure DNS | DoT + DoH, **no DNSSEC validation** | deliberately out of scope; AD/CD/extended-RCODE are passed through only |
| Hot cache | Sharded LRU in memory, bbolt for snapshots | no I/O on the hot path; warm restart |
| k8s networking | **No hostNetwork**, LoadBalancer Service | the server never sets SO_REUSEPORT, so two host-network pods cannot share :53 |
| Memory | Soft GOMEMLIMIT + cache byte budget | a cgroup kill is an outage; a cache miss is not |
| Scale-out | StatefulSet, **no HPA** | an HPA cannot drive a StatefulSet; scale by adding replicas |
| Ingress | A separate ClusterIP Service | a headless Service has no virtual IP for an Ingress to route to |
| Profiling | pprof refused unless admin is loopback | the handlers sit outside the API's auth middleware |

## Invariants

Break these and you break the design, not just the test.

- **Everything the query path reads is behind an `atomic.Pointer` holder** — ACL,
  blocklist, localzone, upstream pool. Readers load the pointer once and work
  against a value that cannot change under them. **A lock on the hot path is a
  regression.** Prefer building a new immutable snapshot and swapping the
  pointer.
- **Cache entries are packed wire bytes, not `*dns.Msg`.** A hit costs one
  `Unpack` and yields a private copy, so serving never mutates shared state and
  TTL ageing is free. Same trade dnsmasq makes.
- **The cache is bounded by bytes as well as by entry count.** `cache.size` alone
  bounds nothing: one cached ANY or TXT answer can be tens of kilobytes.
- **Serve-stale** answers are served with TTL 1. An expired upstream answer is
  never cached. A truncated answer is never cached — it would be served
  truncated forever.
- **The UDP read deadline is reset every iteration.** A deadline set once expires
  for good and turns the loop into a spin that never reads another packet.
- **No step in the shutdown may wait for a client.** `Stop` is: cancel, close
  every listener (not interruptible — releasing the ports is the work the next
  process start needs), then wait for the goroutines (interruptible by the
  caller's context). The wait is the step that breaks when something depends on
  a client: a connection parked in a read with no deadline in front of it never
  notices the cancellation, and one idle connection is enough to spend the whole
  grace period and then report a context error. Reach those connections with a
  read deadline in the past, **not** a close — a query already inside the
  pipeline is past its read, and its answer still has to go out.
- **Every query goroutine is registered with `s.wait`, UDP included**, so `Stop`
  returning means nothing is still inside the engine: the app snapshots the
  cache and closes the store immediately afterwards, and a query still in flight
  would be writing to a store on its way out. A UDP answer has nowhere else to
  go, so that transport also gets a short bounded drain before its socket
  closes — see `udpDrainGrace` in `internal/server/server.go`.
- **Only the Valkey lease holder fetches blocklists**, so N instances do not hit
  a mirror N times. Followers serve their compiled copy.
- **Blocklist changes flush the cache** rather than guessing which subtrees are
  affected (`invalidateBlocklistCache()` in `internal/app/hooks.go`).
- **The Kubernetes source is read-only and is an overlay, not a config.**
  `config.Store` keeps `Base()` (what the YAML says) beside the effective
  config (`Get()`, base + cluster objects). Anything that writes the config back
  to the file — `mutateConfig`, `SetAdminAddr`, the settings editor — must start
  from `Base()`, or cluster objects get copied into the YAML where they shadow
  their own objects. YAML wins every collision (groups run first; a same-named
  list is dropped). A cluster object is validated on its own: a bad object is
  skipped and keeps its last good version, it never fails the merge. A BlockList
  has no `path` and refuses `file://`, because a namespaced object must not
  choose a file the runtime reads or writes.
- **The resolver and the console are separate binaries and only the console
  writes the cluster.** `cmd/fancy-dns-console` serves the UI, authenticates,
  writes objects through `kubeedit` (validated with the runtime's own
  `kubesource.Check*`, so it cannot store what the runtime would reject) and
  proxies every other `/api` route to the resolver with its own key, dropping
  the browser's credentials. `kubeedit` is never imported by the resolver and
  `kubesource` never writes. Reads that mix the two views (`/api/clients`) are
  made consistent with the cluster in the console, because the resolver only
  sees an edit after its watch fires: a save made just after another would
  otherwise delete the first. `source` on a group says where it is defined;
  `history` rows are counters, never groups, and are not sent to the UI.
- **API keys are compared by SHA-256 hash, constant-time**, and
  `X-Forwarded-For` is honoured *only* from `trusted_proxies`. Honouring it from
  anyone lets a client spoof its identity past the ACL and the rate limiter.
- **The API response field names are shared with the frontend.** If you rename
  one, grep `web/src` too.
- **`PatternKind.Pattern` is the display form; `WireName` is the wildcard form.**
  The internal `*.` / `**.` prefixes must not leak into the UI, but the API needs
  them to re-add a rule unchanged.
- **The Go heap does not account for goroutine stacks, runtime metadata, or the
  bbolt mmap.** `memory.reserve` exists to cover them; the cache snapshot and
  query log are mmapped and live outside the heap entirely.
- **The zone writer proves its own output.** Every record is validated by the
  project's own master-file parser, the file is rendered, and the rendering is
  re-parsed and compared with the records it meant to write *before* the file
  is renamed into place. A writer bug that lost a record would otherwise show
  up as a zone that silently stopped answering.
- **A zone the console creates is a real zone.** It is authoritative, so it
  answers NXDOMAIN for the names it does not hold, and `DELETE /api/zones` is
  the only thing that hands a name back to upstream. Deleting a record is not,
  and the API never pretends it is.
- **A delegation takes the names under it.** The store answers them as
  delegated without reading the zone file, so a record at or below a cut is
  refused rather than written as data that looks alive and never answers.
- **Unknown config keys are silently ignored** (a settled decision, pinned by
  `TestLoadIgnoresUnknownKeys` in `internal/config`: `yaml`
  `KnownFields(true)` was considered and rejected as a behaviour change for
  anyone carrying extra keys). If a config key seems to have no effect, check
  the spelling against the struct tags before anything else. The leniency
  covers unknown keys only — a known key with a bad *type* is still a startup
  failure.
- **`dns.EDNS0_NSID.Nsid` is hex on the wire.** A human-readable value there
  builds a message that will not pack, so the client gets SERVFAIL. See the
  test-hygiene traps below.

## Working rules

**Never leave the tree red.** You have a hard budget and no way to detect you are
close to it. Whatever is on disk at the end of any round must compile and pass.
Get a few tests running early and keep them running. If you are running long,
stop adding tests and make the existing ones pass — a smaller green suite is worth
far more than a larger red one.

**Scope your edits.** Check `git status` before you start; other work may be in
flight. Stage only the packages you touched. `.codesearch/` is tracked but
gitignored — never stage it. If you edit a package someone else is also editing,
say so in your report rather than silently overwriting.

**Verify before you claim.** `make check-go` is the fast loop (gofmt, vet,
compile, ~1s). `make check` adds the full test suite. Push only a green tree.

**Commit and push when the task says so.** Read `git log --oneline -5` and
`git show --stat HEAD` first and match the style: an imperative subject line, then
one paragraph per bug covering root cause and fix. A commit that says *what* was
changed without saying *why* is not finished work.

**When a test fails, decide which of three things is wrong before editing
anything:** the test, the implementation, or the specification. All three happen
here. Say which one you concluded and why. Do not relax an assertion to make a
suite pass — if an existing assertion pins buggy behaviour, change it and say so
explicitly in the commit message, because that decision belongs in the history.

**Report honestly.** If you did not finish, say what is unfinished and what state
the tree is in. A partial result that is accurately described is usable; a
partial result described as done is not.

## Test-hygiene traps

- `dns.NewRR` needs a TTL column: `dns.NewRR("host.example.com")` does not parse.
  A failure there is a bad fixture, not a product bug.
- A test that closes a stop channel *after* `wg.Wait()` deadlocks the whole
  package and looks like "the suite hangs". Close first, then wait.
- Assert on invariants, not on what a concurrent goroutine happened to observe.
  "A reader must have seen address X" is a flake; "every list a reader sees is one
  of the published ones" is a real property and cannot flake.
- **A test that only dials a stream transport races the accept loop.** A
  shutdown can close the listener while the connection is still in the backlog,
  so the server never accepts it and the test passes for the wrong reason.
  Exchange one query first, then leave the connection idle
  (`harness.idleStream`).
- **Count leaked goroutines by stack, not by `runtime.NumGoroutine()`.** The
  test binary shares its process with net/http, crypto/tls and the race
  detector, so a baseline count is noise. "Is any goroutine still running code
  in this package?" is a property of the product — and it needs the caller's own
  goroutine filtered out by id, or every run finds exactly one.
- Spin up `dns.Server` fakes on `127.0.0.1:0`: `net.Listen`/`net.ListenPacket` to
  find the port, `ActivateAndServe` in a goroutine, stop in `t.Cleanup`.
- A `dns.Msg` carrying a `TXT` string over 255 bytes **does not pack** in
  miekg/dns v1.1.73 — the error is `dns: string exceeded 255 bytes in txt`, and
  255 is the exact boundary. On a fake upstream's write path that failure is
  silent and presents as a client timeout or a SERVFAIL several layers away.
  Build a multi-kilobyte answer out of many strings under 255 bytes, and assert
  the fake's write error before blaming the client.
- **`dns.EDNS0_NSID.Nsid` is a hex string, not the bytes on the wire.** The
  library hex-decodes it in `pack` and hex-encodes it in `unpack`, so a plain
  value such as `"fancy-dns"` fails with `dns: overflow packing opt` and a
  long NSID packs fine (unlike `TXT`). This bit the engine: an unencoded
  literal made every query from an NSID-using client SERVFAIL. Always
  hex-encode on the way in; `TestPinnedEDNS0NSIDIsHex` in
  `internal/engine/pins_test.go` pins the contract.
- `internal/app/integration_test.go` already builds a fully wired app. Reuse its
  harness rather than rebuilding one.
- `tsc -b` replays **cached diagnostics** from `node_modules/.tmp/*.tsbuildinfo`.
  After a large frontend edit: `rm node_modules/.tmp/*.tsbuildinfo`.
- `vite build` empties `outDir` and **deletes `internal/web/dist/.gitkeep`**.
  Restore it or the directory disappears from git.
- The Go binary embeds `internal/web/dist`, so `npm ci && npm run build` must run
  before a fresh clone will build a binary with the console in it.
- `make build` depends on `web` and shells out to npm. `make go-build` skips the
  console and is what you want in an inner loop.

## Security-sensitive areas

These have been wrong before. Changes here need a test that fails without the fix.

- **Auth** (`internal/api/auth.go`): API key comparison must stay constant-time;
  the key is hashed before comparison so the plaintext is not in a long-lived
  string.
- **Client identity**: `X-Forwarded-For` is only trusted from `trusted_proxies`,
  and `httpRemoteAddr` (used by the DoH handler) deliberately does **not**
  consult it at all. Behind an Ingress every DoH client therefore lands in the
  proxy's ACL group — fail-closed if that group is restrictive, fail-open if it
  is permissive. Pin this behaviour with a test before changing it.
- **Config expansion**: `${VAR}` is expanded in `Load` and an unset variable
  with no default is fatal, so a missing Secret is a crash loop rather than an
  unauthenticated console. The expansion works inside quoted scalars — that is
  the form operators write — and skips YAML comments. Do not "simplify" it into a
  plain `os.ExpandEnv`; that silently breaks the API key path.
- **pprof** is loopback-only by construction, so no config flag can turn it into
  a network-reachable unauthenticated endpoint. Keep it that way.
- **Amplification**: the DoH/UDP handlers cap the client's advertised EDNS0
  buffer. Do not remove the cap.