fancy-dns/AGENTS.md

AGENTS.md — working on fancy-dns

Project-specific knowledge for anyone (human or agent) changing this repo. General state and the continuation plan live in STATUS.md; read that first for what is done and what is next. This file is the part that should not change every week: the decisions, the invariants, and the rules that have been learned the hard way.

What this is

A recursive DNS server in Go with an embedded React console. Config is YAML plus RFC 1035 master files, hot-reloaded over SIGHUP. Blocklists in four formats, a sharded LRU cache with a bbolt snapshot for warm restart, per-client ACLs, an access log on stdout, Prometheus metrics, and optional Valkey/Redis for cross-instance sync.

Layout

cmd/fancy-dns/main.go          entrypoint: flags, signals, SIGHUP reload
internal/
  config/       Config struct, Duration, Size, Validate, atomic Store + hooks,
                ${VAR} environment expansion in Load
  zonefile/     RFC 1035 master-file parser ($TTL/$ORIGIN, parens, serial math)
  blocklist/    Set (exact/sub/root maps), 4 parsers, Manager (fetch+compile+swap)
  cache/        Key, Entry, sharded LRU, TTL policy, serve-stale, bbolt Store
  resolver/     Upstream pool: health, circuit breakers, parallel/failover/random
  acl/          compiled groups, CIDR/PTR/tag matching, per-group domain policy
  localzone/    authoritative data from master files, wildcards, delegations
  engine/       THE QUERY PIPELINE, reverse-PTR cache
  server/       UDP, TCP, DoT, DoH listeners; all funnel into one handler
  accesslog/    async JSON-lines access log to stdout (blocked/errors by default)
  metrics/      Prometheus collectors + rolling time-series rings
  cluster/      Valkey pub/sub, leader lease, divergence detection
  kubeedit/     the console's writes to ClientGroup/BlockList/DNSZone objects
  console/      the UI/edit half: auth, cluster writes, proxy to the resolver
  kubesource/   read-only watch of the fancydns.k6n.net CRDs (DNSZone,
                ClientGroup, BlockList) -> Snapshot -> config overlay
  api/          admin REST + SSE, auth, CORS, rate limiting
  app/          wiring, lifecycle, API hook implementations, memory envelope
  web/          go:embed of the built console
web/            Vite + React + TS + Tailwind console
deploy/         example config, example zones, k8s manifests

Query pipeline order (engine.Handle)

validate → ACL deny → per-client concurrency → global concurrency → rate limit
  → group allow/deny domains → local zone → blocklist → cache → upstream
  → cache put → finalize (EDNS0, UDP truncation) → metrics + access log

Cache is a single shard-local lookup; only a miss touches the network.

Decisions already made — do not relitigate

Decision	Choice	Why
Scope	Full production	clustering, metrics, ACLs, access log, hot reload, Docker, k8s, CI
Config store	YAML + DNS master files	reviewable/diffable in git
Console-written zone data	Its own directory, blocklist.managed_zones, one file per apex	the console is a real operator surface, so it needs somewhere to write; a separate directory keeps the operator's master files read-only to it, and an unset key keeps the console read-only by default
Cluster sync	Valkey (Redis protocol)	simple, mature Go client, easy local dev
Secure DNS	DoT + DoH, no DNSSEC validation	deliberately out of scope; AD/CD/extended-RCODE are passed through only
Observability	Prometheus metrics + a JSON access log on stdout; no in-app dashboard, log store or live tail	the resolver is DNS only. Charts belong in Grafana, search in the log collector, and per-pod state in a multi-node deployment only disagrees with itself. The access log is blocked/errors by default, goes through a bounded buffer that drops (and counts) rather than ever blocking a query, and nothing may be added to the query path to feed a UI
Hot cache	Sharded LRU in memory, bbolt for snapshots	no I/O on the hot path; warm restart
k8s networking	No hostNetwork, LoadBalancer Service	the server never sets SO_REUSEPORT, so two host-network pods cannot share :53
Memory	Soft GOMEMLIMIT + cache byte budget	a cgroup kill is an outage; a cache miss is not
Scale-out	StatefulSet, no HPA	an HPA cannot drive a StatefulSet; scale by adding replicas
Ingress	A separate ClusterIP Service	a headless Service has no virtual IP for an Ingress to route to
Profiling	pprof refused unless admin is loopback	the handlers sit outside the API's auth middleware

Invariants

Break these and you break the design, not just the test.

• Everything the query path reads is behind an atomic.Pointer holder — ACL, blocklist, localzone, upstream pool. Readers load the pointer once and work against a value that cannot change under them. A lock on the hot path is a regression. Prefer building a new immutable snapshot and swapping the pointer.
• Cache entries are packed wire bytes, not *dns.Msg. A hit costs one Unpack and yields a private copy, so serving never mutates shared state and TTL ageing is free. Same trade dnsmasq makes.
• The cache is bounded by bytes as well as by entry count. cache.size alone bounds nothing: one cached ANY or TXT answer can be tens of kilobytes.
• Serve-stale answers are served with TTL 1. An expired upstream answer is never cached. A truncated answer is never cached — it would be served truncated forever.
• The UDP read deadline is reset every iteration. A deadline set once expires for good and turns the loop into a spin that never reads another packet.
• No step in the shutdown may wait for a client. Stop is: cancel, close every listener (not interruptible — releasing the ports is the work the next process start needs), then wait for the goroutines (interruptible by the caller's context). The wait is the step that breaks when something depends on a client: a connection parked in a read with no deadline in front of it never notices the cancellation, and one idle connection is enough to spend the whole grace period and then report a context error. Reach those connections with a read deadline in the past, not a close — a query already inside the pipeline is past its read, and its answer still has to go out.
• Every query goroutine is registered with s.wait, UDP included, so Stop returning means nothing is still inside the engine: the app snapshots the cache and closes the store immediately afterwards, and a query still in flight would be writing to a store on its way out. A UDP answer has nowhere else to go, so that transport also gets a short bounded drain before its socket closes — see udpDrainGrace in internal/server/server.go.
• Only the Valkey lease holder fetches blocklists, so N instances do not hit a mirror N times. Followers serve their compiled copy.
• Blocklist changes flush the cache rather than guessing which subtrees are affected (invalidateBlocklistCache() in internal/app/hooks.go).
• The Kubernetes source is read-only and is an overlay, not a config. config.Store keeps Base() (what the YAML says) beside the effective config (Get(), base + cluster objects). Anything that writes the config back to the file — mutateConfig, SetAdminAddr, the settings editor — must start from Base(), or cluster objects get copied into the YAML where they shadow their own objects. YAML wins every collision (groups run first; a same-named list is dropped). A cluster object is validated on its own: a bad object is skipped and keeps its last good version, it never fails the merge. A BlockList has no path and refuses file://, because a namespaced object must not choose a file the runtime reads or writes.
• The resolver and the console are separate binaries and only the console writes the cluster. cmd/fancy-dns-console serves the UI, authenticates, writes objects through kubeedit (validated with the runtime's own kubesource.Check*, so it cannot store what the runtime would reject) and proxies every other /api route to the resolver with its own key, dropping the browser's credentials. kubeedit is never imported by the resolver and kubesource never writes. Reads that mix the two views (/api/clients) are made consistent with the cluster in the console, because the resolver only sees an edit after its watch fires: a save made just after another would otherwise delete the first. source on a group says where it is defined; history rows are counters, never groups, and are not sent to the UI.
• API keys are compared by SHA-256 hash, constant-time, and X-Forwarded-For is honoured only from trusted_proxies. Honouring it from anyone lets a client spoof its identity past the ACL and the rate limiter.
• The API response field names are shared with the frontend. If you rename one, grep web/src too.
• PatternKind.Pattern is the display form; WireName is the wildcard form. The internal *. / **. prefixes must not leak into the UI, but the API needs them to re-add a rule unchanged.
• The Go heap does not account for goroutine stacks, runtime metadata, or the bbolt mmap. memory.reserve exists to cover them; the cache snapshot is mmapped and lives outside the heap entirely.
• The zone writer proves its own output. Every record is validated by the project's own master-file parser, the file is rendered, and the rendering is re-parsed and compared with the records it meant to write before the file is renamed into place. A writer bug that lost a record would otherwise show up as a zone that silently stopped answering.
• A zone the console creates is a real zone. It is authoritative, so it answers NXDOMAIN for the names it does not hold, and DELETE /api/zones is the only thing that hands a name back to upstream. Deleting a record is not, and the API never pretends it is.
• A delegation takes the names under it. The store answers them as delegated without reading the zone file, so a record at or below a cut is refused rather than written as data that looks alive and never answers.
• Unknown config keys are silently ignored (a settled decision, pinned by TestLoadIgnoresUnknownKeys in internal/config: yaml KnownFields(true) was considered and rejected as a behaviour change for anyone carrying extra keys). If a config key seems to have no effect, check the spelling against the struct tags before anything else. The leniency covers unknown keys only — a known key with a bad type is still a startup failure.
• dns.EDNS0_NSID.Nsid is hex on the wire. A human-readable value there builds a message that will not pack, so the client gets SERVFAIL. See the test-hygiene traps below.

Working rules

Never leave the tree red. You have a hard budget and no way to detect you are close to it. Whatever is on disk at the end of any round must compile and pass. Get a few tests running early and keep them running. If you are running long, stop adding tests and make the existing ones pass — a smaller green suite is worth far more than a larger red one.

Scope your edits. Check git status before you start; other work may be in flight. Stage only the packages you touched. .codesearch/ is tracked but gitignored — never stage it. If you edit a package someone else is also editing, say so in your report rather than silently overwriting.

Verify before you claim. make check-go is the fast loop (gofmt, vet, compile, ~1s). make check adds the full test suite. Push only a green tree.

Commit and push when the task says so. Read git log --oneline -5 and git show --stat HEAD first and match the style: an imperative subject line, then one paragraph per bug covering root cause and fix. A commit that says what was changed without saying why is not finished work.

When a test fails, decide which of three things is wrong before editing anything: the test, the implementation, or the specification. All three happen here. Say which one you concluded and why. Do not relax an assertion to make a suite pass — if an existing assertion pins buggy behaviour, change it and say so explicitly in the commit message, because that decision belongs in the history.

Report honestly. If you did not finish, say what is unfinished and what state the tree is in. A partial result that is accurately described is usable; a partial result described as done is not.

Test-hygiene traps

• dns.NewRR needs a TTL column: dns.NewRR("host.example.com") does not parse. A failure there is a bad fixture, not a product bug.
• A test that closes a stop channel after wg.Wait() deadlocks the whole package and looks like "the suite hangs". Close first, then wait.
• Assert on invariants, not on what a concurrent goroutine happened to observe. "A reader must have seen address X" is a flake; "every list a reader sees is one of the published ones" is a real property and cannot flake.
• A test that only dials a stream transport races the accept loop. A shutdown can close the listener while the connection is still in the backlog, so the server never accepts it and the test passes for the wrong reason. Exchange one query first, then leave the connection idle (harness.idleStream).
• Count leaked goroutines by stack, not by runtime.NumGoroutine(). The test binary shares its process with net/http, crypto/tls and the race detector, so a baseline count is noise. "Is any goroutine still running code in this package?" is a property of the product — and it needs the caller's own goroutine filtered out by id, or every run finds exactly one.
• Spin up dns.Server fakes on 127.0.0.1:0: net.Listen/net.ListenPacket to find the port, ActivateAndServe in a goroutine, stop in t.Cleanup.
• A dns.Msg carrying a TXT string over 255 bytes does not pack in miekg/dns v1.1.73 — the error is dns: string exceeded 255 bytes in txt, and 255 is the exact boundary. On a fake upstream's write path that failure is silent and presents as a client timeout or a SERVFAIL several layers away. Build a multi-kilobyte answer out of many strings under 255 bytes, and assert the fake's write error before blaming the client.
• dns.EDNS0_NSID.Nsid is a hex string, not the bytes on the wire. The library hex-decodes it in pack and hex-encodes it in unpack, so a plain value such as "fancy-dns" fails with dns: overflow packing opt and a long NSID packs fine (unlike TXT). This bit the engine: an unencoded literal made every query from an NSID-using client SERVFAIL. Always hex-encode on the way in; TestPinnedEDNS0NSIDIsHex in internal/engine/pins_test.go pins the contract.
• internal/app/integration_test.go already builds a fully wired app. Reuse its harness rather than rebuilding one.
• tsc -b replays cached diagnostics from node_modules/.tmp/*.tsbuildinfo. After a large frontend edit: rm node_modules/.tmp/*.tsbuildinfo.
• vite build empties outDir and deletes internal/web/dist/.gitkeep. Restore it or the directory disappears from git.
• The Go binary embeds internal/web/dist, so npm ci && npm run build must run before a fresh clone will build a binary with the console in it.
• make build depends on web and shells out to npm. make go-build skips the console and is what you want in an inner loop.

Security-sensitive areas

These have been wrong before. Changes here need a test that fails without the fix.

• Auth (internal/api/auth.go): API key comparison must stay constant-time; the key is hashed before comparison so the plaintext is not in a long-lived string.
• Client identity: X-Forwarded-For is only trusted from trusted_proxies, and httpRemoteAddr (used by the DoH handler) deliberately does not consult it at all. Behind an Ingress every DoH client therefore lands in the proxy's ACL group — fail-closed if that group is restrictive, fail-open if it is permissive. Pin this behaviour with a test before changing it.
• Config expansion: ${VAR} is expanded in Load and an unset variable with no default is fatal, so a missing Secret is a crash loop rather than an unauthenticated console. The expansion works inside quoted scalars — that is the form operators write — and skips YAML comments. Do not "simplify" it into a plain os.ExpandEnv; that silently breaks the API key path.
• pprof is loopback-only by construction, so no config flag can turn it into a network-reachable unauthenticated endpoint. Keep it that way.
• Amplification: the DoH/UDP handlers cap the client's advertised EDNS0 buffer. Do not remove the cap.