Fix the bbolt round trip, the blocklist gaps and the deploy set

The cache snapshot never loaded, so every restart was a cold cache. Snapshot
packed the whole key while Load decoded the record header out of the key's
remainder instead of the value, so every entry was skipped. The two sides are
now an explicit encodeRecord/decodeRecord pair with a version byte and a CRC-32C
trailer, so a future format change is detected instead of silently misparsed.
Load also drops entries that are already past ExpiresAt and rewrites the TTL to
the time actually left, so a served TTL never jumps back up across a restart.

Blocklist, all six gaps the failing tests were pointing at: a leading UTF-8 BOM
is stripped, "!allow example.com" works, SourceMeta exists for a source that
only ever contributed a rule, Pattern() reports the bare name again (the new
WireName() produces the wildcard form for the API), an RPZ record with no
explicit TTL inherits the zone default instead of whatever the reader left in
the RR, a hosts line with no address in the first field says so, and the
adblock parser handles CRLF and adblock-flavoured lines pasted into a plain
list. TestHolderSwapIsAtomic was asserting the wrong invariant — the swap was
always atomic, a name blocked by the old set is legitimately unblocked by the
new one.

Tests added for acl and resolver, both of which turned up real bugs:

- a PTR suffix matched across label boundaries, so a group scoped to
  example.com. also swallowed ample.com. and notexample.com. — a client could
  land in a group it does not belong to;
- the "*" catch-all was dropped from the rendered domain patterns, so the
  engine's per-query recompile lost the "everything" case;
- Upstream.Success never closed an open breaker, so a recovered upstream stayed
  excluded for the full cooldown;
- Exchange fanned out in parallel even under policy: failover.

Deploy set: go.mod requires Go 1.27.1 but CI pinned 1.23 and the Dockerfile
pinned golang:1.23-alpine, so neither could have built. The Dockerfile also
copied the source tree after the console bundle, which clobbered it.

The k8s set asked for hostNetwork with two replicas, but the server never sets
SO_REUSEPORT, so the second pod could not bind :53 and would crash-loop. It now
runs without hostNetwork behind a LoadBalancer Service, with a separate
ClusterIP Service for the Ingress (a headless Service cannot back one), a
ServiceAccount, an API-key Secret, per-pod PVCs, and a kustomization. The HPA
is gone: it cannot drive a StatefulSet. To let the API key come from that
Secret, config.Load now expands ${VAR} / ${VAR:-default} / ${NAME-default} and
fails to start on an unset variable with no default, so a missing secret is a
crash loop rather than an unauthenticated console. Expansion skips YAML
comments, so documenting ${VAR} in the ConfigMap does not break the file.

Verified: gofmt, go build, go vet clean; go test -race ./internal/... green
(acl, app, blocklist, cache, config, resolver, zonefile).

deploy/k8s/00-config.yaml

@@ -91,11 +91,14 @@ data:
       base_path: /api
       ui: true
       allow_config_write: false
-      # Read the key from a Secret and mount it; the value here is a placeholder
-      # that MUST be replaced before this reaches a network.
+      # The key comes from the environment (a Secret mounted into the pod), not
+      # from this ConfigMap, so `kubectl get cm` never prints it. The loader
+      # substitutes ${VAR} before parsing and *fails to start* if the variable
+      # is unset, so a missing secret is a crash loop rather than an
+      # unauthenticated console.
       auth:
         mode: apikey
-        api_keys: ["REPLACE-ME"]
+        api_keys: ["${FANCYDNS_API_KEY}"]
         trust_loopback: false
 
     metrics:

deploy/k8s/05-serviceaccount.yaml

@@ -0,0 +1,12 @@
+# A stable identity for the pods. The server never talks to the Kubernetes API
+# (blocklist replication goes through Valkey, not the API server), so the token
+# is not just unused but actively undesirable: automounting it would hand every
+# pod a credential it has no reason to hold.
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: fancy-dns
+  namespace: dns
+  labels:
+    app.kubernetes.io/name: fancy-dns
+automountServiceAccountToken: false

deploy/k8s/08-secret-apikey.yaml

@@ -0,0 +1,22 @@
+# The admin API key. It is read from the environment, not from the ConfigMap,
+# so the key never appears in `kubectl get configmap -o yaml`.
+#
+#   kubectl -n dns create secret generic fancy-dns-apikey \
+#     --from-literal=api-key="$(openssl rand -hex 32)"
+#
+# With auth.mode: apikey and api_keys: ["${FANCYDNS_API_KEY}"] in the config,
+# a missing FANCYDNS_API_KEY is a startup error rather than an empty key, so a
+# forgotten secret shows up as a CrashLoopBackOff and not as an unauthenticated
+# console on the network.
+apiVersion: v1
+kind: Secret
+metadata:
+  name: fancy-dns-apikey
+  namespace: dns
+  labels:
+    app.kubernetes.io/name: fancy-dns
+type: Opaque
+stringData:
+  # PLACEHOLDER — replace before this reaches a cluster. `kubectl apply` on this
+  # file as-is will overwrite a real key with this value.
+  api-key: REPLACE-ME

deploy/k8s/10-deployment.yaml

@@ -7,6 +7,11 @@ metadata:
     app.kubernetes.io/name: fancy-dns
 spec:
   serviceName: fancy-dns
+  # A StatefulSet rather than a Deployment because each pod owns a PVC (the
+  # cache snapshot and the query log are per-pod state worth keeping across a
+  # reschedule). The flip side: the HPA does not work on a StatefulSet — the
+  # controller and the autoscaler fight over the replica count — so scaling
+  # means adding nodes and replicas here, not CPU targets.
   replicas: 2
   podManagementPolicy: Parallel
   selector:
@@ -18,14 +23,19 @@ spec:
         app.kubernetes.io/name: fancy-dns
       annotations:
         # A config change is a SIGHUP, not a restart: the server re-reads the
-        # file and hot-swaps every component.
+        # file and hot-swaps every component. This annotation is for
+        # stakater/reloader-class controllers; without one, `kubectl rollout
+        # restart` is the manual equivalent.
         reloader.stakater.com/auto: "true"
     spec:
-      # DNS on a non-privileged port needs NET_BIND_SERVICE on Linux, or
-      # hostNetwork. hostNetwork is used here so the node's port 53 is served
-      # directly and the load balancer does not add a hop.
-      hostNetwork: true
-      dnsPolicy: Default
+      serviceAccountName: fancy-dns
+      # No hostNetwork. The server does not set SO_REUSEPORT, so a second
+      # host-network pod would fail to bind :53 and crash-loop; the ports are
+      # served by a LoadBalancer Service instead (see 20-service.yaml).
+      dnsPolicy: ClusterFirst
+      # The app drains in-flight queries for up to 15s on SIGTERM (see the
+      # shutdown timeout in cmd/fancy-dns/main.go); give it room beyond that.
+      terminationGracePeriodSeconds: 30
       securityContext:
         runAsNonRoot: true
         runAsUser: 65532
@@ -47,6 +57,14 @@ spec:
           args:
             - -config
             - /etc/fancy-dns/config.yaml
+          env:
+            # Substituted into the config by the loader, so the admin API key
+            # comes from the Secret and not from the ConfigMap.
+            - name: FANCYDNS_API_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: fancy-dns-apikey
+                  key: api-key
           ports:
             - name: dns-udp
               containerPort: 53
@@ -68,6 +86,10 @@ spec:
             readOnlyRootFilesystem: true
             capabilities:
               drop: ["ALL"]
+              # Only because of the low ports. The pod is not privileged and
+              # has no other capability left; drop the Service and the
+              # Capability together if you move the listeners off 53/443.
+              add: ["NET_BIND_SERVICE"]
           # Liveness: the process is up. /api/healthz is unauthenticated on
           # purpose, because a probe cannot carry an API key.
           livenessProbe:
@@ -85,9 +107,9 @@ spec:
             initialDelaySeconds: 3
             periodSeconds: 5
             failureThreshold: 2
-          # Startup: binding :53 and fetching the first blocklist can take a
-          # while on a cold node, so give it room before the liveness probe
-          # starts killing it.
+          # Startup: binding the listeners and fetching the first blocklist
+          # can take a while on a cold node, so give it room before the
+          # liveness probe starts killing it.
           startupProbe:
             httpGet:
               path: /api/healthz
@@ -123,6 +145,10 @@ spec:
         - name: config
           configMap:
             name: fancy-dns-config
+        # Local zones are ordinary master files; ship them in a ConfigMap or
+        # manage them through the admin API (`/api/zones`), which writes into
+        # the data volume. Optional because the glob in the config matches
+        # nothing cleanly.
         - name: zones
           configMap:
             name: fancy-dns-zones
@@ -130,116 +156,24 @@ spec:
         - name: tls
           secret:
             secretName: fancy-dns-tls
-        - name: data
-          emptyDir: {}
         - name: tmp
           emptyDir: {}
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: fancy-dns
-  namespace: dns
-  labels:
-    app.kubernetes.io/name: fancy-dns
-spec:
-  clusterIP: None
-  selector:
-    app.kubernetes.io/name: fancy-dns
-  ports:
-    - name: dns-udp
-      port: 53
-      targetPort: dns-udp
-      protocol: UDP
-    - name: dns-tcp
-      port: 53
-      targetPort: dns-tcp
-      protocol: TCP
-    - name: dot
-      port: 853
-      targetPort: dot
-    - name: doh
-      port: 443
-      targetPort: doh
-    - name: admin
-      port: 8080
-      targetPort: admin
----
-# The admin console and the DoH endpoint share a port only if the ingress
-# terminates TLS; when it does, this routes /api and /dns-query to the same
-# backend. Kept separate from the Service above because a headless Service
-# cannot front an Ingress.
-apiVersion: networking.k8s.io/v1
-kind: Ingress
-metadata:
-  name: fancy-dns
-  namespace: dns
-  annotations:
-    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
-    nginx.ingress.kubernetes.io/proxy-buffering: "off"
-    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
-spec:
-  rules:
-    - host: dns.example.com
-      http:
-        paths:
-          - path: /api
-            pathType: Prefix
-            backend:
-              service:
-                name: fancy-dns
-                port:
-                  name: admin
-          - path: /dns-query
-            pathType: Prefix
-            backend:
-              service:
-                name: fancy-dns
-                port:
-                  name: doh
-          - path: /
-            pathType: Prefix
-            backend:
-              service:
-                name: fancy-dns
-                port:
-                  name: admin
-  tls:
-    - hosts: [dns.example.com]
-      secretName: fancy-dns-tls
----
-apiVersion: policy/v1
-kind: PodDisruptionBudget
-metadata:
-  name: fancy-dns
-  namespace: dns
-spec:
-  minAvailable: 1
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: fancy-dns
----
-apiVersion: autoscaling/v2
-kind: HorizontalPodAutoscaler
-metadata:
-  name: fancy-dns
-  namespace: dns
-spec:
-  scaleTargetRef:
-    apiVersion: apps/v1
-    kind: StatefulSet
-    name: fancy-dns
-  minReplicas: 2
-  maxReplicas: 6
-  metrics:
-    - type: Resource
-      resource:
-        name: cpu
-        target:
-          type: Utilization
-          averageUtilization: 70
-  behavior:
-    scaleDown:
-      # A DNS server that flaps is worse than one that is briefly oversized:
-      # every scale-down drops the local cache and the blocklist lease.
-      stabilizationWindowSeconds: 600
+  volumeClaimTemplates:
+    # Cache snapshot (cache.persist.path) and query log (qlog.path) live here,
+    # along with downloaded blocklists. ReadWriteOnce is per-pod: a pod keeps
+    # its own cache across a reschedule, which is the whole point of the
+    # snapshot, but it cannot be shared between replicas.
+    #
+    # Sizing: cache.size is an entry count, not bytes, so disk usage has to be
+    # estimated. A packed A/AAAA answer with a small authority section is
+    # roughly 300-600 bytes on the wire and nearer 1 kB resident, so 500k
+    # entries lands around 500 MB; qlog is the other consumer (see
+    # qlog.max_entries). 4Gi leaves room for both plus a re-fetched list.
+    - metadata:
+        name: data
+      spec:
+        accessModes: ["ReadWriteOnce"]
+        resources:
+          requests:
+            storage: 4Gi
+        # storageClassName: <your default class>

deploy/k8s/20-service.yaml

@@ -0,0 +1,103 @@
+---
+# Governing Service for the StatefulSet. Headless, and never used for traffic:
+# it exists so pods get a stable DNS name, and so a PVC per pod has a home.
+apiVersion: v1
+kind: Service
+metadata:
+  name: fancy-dns
+  namespace: dns
+  labels:
+    app.kubernetes.io/name: fancy-dns
+spec:
+  clusterIP: None
+  selector:
+    app.kubernetes.io/name: fancy-dns
+  ports:
+    - name: dns-udp
+      port: 53
+      targetPort: dns-udp
+      protocol: UDP
+    - name: dns-tcp
+      port: 53
+      targetPort: dns-tcp
+      protocol: TCP
+    - name: dot
+      port: 853
+      targetPort: dot
+    - name: doh
+      port: 443
+      targetPort: doh
+    - name: admin
+      port: 8080
+      targetPort: admin
+---
+# The DNS service. This is the one clients resolve.
+#
+# Why a LoadBalancer and not hostNetwork: the server does not set SO_REUSEPORT,
+# so two host-network pods cannot share the node's :53 and the second one
+# would crash-loop on bind. Giving the cloud load balancer the port also keeps
+# the node's own resolver untouched, which matters on a node that runs kubelet
+# and CoreDNS.
+#
+# A cloud LB health-checks this Service on port 53 and needs the probe to
+# actually answer a query: the default TCP check is not enough for UDP. If your
+# provider supports UDP health checks, point them at 53/UDP; otherwise leave
+# the load balancer health check on the HTTP admin port (see the comment on the
+# web Service) and let the provider's DNS failover be the only response to a
+# dead resolver.
+apiVersion: v1
+kind: Service
+metadata:
+  name: fancy-dns-dns
+  namespace: dns
+  labels:
+    app.kubernetes.io/name: fancy-dns
+  annotations:
+    # Cloud-specific knobs go here; uncomment for the matching provider.
+    # service.beta.kubernetes.io/aws-load-balancer-type: nlb
+    # service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
+    # service.beta.kubernetes.io/gcp-load-balancer-type: EXTERNAL
+spec:
+  type: LoadBalancer
+  externalTrafficPolicy: Local
+  selector:
+    app.kubernetes.io/name: fancy-dns
+  ports:
+    - name: dns-udp
+      port: 53
+      targetPort: dns-udp
+      protocol: UDP
+    - name: dns-tcp
+      port: 53
+      targetPort: dns-tcp
+      protocol: TCP
+    - name: dot
+      port: 853
+      targetPort: dot
+---
+# A normal ClusterIP Service for everything the Ingress fronts. It has to be a
+# separate object because a headless Service has no virtual IP for an ingress
+# controller to route to.
+#
+# It is also the only reliable health-check target for the cloud load balancers
+# on fancy-dns-dns: the admin port speaks HTTP and answers /api/healthz
+# unauthenticated (a health check cannot carry an API key), so a probe can tell
+# "bound and serving" from "process up but not listening".
+apiVersion: v1
+kind: Service
+metadata:
+  name: fancy-dns-web
+  namespace: dns
+  labels:
+    app.kubernetes.io/name: fancy-dns
+spec:
+  type: ClusterIP
+  selector:
+    app.kubernetes.io/name: fancy-dns
+  ports:
+    - name: doh
+      port: 443
+      targetPort: doh
+    - name: admin
+      port: 8080
+      targetPort: admin

deploy/k8s/30-ingress.yaml

@@ -0,0 +1,64 @@
+# The console, the admin API and the DoH endpoint, behind TLS.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: fancy-dns
+  namespace: dns
+  labels:
+    app.kubernetes.io/name: fancy-dns
+  annotations:
+    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
+    # SSE (the live query log) is the reason for these three. With response
+    # buffering on, nginx holds events until the buffer fills and the log page
+    # looks frozen; with a 60s read timeout the stream is cut every minute and
+    # the page reconnects in a loop.
+    nginx.ingress.kubernetes.io/proxy-buffering: "off"
+    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
+    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
+spec:
+  ingressClassName: nginx
+  rules:
+    - host: dns.example.com
+      http:
+        paths:
+          # DoH must be matched before the catch-all below, and both are
+          # Prefix paths on distinct roots so the order is not load-bearing.
+          - path: /dns-query
+            pathType: Prefix
+            backend:
+              service:
+                name: fancy-dns-web
+                port:
+                  name: doh
+          - path: /api
+            pathType: Prefix
+            backend:
+              service:
+                name: fancy-dns-web
+                port:
+                  name: admin
+          # The embedded console.
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: fancy-dns-web
+                port:
+                  name: admin
+  tls:
+    - hosts: [dns.example.com]
+      secretName: fancy-dns-tls
+---
+# Two replicas means one of them can always be taken away for a node drain.
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+  name: fancy-dns
+  namespace: dns
+  labels:
+    app.kubernetes.io/name: fancy-dns
+spec:
+  minAvailable: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: fancy-dns

deploy/k8s/kustomization.yaml

@@ -0,0 +1,26 @@
+# kustomize build deploy/k8s | kubectl apply -f -
+#
+# The files are numbered so a plain `kubectl apply -f deploy/k8s/` in
+# alphabetical order also lands in a working order: namespace and config
+# first, then identity, then the workload, then the networking.
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+namespace: dns
+
+resources:
+  - 00-config.yaml
+  - 05-serviceaccount.yaml
+  - 08-secret-apikey.yaml
+  - 10-deployment.yaml
+  - 20-service.yaml
+  - 30-ingress.yaml
+
+labels:
+  - includeSelectors: false
+    pairs:
+      app.kubernetes.io/part-of: fancy-dns
+
+images:
+  - name: fancydns/fancy-dns
+    newTag: latest