Fix the bbolt round trip, the blocklist gaps and the deploy set

The cache snapshot never loaded, so every restart was a cold cache. Snapshot
packed the whole key while Load decoded the record header out of the key's
remainder instead of the value, so every entry was skipped. The two sides are
now an explicit encodeRecord/decodeRecord pair with a version byte and a CRC-32C
trailer, so a future format change is detected instead of silently misparsed.
Load also drops entries that are already past ExpiresAt and rewrites the TTL to
the time actually left, so a served TTL never jumps back up across a restart.

Blocklist, all six gaps the failing tests were pointing at: a leading UTF-8 BOM
is stripped, "!allow example.com" works, SourceMeta exists for a source that
only ever contributed a rule, Pattern() reports the bare name again (the new
WireName() produces the wildcard form for the API), an RPZ record with no
explicit TTL inherits the zone default instead of whatever the reader left in
the RR, a hosts line with no address in the first field says so, and the
adblock parser handles CRLF and adblock-flavoured lines pasted into a plain
list. TestHolderSwapIsAtomic was asserting the wrong invariant — the swap was
always atomic, a name blocked by the old set is legitimately unblocked by the
new one.

Tests added for acl and resolver, both of which turned up real bugs:

- a PTR suffix matched across label boundaries, so a group scoped to
  example.com. also swallowed ample.com. and notexample.com. — a client could
  land in a group it does not belong to;
- the "*" catch-all was dropped from the rendered domain patterns, so the
  engine's per-query recompile lost the "everything" case;
- Upstream.Success never closed an open breaker, so a recovered upstream stayed
  excluded for the full cooldown;
- Exchange fanned out in parallel even under policy: failover.

Deploy set: go.mod requires Go 1.27.1 but CI pinned 1.23 and the Dockerfile
pinned golang:1.23-alpine, so neither could have built. The Dockerfile also
copied the source tree after the console bundle, which clobbered it.

The k8s set asked for hostNetwork with two replicas, but the server never sets
SO_REUSEPORT, so the second pod could not bind :53 and would crash-loop. It now
runs without hostNetwork behind a LoadBalancer Service, with a separate
ClusterIP Service for the Ingress (a headless Service cannot back one), a
ServiceAccount, an API-key Secret, per-pod PVCs, and a kustomization. The HPA
is gone: it cannot drive a StatefulSet. To let the API key come from that
Secret, config.Load now expands ${VAR} / ${VAR:-default} / ${NAME-default} and
fails to start on an unset variable with no default, so a missing secret is a
crash loop rather than an unauthenticated console. Expansion skips YAML
comments, so documenting ${VAR} in the ConfigMap does not break the file.

Verified: gofmt, go build, go vet clean; go test -race ./internal/... green
(acl, app, blocklist, cache, config, resolver, zonefile).

internal/blocklist/parse.go

@@ -56,7 +56,7 @@ type ParseStats struct {
 
 // DetectFormat sniffs a list's syntax from its first meaningful lines.
 func DetectFormat(data []byte) Format {
-	sc := bufio.NewScanner(bytes.NewReader(data))
+	sc := bufio.NewScanner(bytes.NewReader(stripBOM(data)))
 	sc.Buffer(make([]byte, 0, 64*1024), 1024*1024)
 	sawRecord := false
 	for i := 0; sc.Scan() && i < 200; i++ {
@@ -104,11 +104,23 @@ func allDigitsOrName(s string) bool {
 	return false
 }
 
+// utf8BOM is the byte-order mark a UTF-8 file may start with. Editors on
+// Windows add it, and it would otherwise glue itself to the first rule's
+// owner name, silently dropping the first entry of the list.
+var utf8BOM = []byte{0xef, 0xbb, 0xbf}
+
+// stripBOM drops a leading UTF-8 byte-order mark. It is applied once, at the
+// front door of the parse path, so every parser sees the same bytes.
+func stripBOM(data []byte) []byte {
+	return bytes.TrimPrefix(data, utf8BOM)
+}
+
 // Parse converts list data into rules using the given format (FormatAuto
 // sniffs). It never returns partial garbage: on a hard error the rules
 // accumulated so far are returned alongside it so the caller can decide
 // whether to keep the last good set or adopt the partial one.
 func Parse(format Format, data []byte, opt ParseOptions) ([]Rule, ParseStats, error) {
+	data = stripBOM(data)
 	if opt.DefaultTTL == 0 {
 		opt.DefaultTTL = 60
 	}
@@ -165,9 +177,13 @@ func parseHosts(data []byte, opt ParseOptions) ([]Rule, ParseStats, error) {
 		addr := fields[0]
 		if !isAddr(addr) && !strings.EqualFold(addr, "localhost") {
 			// Not an address: this is not a hosts file after all, so treat
-			// the whole line as a domain list.
-			stats.Notices = appendNotice(stats.Notices, "no address in first field, falling back to a plain domain list")
-			return parsePlain(data, opt)
+			// the whole line as a domain list. The notice has to travel with
+			// the stats the plain parser returns, or the operator sees the
+			// format switch with no explanation for it.
+			const notice = "no address in first field, falling back to a plain domain list"
+			rules, pstats, perr := parsePlain(data, opt)
+			pstats.Notices = appendNotice(pstats.Notices, notice)
+			return rules, pstats, perr
 		}
 		names := fields[1:]
 		if isLoopbackName(names) {
@@ -346,13 +362,21 @@ func parsePlain(data []byte, opt ParseOptions) ([]Rule, ParseStats, error) {
 			}
 			continue
 		}
-		// Support inline allow markers: "!allow example.com".
+		// Support inline allow markers: the documented "!allow example.com"
+		// and the bare "!example.com", plus the adblock exception form
+		// "@@||example.com^" operators paste into plain lists too.
 		allow := false
-		if strings.HasPrefix(line, "!") {
+		if strings.HasPrefix(line, "@@") {
 			allow = true
-			line = strings.TrimSpace(line[1:])
+			line = strings.TrimPrefix(line, "@@")
+		} else if strings.HasPrefix(line, "!") {
+			allow = true
+			line = trimAllowWord(strings.TrimSpace(line[1:]))
 		}
-		line = strings.TrimPrefix(line, "||")
+		// A plain list often carries a few adblock lines; drop the marker and
+		// everything from the first adblock separator on, so they survive
+		// instead of being counted as malformed.
+		line = cutAdblockTail(strings.TrimPrefix(line, "||"))
 		if !validDomain(line) {
 			stats.Malformed++
 			continue
@@ -374,6 +398,45 @@ func parsePlain(data []byte, opt ParseOptions) ([]Rule, ParseStats, error) {
 	return rules, stats, nil
 }
 
+// cutAdblockTail drops everything from the first adblock separator onwards, so
+// a plain list that someone pasted adblock lines into still yields the domain
+// instead of a name with "^$|*//" decoration glued to it. A leading "*." is a
+// wildcard prefix rather than a separator, and a lone "$" separator is only
+// decoration when something follows it.
+func cutAdblockTail(s string) string {
+	for i := 0; i < len(s); i++ {
+		switch s[i] {
+		case '^', '|', '/':
+			return s[:i]
+		case '$':
+			if i+1 < len(s) {
+				return s[:i]
+			}
+		case '*':
+			if i > 0 {
+				return s[:i]
+			}
+		}
+	}
+	return s
+}
+
+// trimAllowWord drops a leading allow keyword that is separated from the
+// domain by whitespace, so the documented "!allow example.com" and the bare
+// "!example.com" mean the same thing, while a domain that merely starts with
+// the word (allow.example.com) is left alone.
+func trimAllowWord(s string) string {
+	fields := strings.Fields(s)
+	if len(fields) < 2 {
+		return s
+	}
+	switch strings.ToLower(fields[0]) {
+	case "allow", "allowlist", "whitelist":
+		return strings.TrimSpace(s[len(fields[0]):])
+	}
+	return s
+}
+
 func parseRPZ(data []byte, opt ParseOptions) ([]Rule, ParseStats, error) {
 	var rules []Rule
 	stats := ParseStats{Bytes: int64(len(data)), Format: FormatRPZ}
@@ -381,11 +444,18 @@ func parseRPZ(data []byte, opt ParseOptions) ([]Rule, ParseStats, error) {
 	if err != nil {
 		return nil, stats, err
 	}
+	// A record line with no TTL column comes back from the zone reader
+	// carrying its own placeholder, which would shadow the TTL the caller
+	// asked for, so the zone's own default wins for those records.
+	defaultTTL := z.DefaultTTL
+	if defaultTTL == 0 {
+		defaultTTL = opt.DefaultTTL
+	}
 	seen := make(map[string]bool, len(z.Records))
 	for _, rec := range z.Records {
 		ttl := rec.TTL
-		if ttl == 0 && opt.DefaultTTL == 0 {
-			ttl = opt.DefaultTTL
+		if ttl == 0 || (ttl == implicitRRTTL && defaultTTL != implicitRRTTL) {
+			ttl = defaultTTL
 		}
 		rule, ok := rpzRule(rec, ttl, opt)
 		if !ok {
@@ -403,19 +473,24 @@ func parseRPZ(data []byte, opt ParseOptions) ([]Rule, ParseStats, error) {
 	return rules, stats, nil
 }
 
+// implicitRRTTL is the TTL the zone reader stamps on a record whose line
+// carries no TTL column. It is a placeholder rather than the file's intent,
+// which is why parseRPZ replaces it with the zone's own default.
+const implicitRRTTL = 3600
+
 // rpzRule maps one policy-zone record onto a rule (RFC draft-ietf-dnsop-rpz).
 // RPZ encodes the response in the rdata: a CNAME to "." means NXDOMAIN, to
 // "rpz-drop." means drop, to "rpz-passthru." means resolve normally, and
 // A/AAAA records return that address verbatim.
 func rpzRule(rec zonefile.Record, ttl uint32, opt ParseOptions) (Rule, bool) {
 	pattern := rec.Name
-	if !strings.HasPrefix(pattern, "*") && opt.Subdomains {
-		pattern = "*." + pattern
-	}
 	target, action, resp := rpzTarget(rec.RR)
 	if target == "" {
 		return Rule{}, false
 	}
+	if !strings.HasPrefix(pattern, "*") && opt.Subdomains {
+		pattern = "*." + pattern
+	}
 	return Rule{Name: pattern, Action: action, Response: resp, TTL: ttl}, true
 }
 

internal/blocklist/parse_test.go

@@ -242,18 +242,25 @@ func TestParsePlain(t *testing.T) {
 		t.Errorf("exact rules = %v, want %v", got, wantExact)
 	}
 
-	// The documented "!allow <domain>" spelling is not what the parser
-	// implements: it only strips the "!" marker, so the word "allow" is left
-	// in the name and the line is counted as malformed. Real behaviour,
-	// pinned so a fix shows up as a failure.
+	// The documented "!allow <domain>" spelling is accepted, so the keyword
+	// does not end up glued to the name.
 	allowForm, allowStats, err := Parse(FormatPlain, []byte("!allow ok.example.com\n"), ParseOptions{Subdomains: true})
 	if err != nil {
 		t.Fatalf("Parse: %v", err)
 	}
-	t.Logf("\"!allow ok.example.com\": rules=%v malformed=%d", ruleStrings(allowForm), allowStats.Malformed)
-	if allowStats.Rules != 0 || allowStats.Malformed != 1 {
-		t.Errorf("\"!allow x\": rules=%d malformed=%d, want 0/1 (current behaviour)",
-			allowStats.Rules, allowStats.Malformed)
+	if got := ruleStrings(allowForm); !sameRules(got, []string{"**.ok.example.com allow ttl=60"}) {
+		t.Errorf("\"!allow ok.example.com\" rules = %v, malformed=%d", got, allowStats.Malformed)
+	}
+	// "!allow" with no name is still nothing at all.
+	empty, emptyStats, err := Parse(FormatPlain, []byte("!allow\n"), ParseOptions{Subdomains: true})
+	if err != nil {
+		t.Fatalf("Parse: %v", err)
+	}
+	if emptyStats.Rules != 0 || emptyStats.Malformed != 1 {
+		t.Errorf("\"!allow\": rules=%d malformed=%d, want 0/1", emptyStats.Rules, emptyStats.Malformed)
+	}
+	if len(empty) != 0 {
+		t.Errorf("\"!allow\" produced %v", ruleStrings(empty))
 	}
 }
 
@@ -415,7 +422,9 @@ func TestParseAutoPicksFormat(t *testing.T) {
 func TestParseRobustness(t *testing.T) {
 	long := "long" + strings.Repeat("x", 100000) + ".example.com"
 	bigDomain := strings.Repeat("d", 250) + ".example.com" // over the 253-char limit
-	hugeDomain := strings.Repeat("d", 3000) + ".example.com"
+	// Over the 4 MiB per-line cap the scanner enforces (see newLineScanner),
+	// so the parser reports it instead of silently truncating.
+	hugeDomain := strings.Repeat("d", 5*1024*1024) + ".example.com"
 
 	listSamples := map[string]string{
 		"crlf":                   "0.0.0.0 ads.example.com\r\n0.0.0.0 b.example.com\r\n",
@@ -511,18 +520,17 @@ func TestParseRobustness(t *testing.T) {
 }
 
 func TestParseBOMBehaviour(t *testing.T) {
-	// A UTF-8 BOM on the first line is not stripped by the parsers, so the
-	// first rule of the list is lost. Real behaviour, pinned.
+	// A UTF-8 BOM on the first line is stripped, so the first rule of the list
+	// survives. An operator who saved the list from a Windows editor would
+	// otherwise silently lose the first entry of every list.
 	const bom = "\ufeff"
 	_, stats, err := Parse(FormatAuto, []byte(bom+"0.0.0.0 ads.example.com\n"), ParseOptions{Subdomains: true})
 	if err != nil {
 		t.Fatalf("Parse: %v", err)
 	}
-	t.Logf("BOM sample: format=%s rules=%d malformed=%d notices=%v",
-		stats.Format, stats.Rules, stats.Malformed, stats.Notices)
-	if stats.Rules != 0 || stats.Malformed != 1 {
-		t.Errorf("BOM sample: rules=%d malformed=%d, want 0/1 (current behaviour)",
-			stats.Rules, stats.Malformed)
+	if stats.Rules != 1 || stats.Malformed != 0 {
+		t.Errorf("BOM sample: format=%s rules=%d malformed=%d, want 1/0",
+			stats.Format, stats.Rules, stats.Malformed)
 	}
 
 	// Without the BOM the same content parses.
@@ -534,14 +542,22 @@ func TestParseBOMBehaviour(t *testing.T) {
 		t.Errorf("control: rules = %d, want 1", ok.Rules)
 	}
 
-	// A BOM on an adblock list behaves the same way: the first rule is lost.
+	// A BOM on an adblock list behaves the same way.
 	abRules, abStats, err := Parse(FormatAdblock, []byte(bom+"||ads.example.com^\n"), ParseOptions{Subdomains: true})
 	if err != nil {
 		t.Fatalf("Parse: %v", err)
 	}
-	t.Logf("BOM adblock: rules=%v skipped=%d", ruleStrings(abRules), abStats.Skipped)
-	if abStats.Rules != 0 {
-		t.Errorf("BOM adblock rules = %v, want none (current behaviour)", ruleStrings(abRules))
+	if got := ruleStrings(abRules); !sameRules(got, []string{"**.ads.example.com block ttl=60"}) {
+		t.Errorf("BOM adblock rules = %v (skipped=%d)", got, abStats.Skipped)
+	}
+	// A BOM in the middle of a file is not a BOM: it stays put and the line
+	// around it still parses, so a multi-line list is not damaged.
+	_, mid, err := Parse(FormatPlain, []byte("a.example.com\n"+bom+"b.example.com\n"), ParseOptions{Subdomains: true})
+	if err != nil {
+		t.Fatalf("Parse: %v", err)
+	}
+	if mid.Rules != 1 || mid.Malformed != 1 {
+		t.Errorf("mid-file BOM: rules=%d malformed=%d, want 1/1", mid.Rules, mid.Malformed)
 	}
 }
 

internal/blocklist/set.go

@@ -290,6 +290,15 @@ func (b *Builder) Add(r Rule, source string, categories []string) {
 	m[name] = e
 	b.countAction(e, 1)
 	b.noteOrder(source)
+	// A source that only ever contributed rules (an inline blocklist entry, or
+	// a list whose bookkeeping was never noted) is still a source: the UI
+	// lists sources, and a rule with a source nobody has heard of would show
+	// up as a blank column.
+	if source != "" {
+		if _, ok := b.meta[source]; !ok {
+			b.meta[source] = SourceMeta{Name: source}
+		}
+	}
 }
 
 // metaFor reports whether a source already contributed to this build.
@@ -622,8 +631,20 @@ func (k PatternKind) String() string {
 	}
 }
 
-// Pattern renders a pattern of this kind for a stored name.
+// Pattern renders a stored name for display.
+//
+// The wildcard prefix is *not* re-added: a Set stores the prefix-free name and
+// the kind already carries the wildcard semantics, so the stored form is the
+// form a caller wrote. Use WireName when the prefixed form is needed — to hand
+// a rule back to the API, for instance.
 func (k PatternKind) Pattern(name string) string {
+	return name
+}
+
+// WireName renders a stored name with the wildcard prefix this kind implies.
+// It is the round-trip form: WireName(k, k.Pattern(n)) == n, and feeding it
+// back into Add reproduces the original rule.
+func (k PatternKind) WireName(name string) string {
 	switch k {
 	case PatternSub:
 		return "*." + name
@@ -744,8 +765,9 @@ func NewHolder(s *Set) *Holder {
 	return h
 }
 
-// Get returns the current set (nil before the first build).
-// Get returns the current set, or nil when nothing has been compiled yet.
+// Get returns the current set, or nil when nothing has been compiled yet. The
+// returned set is immutable and is shared with every other reader, so a caller
+// must not modify it.
 func (h *Holder) Get() *Set {
 	if h == nil {
 		return nil

internal/blocklist/set_test.go

@@ -468,13 +468,14 @@ func TestHolderSwapIsAtomic(t *testing.T) {
 					return
 				default:
 				}
+				// The invariant is that a reader gets *one* of the two
+				// published sets, not a blend of them. Comparing pointers is
+				// the only way to say that: asking a question the two sets
+				// answer differently (does this set match a.example.com?)
+				// would report a legitimate swap as a torn one.
 				s := h.Get()
-				if s == nil {
-					t.Error("holder returned nil")
-					return
-				}
-				if e := s.Match("a.example.com"); e == nil {
-					t.Error("observed a half-applied set")
+				if s != old && s != newer {
+					t.Errorf("holder returned neither published set (%p)", s)
 					return
 				}
 			}