03d771430bf0aac44239b924a87bc6da24110d5c
10
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  matsandClaude Sonnet 5.5
|
0fdc828a10 |
Keep list copies on disk, start from them, and stop logging address overrides
Found by rolling the resolver out to the real cluster. BlockList objects have no path, deliberately, so the first rollout downloaded the list at every start. The cluster DNS that the download needs is this very resolver, which was not up yet, so the first compile failed after 14 seconds, held the DNS listeners closed for that long, and left the pod with no ad blocking at all. Three changes: * Lists from cluster objects get a copy under the runtime's own data directory (data/lists/<name>), chosen by the runtime from the object's name, never by the object. * The copy is used as the answer only while it is younger than the list's refresh interval, and otherwise the list is downloaded again, with the old copy as the fallback if that fails. Before, any copy on disk was treated as current for ever, so a list with a path was frozen at its first download. A source now ages from its file, an operator's "refresh now" forces a download, and a failed download keeps the stale copy. * Startup loads only what is on disk (RefreshOffline), opens the listeners, and fetches stale or missing lists in the background. The resolver answers at once and the network is never on its critical path. An address override answered as a redirect and so counted as a policy action: with the access log logging redirects, every lookup of an internal name wrote a line. Overrides from the managed source are local answers now; a redirect a list imposes is still logged. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
matsandClaude Sonnet 5.5
|
b513501773 |
Fix block-answer TTLs that were nanoseconds, and stop counting NXDOMAIN as an error
Both found by running the resolver locally against real upstreams. response_ttl is a duration (60s) and a DNS TTL is seconds, but eight places converted with a bare uint32(), which keeps the low 32 bits of the nanosecond count. A blocked name's negative answer carried an SOA TTL and minimum of 997195776 (about 31 years), so any client or downstream cache that honoured it would remember the block, and keep remembering it after the name was unblocked. blocklist.TTLSeconds converts properly (whole seconds, clamped to 2^31-1) and is used by the rule parser and the block, sinkhole and redirect answers. The tests read the TTL off the wire; with the old conversion they fail with 4165425152. An ordinary upstream NXDOMAIN was classified as an error. A name that does not exist is the normal result of a typo, a search-domain expansion or a leaked mDNS query, so on a real network the error metric counted mostly those and, with the access log's errors-only default, every one wrote a line. Only the upstream failing or refusing (SERVFAIL, REFUSED, FORMERR, NOTIMP) is an error now. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
mats
|
0b6dcbd547 |
overrides
build / build-push (push) Successful in 29s
|
||
|
developer
|
70d85eac59 |
Bring the console onto the contract the Go side now pins
build / build-push (push) Failing after 25s
internal/app/contract_test.go asserts the exact JSON key set of every admin
endpoint, and the console was reading fields that shape does not contain —
which is how a page can be entirely blank while the Go suite is green. The
types file is rewritten against the canonical shapes and every consumer is
fixed onto it.
The page bugs that came out of it:
The cache "Lookups" column read `count` and "Last seen" read `last_seen`; the
cache tracks a hit counter and a byte count per name and sends `hits`, so
there is no last-seen data to show at all. Lookups now reads hits, Last seen
is gone, and the row's bytes are shown. The "Size" card shows bytes_human,
with size as the entry-budget hint it actually is.
The Sources table read `item.enabled`, `item.rules`, `item.bytes`,
`item.updated_at` and `item.err`; those live under `item.meta`, so the table
was blank and the per-row toggle posted `name: undefined`. The refresh toast
counted the names echoed back and said "every source" for an empty list.
The dashboard read `qps` and `total_queries` where the server sends
`queries_per_sec` and `queries_total`, and rendered a `latency`/`p99` block
the dashboard response never contained. The `Partial<Dashboard>`-and-cast
that papered over this is gone. `blocked_pct` arrives as 0..100 from REST and
0..1 over SSE, so formatPercent is 0..100 only and a new formatFraction
handles the SSE frame.
Logs rendered `qtype` as a string; it is a number, with `qtype_name`
alongside, so there is a formatQtype that prefers the name. The Domains page
read `item.list` where the server sends `source`, and showed an Updated column
with no timestamp behind it.
Upstreams treated `healthcheck_interval` and `breaker_cooldown` as numbers of
seconds; they are duration strings now, so the page parses them like the
bootstrap refresh it already handled and the write payload no longer sends the
runtime health fields the server ignores. A group edit stripped the four
server-owned counters before writing, because config.Group would reject them.
`whoami.can_write` and `whoami.read_only` were fetched and never used, so a
read-only API key still showed every Save control. A single whoami fetch now
lives in a session provider behind the app shell, every write control is
gated on it, and a banner says why when the key is read-only. It fails open:
an unreachable probe must not lock an operator out of their own console.
Two server fixes the console work exposed, both pinned by the contract test:
POST /api/blocklists/refresh with a body that omits `names` answered
{"names": null}, and the console counts the names it gets back to build its
toast — so "refresh everything" said "0 sources". A nil slice now becomes an
empty one and the key set does not depend on the request.
blocklist.SourceMeta.duration was a raw time.Duration, so it went out as a
nanosecond count and the console divided by 1e6 to compensate — the only
duration on the wire that was not a duration string. It is a config.Duration
now, and formatDuration parses it like the others.
The contract test grew to cover what the first pass left out: the domain
check in all three of its shapes (match, miss, and an error delivered as 200),
the log clear, the settings validate in both the valid and the malformed case,
the settings save and reload, and the first SSE frame.
|
||
|
developer
|
9881d2a095 |
Make the API response contract a test; fix what it found
The console's TypeScript types are hand-written and apiFetch ends in
`JSON.parse(text) as T`, so nothing in the toolchain compared them with the
Go structs. A field that lost its json tag, or a response a struct never had,
showed up as an empty table in a browser and nowhere else: the Go suite was
green and the console was wrong. internal/app/contract_test.go now asserts the
exact JSON key set of every admin endpoint, driven through the real router
with the real hook implementations, plus a test that drives the two write
endpoints with the exact payload web/src builds.
The drift it found, all four fixed here:
blocklist.Source and SourceMeta had no json tags at all, so GET /api/blocklists
sent Go field names ("URL"/"Format") where the console expects snake_case: the
Sources table rendered blank rows and its per-row toggle posted
`name: undefined`. Tagged to match ParseStats.
acl.GroupSummary was flat where the console reads a nested `match` object — the
same shape config.Group has — so every group card read "matches nothing". The
four match fields are nested now, under a new acl.Match.
PUT /api/clients answered 400 `json: unknown field "ptr_suffixes"` to the
console's payload, because config.Group and config.Match carried yaml tags only
and the decoder runs with DisallowUnknownFields. Both structs have json tags
now; the same applies to config.UpstreamServer and UpstreamBootstrap.
PUT /api/upstreams answered 400 `json: unknown field "health"`, because the
console round-trips the whole server object it read, runtime health fields
included. They are accepted and ignored, because the server owns them, and they
are `yaml:"-"` so settings.yaml does not gain runtime state.
Also fixed, each with a test:
A duration on the wire was a nanosecond count. config.Duration had a
MarshalYAML but no MarshalJSON, so bootstraps[].refresh arrived as 30000000000
while the console typed the field as a string, and healthcheck_interval was a
float of seconds in one place and a string in another. Durations on the wire
are duration strings everywhere now — the same spelling config.yaml uses — and a
bare JSON number is still accepted and means seconds, not nanoseconds.
TestDurationJSONForm used to pin the old behaviour and now pins the new one,
saying in its comment that the old behaviour was the bug.
A log row's key set depended on the row. api.LogEntry was omitempty on
client_group, response, upstream and list, so a query with no ACL group
arrived without the key and the console rendered "undefined" where the column
should be empty. The contract test can only pin one shape if the server emits
one, so nothing on that struct is omitempty any more.
A dead peer stayed in the cluster peer list forever. Cluster.seen was only ever
added to and deleted from on an explicit `bye`, so a node that crashed — or
whose Valkey connection died — was reported as a live peer for the life of the
process, and its stale blocklist digest was flagged as divergence forever.
Peers() now reaps peers unheard for peer_ttl*4 and stamps each state with the
receiver's own clock, so clock skew between two machines can neither reap a
healthy peer nor keep a dead one. Discover() ages out the same way on the
publisher's own last_seen stamp, because the keyed Valkey record outlives a
crash: a live node keeps refreshing the key's TTL, so a dead node's field stayed
readable for the life of the fleet.
peer_ttl: 0 would have crashed the process. time.NewTicker(PeerTTL/2) panics on
a non-positive period and `EXPIRE key 0` deletes the heartbeat key, so a config
that left peer_ttl unset would have taken the process down at Start, or made
every instance believe it is alone in the fleet. Both go through peerTTL() and
peerDeadline(), which fall back to the default.
Two cluster tests raced on the wire, not on the product, and one was
environment-dependent rather than wrong. A node that joined after another's
heartbeat was in flight hears about it on the next tick, so "b does not see a"
read immediately after "a sees b" is a coin flip; the divergence test waited for
a digest on the wrong node's view of the fleet; and 127.0.0.1:1 is only
unbindable outside a container that sets
net.ipv4.ip_unprivileged_port_start=0, where it binds and the partial-bind test
fails with both transports up. That test now occupies a port with a live
listener in the test process, which is a deterministic EADDRINUSE on every
machine and every uid.
internal/server/counters_test.go covers the counter-key layer that no other
test touched: the "<transport>.<field>" keyspace, that the aggregate folds only
the known fields, and that the snapshot is safe while counters move.
Two pinned behaviours changed on purpose, each with a test that says so:
An RPZ record did not cover the name it names. rpzRule gave a bare owner name
the subdomain form (*.name), so analytics.example.org. in a policy zone blocked
cdn.analytics.example.org. and left analytics.example.org. resolving normally —
and an rpz-passthru. record did not allowlist its own domain either. RPZ
semantics say the record applies to the domain and everything under it, so a
bare owner now takes the apex form **.name; an owner written with an explicit *.
is still strict, so a zone that wants subdomain-only behaviour can still ask for
it. This changes which queries get blocked for an existing policy zone, which is
why it wanted a decision. TestExampleRPZZoneCoversTheApexItNames pins both
halves and the example zone now says which half it relies on.
metrics.OutcomeLocal had no producer. The engine reported a local-zone answer as
qlog.ActionAllow, which outcomeFor mapped to OutcomeUpstream, so every locally
answered query was counted as an upstream one and the outcome="local" series was
never published — the dashboard's local column was permanently empty.
Response.Local is now set on the local-zone path and outcomeFor consults it
first. That moves published counters, so the README's metrics section says so
too.
|
||
|
developer
|
e8c5b414d9 |
half broken
build / build-push (push) Failing after 23s
|
||
|
developer
|
26134ba437 |
Fix the bbolt round trip, the blocklist gaps and the deploy set
The cache snapshot never loaded, so every restart was a cold cache. Snapshot
packed the whole key while Load decoded the record header out of the key's
remainder instead of the value, so every entry was skipped. The two sides are
now an explicit encodeRecord/decodeRecord pair with a version byte and a CRC-32C
trailer, so a future format change is detected instead of silently misparsed.
Load also drops entries that are already past ExpiresAt and rewrites the TTL to
the time actually left, so a served TTL never jumps back up across a restart.
Blocklist, all six gaps the failing tests were pointing at: a leading UTF-8 BOM
is stripped, "!allow example.com" works, SourceMeta exists for a source that
only ever contributed a rule, Pattern() reports the bare name again (the new
WireName() produces the wildcard form for the API), an RPZ record with no
explicit TTL inherits the zone default instead of whatever the reader left in
the RR, a hosts line with no address in the first field says so, and the
adblock parser handles CRLF and adblock-flavoured lines pasted into a plain
list. TestHolderSwapIsAtomic was asserting the wrong invariant — the swap was
always atomic, a name blocked by the old set is legitimately unblocked by the
new one.
Tests added for acl and resolver, both of which turned up real bugs:
- a PTR suffix matched across label boundaries, so a group scoped to
example.com. also swallowed ample.com. and notexample.com. — a client could
land in a group it does not belong to;
- the "*" catch-all was dropped from the rendered domain patterns, so the
engine's per-query recompile lost the "everything" case;
- Upstream.Success never closed an open breaker, so a recovered upstream stayed
excluded for the full cooldown;
- Exchange fanned out in parallel even under policy: failover.
Deploy set: go.mod requires Go 1.27.1 but CI pinned 1.23 and the Dockerfile
pinned golang:1.23-alpine, so neither could have built. The Dockerfile also
copied the source tree after the console bundle, which clobbered it.
The k8s set asked for hostNetwork with two replicas, but the server never sets
SO_REUSEPORT, so the second pod could not bind :53 and would crash-loop. It now
runs without hostNetwork behind a LoadBalancer Service, with a separate
ClusterIP Service for the Ingress (a headless Service cannot back one), a
ServiceAccount, an API-key Secret, per-pod PVCs, and a kustomization. The HPA
is gone: it cannot drive a StatefulSet. To let the API key come from that
Secret, config.Load now expands ${VAR} / ${VAR:-default} / ${NAME-default} and
fails to start on an unset variable with no default, so a missing secret is a
crash loop rather than an unauthenticated console. Expansion skips YAML
comments, so documenting ${VAR} in the ConfigMap does not break the file.
Verified: gofmt, go build, go vet clean; go test -race ./internal/... green
(acl, app, blocklist, cache, config, resolver, zonefile).
|
||
|
developer
|
aa12b03eb9 |
fix data races in cache, resolver, qlog; add cache/zonefile/blocklist tests; update STATUS
- cache.Entry.Hits and LRU.perShard become atomics; SetCapacity now evicts down to the new budget; packKey fully qualifies names before packing - resolver.Pool rewritten copy-on-write (atomic.Pointer to an immutable state); Upstream is immutable with shared *upstreamStats, so health survives a Rebind and the query path stops racing the config hook - Pool.Exchange no longer marks a dead upstream healthy: Success was called before the error check, so the breaker never tripped - qlog live ring publishes through atomic pointers - blocklist.Holder.Get nil-safe; Builder lists each source once - web: fix toast(string) misuse that rendered empty mutation toasts - add unit tests for cache (with benchmarks) and zonefile - STATUS.md: green -race integration suite, remaining failures and the broken bbolt snapshot round trip documented |
||
|
developer
|
ee72257c64 | more code | ||
|
developer
|
85453852d5 | initial commit |