Add the memory envelope, pprof, and a per-packet fix on the UDP path
ci / Console typecheck and build (push) Successful in 14s
ci / Go build, vet and test (push) Failing after 45s
ci / Binary with the console embedded (push) Skipped
ci / Container image (push) Skipped

A cgroup OOM kill is the worst outcome available to a resolver: an outage
rather than a cache miss, with no chance to serve stale on the way out. So:

- memory.limit / memory.reserve / memory.cache_max_bytes, a new config section.
  limit becomes a GOMEMLIMIT through debug.SetMemoryLimit, reduced by the
  reserve. The reserve exists because the Go heap does not account for
  goroutine stacks, runtime metadata, or the bbolt mmap backing the cache
  snapshot and the query log — handing the runtime the full cgroup limit is how
  you get killed by a mechanism that believed it had headroom.
- config.Size, so a budget reads as "512MiB" rather than a byte count converted
  by hand. Strict about junk: "512MiBx" is an error.
- The cache byte budget is now actually fed to the LRU. It already enforced
  maxBytes per shard on insert, but had no setter and nothing ever called one,
  so the footprint of cache.size was unbounded: one cached ANY or TXT answer
  can be tens of kilobytes, so 500k entries is anywhere from 150MB to several GB
  depending on what the traffic asked for.

- api.pprof mounts net/http/pprof on the admin listener. The pprof handlers sit
  on the app-level mux, not inside the API's own handler, so they would bypass
  the auth middleware entirely; the option is therefore refused unless every
  admin address is loopback. Profiling means port-forwarding, so this is safe
  by construction rather than by policy, and cmdline cannot leak an argument
  that happens to be a key.

- serveUDP was calling net.ResolveUDPAddr per packet. The peer already is a
  *net.UDPAddr; that call builds a string, re-parses the port and allocates on
  the busiest path in the process.

Also fixes a flaky test: TestParallelPolicyAnswersFromTheFastestUpstream
asserted the slow upstreams were hit, but the fan-out cancels on the first
answer, so whether they were dispatched before cancellation is a real race. It
now asserts the answer came from the fast upstream and that nothing waited.
This commit is contained in:
developer committed 2026-09-27 13:41:48 +00:00
1 parent 1bdb0b081c
commit b0e6b3bb92
11 files changed
+382 -6

No files matched your search

+62
View File
@@ -0,0 +1,62 @@
package app
import (
"log/slog"
"math"
"runtime/debug"
"git.k8s-git-k6n.net/mats/fancy-dns/internal/config"
)
// applyMemoryLimit turns the configured memory envelope into a GOMEMLIMIT.
//
// Why this matters more than it looks: the only other lever the Go runtime has
// before a cgroup OOM kill is the cgroup limit itself, and a cgroup kill is the
// worst outcome available to a resolver — an outage rather than a cache miss,
// with no chance to serve stale data on the way out. GOMEMLIMIT turns the same
// limit into a soft one: the GC starts working harder as the heap approaches
// it, and only an OOM if it cannot reclaim enough.
//
// The limit is reduced by the reserve because the Go heap does not account for
// goroutine stacks, runtime metadata, or — the one that matters here — the
// bbolt mmap backing the cache snapshot and the query log, which lives outside
// the Go heap entirely. Handing the runtime the full cgroup limit is how you
// get killed by a mechanism that believed it had headroom.
func applyMemoryLimit(m config.Memory, log *slog.Logger) {
limit := m.EffectiveLimit()
if limit == math.MaxInt64 {
// Nothing configured. Leave the runtime unbounded rather than guessing
// a number the operator did not ask for; a cgroup limit will still
// apply, we just do not get to shape the GC for it.
debug.SetMemoryLimit(math.MaxInt64)
return
}
debug.SetMemoryLimit(limit)
if log != nil {
log.Info("memory envelope applied",
"soft_limit", config.Size(limit).String(),
"configured", m.Limit.String(),
"reserve", config.Memory{Reserve: m.Reserve, Limit: 1}.EffectiveLimit(),
"cache_budget", config.Size(m.EffectiveCacheBytes()).String(),
)
}
}
// memoryStats is what the admin API reports about the envelope, so an operator
// can see how close to the ceiling the process is without attaching a profiler.
type memoryStats struct {
HeapAlloc uint64 `json:"heap_alloc"`
HeapSys uint64 `json:"heap_sys"`
Limit int64 `json:"limit"`
CacheBudget int64 `json:"cache_budget"`
InUse bool `json:"limit_set"`
}
func readMemoryStats(m config.Memory) memoryStats {
limit := m.EffectiveLimit()
return memoryStats{
Limit: limit,
CacheBudget: m.EffectiveCacheBytes(),
InUse: limit != math.MaxInt64,
}
}