go-otel/docs/STATUS.md

Status: error triage, spike detection, IP blocking (2026-09-19)

Pick-up notes. Nothing below is committed, deployed or applied to the cluster unless stated. Greenfield, single user: radical changes are fine.

Access

• ClickHouse (otel instance) is now a LoadBalancer: monitoring/clickhouse at 10.10.3.45 (native 9000, HTTP 8123). Set in k8s/clickhouse.yaml and patched live. docs/ML.md still says "keep ClusterIP"; that is superseded.
• Credentials are in .env (gitignored, mode 600). Load with set -a; . ./.env; set +a. clickhouse-client is installed locally.

What exists now

Piece	Where	State
Normalized rows written on ingest	internal/storage/normalized.go, Append in clickhouse.go	Code done, tests pass. Running collector is still the old build, so nothing new is written until deployed.
Live schema migration	EnsureNormalizedSchema (runs at collector start)	Applied once by hand via backfill; issue_fingerprint etc. now exist.
404/410 are not errors	isErrorStatus in internal/normalization/normalizer.go	Done. Rows stored before this still count 404s as errors.
Backfill	cmd/normalized-backfill	Ran 2026-09-19 12:00 to ~21:00 UTC. Queries need FINAL (duplicates until merge).
Spike rule	internal/features/spike.go	Excess-error rule, trimmed-mean baseline, Suppressor (cooldown, 2x escalation). Tested.
Replay CLI	cmd/spike-alerts	Replays a range; mutes non-actionable categories.
Cause categories	internal/triage	Deterministic rules, first match wins. Tested.
Rules scoring	cmd/triage-eval	Scores a labelled JSONL; also prepares rows for labelling.
Offender report	cmd/bad-actors	Read-only. Ranks IPs from raw nginx spans, marks [BLOCK] / [INTERNAL].
Tracker	internal/honeypot	Now needs distinct paths (default 3), never tracks internal/reserved IPs, uses triage.IsProbePath.
Blocker	internal/blocker	Writes ingress-nginx block-cidrs. Dry-run by default. Tested with a fake client. Wired into cmd/server.
RBAC + env	k8s/deployment.yaml	Role limited to the one ConfigMap; blocker env set to enabled + dry-run. Not applied.

Categories: scanner_probe, dependency_unreachable, client_credential_issue, stale_reference, service_defect, capacity_saturation, expected_lifecycle, needs_review. They are cause-level verdicts, not restatements of status.

Findings worth remembering

• Needle was removed (2026-11-01). It scored 10% on the 40-row golden set (about chance for 8 classes), picked by tool position rather than content, and its confidence was uncorrelated with correctness. Sidecar, proxy handlers, UI and k8s container are gone; the labelled sets live in cmd/triage-eval/data/.
• Edge 4xx never pages. nginx 4xx (registry 401 challenges, /oauth2/auth, scanner 405s) is dropped in FilterActionable; nginx 5xx still alerts.
• Rules scored 39/40 on the golden set, but that is a fit (written after seeing the rows). A fresh 60-row holdout is in cmd/triage-eval/data/holdout.jsonl (from 2026-09-18, unseen); it still needs your expected labels. Known probe-like paths that still land in stale_reference: /ssl/server.key, /_debugbar/open, /debug/pprof, /actuator/*, /graphql, /server-info, /.bash_profile, /settings.py, /__env.js.
• The top error sources were noise: 404 probes and a 401 poller (/api/repos/available, /api/controller/instances, about 700 per hour, counted twice: nginx and the orchestrator).
• IP data: 44 public IPs account for 63% of probe requests (12.5 h window). 34% show a peer of 127.0.0.1 because net.peer.ip is the TCP peer behind the PROXY-protocol hop, so the tracker cannot see them. The ingress has enable-real-ip=true, forwarded-for-header=proxy_protocol, proxy-real-ip-cidr=127.0.0.1/32, so block-cidrs itself works on the real client address.
• Internal addresses are a hard stop in both tracker and blocker (honeypot.Untrackable): loopback, RFC1918, ULA, link-local, multicast, unspecified, IPv4-mapped, CGNAT 100.64/10 (Tailscale), 192.0.0.0/24, 198.18/15, 240/4. The allowlist only adds exclusions.

Before enabling enforcement

1. Deploy the new build (dry-run on). Watch GET /api/blocker/status for a day; compare against go run ./cmd/bad-actors -since ... -until ....
2. Put your own public IP(s) in OTEL_BLOCKER_ALLOW. A blocked IP gets 403, so you would lock yourself out of the UI. Recovery: kubectl edit cm ingress-nginx-controller -n ingress-nginx (remove entries; managed ones are listed in the otel-debug.k6n.net/blocked annotation).
3. Then set OTEL_BLOCKER_DRY_RUN=false.
4. Do not kubectl apply the whole k8s/deployment.yaml blindly: the dry run showed it would create a new otel-sourcemaps PVC not present in the cluster.

Next steps (suggested order)

1. Deploy the collector so normalized rows stay current; verify with SELECT max(timestamp) FROM otel.normalized_events.
2. Label holdout.jsonl (fill expected), run go run ./cmd/triage-eval -input cmd/triage-eval/data/holdout.jsonl, fix the rule families that miss. Then re-export a fresh holdout so scoring stays honest.
3. Warm-up guard in DetectSpikes: require baseline data before alerting (the first window after data begins alerts on everything as NEW).
4. Real client IP in nginx spans, to close the 127.0.0.1 gap (inspect the ingress OTel config first; not looked at yet).
5. Alert delivery: run DetectSpikes on a timer in the collector and post somewhere (UI panel / ntfy / Slack). Category chips and same / different / noise + "mute pattern" in the UI, storing labels.
6. Reduce 401-poller noise: mute list or per-pattern rate limit; client_credential_issue is the caller's fault, likely a lower severity.
7. After 2+ weeks of data: time-of-day baseline, rollout proximity, then the logistic regression / MLP from docs/ML.md; the rule detector is what they must beat.

Update 2026-09-20

• Warm-up guard: SpikeConfig.DataSince / WarmupWindows (default 6). No alert until that much baseline exists; spike-alerts sets it from min(timestamp) of normalized_events. Test: TestDetectSpikesWarmupGuard.

• holdout.jsonl labelled (by Claude, review it; ambiguous 404s such as /mcp, /graphql, /api, /api/v2/* were labelled stale_reference). Rules fixed for the misses: more probe paths (_debugbar, debug/pprof, server-info, actuator, __env, .bash_*, *.key|pem|py, graphql/console) and all 1xx/2xx/3xx are expected_lifecycle. 60/60, but the set is now tuned against, so re-export a fresh holdout before trusting the number. IsProbePath is shared with the blocker, so these also widen what counts as a probe.

• Client IP gap (last 6h of nginx spans): public IPs 10.3k spans (2192 IPs), 10.10.0.1 25k (the user's firewall: traffic it NATs to ingress arrives with the firewall as source), loopback 4.1k. The loopback rows include real external scanners (CameraScannerV3, MSIE8 UAs, raw-IP hosts), so ~10% of spans lose the attacker IP. Ingress has externalTrafficPolicy: Local, no use-proxy-protocol, and x-real-ip is never attached. Cause: something on the node forwards to ingress from 127.0.0.1; not yet identified.

• UA classes: internal/uaclass holds the known crawler / AI agent / monitor / scan-tool list (first match wins; UA is spoofable, so a class is a positive signal only). Stored as normalized_events.ua_class (added by EnsureNormalizedSchema on next start; old rows are ''), served as uaClass on issue samples and endpoint stats. Issues UI: add-filter offers client: X and hide client: X. Not yet a triage/spike feature.

• Trace context (internal/tracectx): one request is several spans in several envelopes (nginx outer span on :442 with the real client, nginx inner span on :8181 from 127.0.0.1, then the app span), so the ingest-time same-envelope lookup in normalizeTraces almost never finds the parent (7 of 1154 nginx spans measured). tracectx.Resolve works on a loaded span set instead and is order independent. It yields ClientIP (first public peer up the chain), IsDuplicate (repeats an ancestor's failure: same status code), IsRootCause, child / upstream-child counts, ParentMissing.

  • bad-actors resolves loopback hops to the real client (839 hop spans / 583 probes recovered in a 3h sample).
  • spike-alerts loads the spans of every trace with an error, drops duplicate hops from the counts and reports distinct clients per pattern; it falls back when the trace columns are absent. Historical rows have empty span_id, so it only helps for data written since the trace columns shipped (~08:09 UTC 2026-09-20).
  • features.Row gained LogicalErrors, DuplicateErrors, RootCauseErrors, DistinctClients, TopClientShare, BotErrorShare (additive). window-features reads optional clientIp/uaClass/isDuplicate/isRootCause/hasUpstream.
  • Trace view in the UI shows the resolved client (lib/trace-client.ts).
  • Not done: the collector still records the hop IP on issue samples at ingest; span name GET / for orchestrator spans is unchanged (changing it alters every orchestrator pattern signature); HasUpstream is loaded but not yet a triage rule; trace propagation is missing in cart/checkout/profile/scheduling (roots only).

• Alerts (internal/alerts, env OTEL_ALERT_*, secret otel-alerts): the server runs the spike detector every minute and posts to Telegram (and/or a generic webhook). First tick after a start only primes the suppressor. GET /api/alerts, POST /api/alerts/test, go run ./cmd/alert-test. Recent alerts are in memory only.

• Bad actors (features.MarkBadActors): an event is marked, never dropped, when its client hit >=3 probe paths (probe-scanner), walked >=15 not-found routes (enumerator), or is a declared crawler/agent/scanner getting a 4xx (bot-4xx; their 5xx stay visible). Marked events are ignored by DetectSpikes/alerts and counted in Row.BadActorErrors. Row.Label now targets the clean future error rate (logical requests, no bad actors); RawLabel is the old raw-count label. AI crawler rows stay in the data on purpose (see memory: flag, not block). Client rules need a resolved client IP, so they only cover data since the trace columns shipped; the bot rule works on any row with a UA class.

• Route ids: term_<hex> style segments normalize to {id} (per-session websocket routes were creating a new pattern each session).

Commands

set -a; . ./.env; set +a

# replay the spike rule over a range
go run ./cmd/spike-alerts -since 2026-09-19T12:00:00Z -until 2026-09-19T20:30:00Z -baseline-windows 12
# add -show-muted to see probes / stale / lifecycle patterns too

# who would be blocked
go run ./cmd/bad-actors -since 2026-09-19T08:00:00Z -until 2026-09-19T20:30:00Z

# score the rules against labelled rows
go run ./cmd/triage-eval -input cmd/triage-eval/data/holdout.jsonl

Backfill in small windows only (10 to 30 minutes over the LAN); one-hour windows through a kubectl port-forward hung and dropped.

Git state

Everything is uncommitted. Files changed by this work: internal/{storage, normalization,features,triage,honeypot,blocker}, cmd/{spike-alerts, triage-eval,bad-actors,normalized-backfill,server}, k8s/{clickhouse,deployment}.yaml, .gitignore.

Not from this work; check before committing: AGENTS.md, internal/collector/{collector_test,issue_extract}.go, otel-ui/src/** (including a deleted GaugeMeter.tsx).

IP details (MaxMind)

The issues view's source IP opens a dialog (GET /api/ip/{ip}, authed) with country/region/city, timezone, ASN and network. Lookups are local. Enable by mounting GeoLite2 files and setting OTEL_GEOIP_CITY_DB and/or OTEL_GEOIP_ASN_DB (the Dockerfile copies db/*.mmdb into the image and sets both); without them the dialog says so. Private/loopback addresses are labelled instead of looked up.