go-otel/docs/STATUS.md

Status: error triage, spike detection, IP blocking (2026-09-19)

Pick-up notes. Nothing below is committed, deployed or applied to the cluster unless stated. Greenfield, single user: radical changes are fine.

Access

• ClickHouse (otel instance) is now a LoadBalancer: monitoring/clickhouse at 10.10.3.45 (native 9000, HTTP 8123). Set in k8s/clickhouse.yaml and patched live. docs/ML.md still says "keep ClusterIP"; that is superseded.
• Credentials are in .env (gitignored, mode 600). Load with set -a; . ./.env; set +a. clickhouse-client is installed locally.
• .env deliberately does not enable Needle; pass -needle URL explicitly.

What exists now

Piece	Where	State
Normalized rows written on ingest	internal/storage/normalized.go, Append in clickhouse.go	Code done, tests pass. Running collector is still the old build, so nothing new is written until deployed.
Live schema migration	EnsureNormalizedSchema (runs at collector start)	Applied once by hand via backfill; issue_fingerprint etc. now exist.
404/410 are not errors	isErrorStatus in internal/normalization/normalizer.go	Done. Rows stored before this still count 404s as errors.
Backfill	cmd/normalized-backfill	Ran 2026-09-19 12:00 to ~21:00 UTC. Queries need FINAL (duplicates until merge).
Spike rule	internal/features/spike.go	Excess-error rule, trimmed-mean baseline, Suppressor (cooldown, 2x escalation). Tested.
Replay CLI	cmd/spike-alerts	Replays a range; mutes non-actionable categories; Needle optional and off.
Cause categories	internal/triage	Deterministic rules, first match wins. Tested.
Rules scoring	cmd/triage-eval	Scores a labelled JSONL; also prepares rows for labelling.
Offender report	cmd/bad-actors	Read-only. Ranks IPs from raw nginx spans, marks [BLOCK] / [INTERNAL].
Tracker	internal/honeypot	Now needs distinct paths (default 3), never tracks internal/reserved IPs, uses triage.IsProbePath.
Blocker	internal/blocker	Writes ingress-nginx block-cidrs. Dry-run by default. Tested with a fake client. Wired into cmd/server.
RBAC + env	k8s/deployment.yaml	Role limited to the one ConfigMap; blocker env set to enabled + dry-run. Not applied.

Categories: scanner_probe, dependency_unreachable, client_credential_issue, stale_reference, service_defect, capacity_saturation, expected_lifecycle, needs_review. They are cause-level verdicts, not restatements of status.

Findings worth remembering

• Needle scored 10% on the 40-row golden set (about chance for 8 classes). Reversing the tool order changed its answers wholesale (15%, 34/40 service_defect), so it picks by position, not content. Confidence is uncorrelated with correctness. Decision: Needle stays dormant. Intended later use: a gate that decides whether a larger model should act on an issue (affects anyone? fixable by a smart model?). Do not use it for categorization.
• Rules scored 39/40 on the golden set, but that is a fit (written after seeing the rows). A fresh 60-row holdout is in cmd/needle-sidecar/eval/holdout.jsonl (from 2026-09-18, unseen); it still needs your expected labels. Known probe-like paths that still land in stale_reference: /ssl/server.key, /_debugbar/open, /debug/pprof, /actuator/*, /graphql, /server-info, /.bash_profile, /settings.py, /__env.js.
• The top error sources were noise: 404 probes and a 401 poller (/api/repos/available, /api/controller/instances, about 700 per hour, counted twice: nginx and the orchestrator).
• IP data: 44 public IPs account for 63% of probe requests (12.5 h window). 34% show a peer of 127.0.0.1 because net.peer.ip is the TCP peer behind the PROXY-protocol hop, so the tracker cannot see them. The ingress has enable-real-ip=true, forwarded-for-header=proxy_protocol, proxy-real-ip-cidr=127.0.0.1/32, so block-cidrs itself works on the real client address.
• Internal addresses are a hard stop in both tracker and blocker (honeypot.Untrackable): loopback, RFC1918, ULA, link-local, multicast, unspecified, IPv4-mapped, CGNAT 100.64/10 (Tailscale), 192.0.0.0/24, 198.18/15, 240/4. The allowlist only adds exclusions.

Before enabling enforcement

1. Deploy the new build (dry-run on). Watch GET /api/blocker/status for a day; compare against go run ./cmd/bad-actors -since ... -until ....
2. Put your own public IP(s) in OTEL_BLOCKER_ALLOW. A blocked IP gets 403, so you would lock yourself out of the UI. Recovery: kubectl edit cm ingress-nginx-controller -n ingress-nginx (remove entries; managed ones are listed in the otel-debug.k6n.net/blocked annotation).
3. Then set OTEL_BLOCKER_DRY_RUN=false.
4. Do not kubectl apply the whole k8s/deployment.yaml blindly: the dry run showed it would create a new otel-sourcemaps PVC not present in the cluster.

Next steps (suggested order)

1. Deploy the collector so normalized rows stay current; verify with SELECT max(timestamp) FROM otel.normalized_events.
2. Label holdout.jsonl (fill expected), run go run ./cmd/triage-eval -input cmd/needle-sidecar/eval/holdout.jsonl, fix the rule families that miss. Then re-export a fresh holdout so scoring stays honest.
3. Warm-up guard in DetectSpikes: require baseline data before alerting (the first window after data begins alerts on everything as NEW).
4. Real client IP in nginx spans, to close the 127.0.0.1 gap (inspect the ingress OTel config first; not looked at yet).
5. Alert delivery: run DetectSpikes on a timer in the collector and post somewhere (UI panel / ntfy / Slack). Category chips and same / different / noise + "mute pattern" in the UI, storing labels.
6. Reduce 401-poller noise: mute list or per-pattern rate limit; client_credential_issue is the caller's fault, likely a lower severity.
7. After 2+ weeks of data: time-of-day baseline, rollout proximity, then the logistic regression / MLP from docs/ML.md; the rule detector is what they must beat.

Update 2026-09-20

• Warm-up guard: SpikeConfig.DataSince / WarmupWindows (default 6). No alert until that much baseline exists; spike-alerts sets it from min(timestamp) of normalized_events. Test: TestDetectSpikesWarmupGuard.

• holdout.jsonl labelled (by Claude, review it; ambiguous 404s such as /mcp, /graphql, /api, /api/v2/* were labelled stale_reference). Rules fixed for the misses: more probe paths (_debugbar, debug/pprof, server-info, actuator, __env, .bash_*, *.key|pem|py, graphql/console) and all 1xx/2xx/3xx are expected_lifecycle. 60/60, but the set is now tuned against, so re-export a fresh holdout before trusting the number. IsProbePath is shared with the blocker, so these also widen what counts as a probe.

• Client IP gap (last 6h of nginx spans): public IPs 10.3k spans (2192 IPs), 10.10.0.1 25k (the user's firewall: traffic it NATs to ingress arrives with the firewall as source), loopback 4.1k. The loopback rows include real external scanners (CameraScannerV3, MSIE8 UAs, raw-IP hosts), so ~10% of spans lose the attacker IP. Ingress has externalTrafficPolicy: Local, no use-proxy-protocol, and x-real-ip is never attached. Cause: something on the node forwards to ingress from 127.0.0.1; not yet identified.

• UA classes: internal/uaclass holds the known crawler / AI agent / monitor / scan-tool list (first match wins; UA is spoofable, so a class is a positive signal only). Stored as normalized_events.ua_class (added by EnsureNormalizedSchema on next start; old rows are ''), served as uaClass on issue samples and endpoint stats. Issues UI: add-filter offers client: X and hide client: X. Not yet a triage/spike feature.

• Trace context (internal/tracectx): one request is several spans in several envelopes (nginx outer span on :442 with the real client, nginx inner span on :8181 from 127.0.0.1, then the app span), so the ingest-time same-envelope lookup in normalizeTraces almost never finds the parent (7 of 1154 nginx spans measured). tracectx.Resolve works on a loaded span set instead and is order independent. It yields ClientIP (first public peer up the chain), IsDuplicate (repeats an ancestor's failure: same status code), IsRootCause, child / upstream-child counts, ParentMissing.

  • bad-actors resolves loopback hops to the real client (839 hop spans / 583 probes recovered in a 3h sample).
  • spike-alerts loads the spans of every trace with an error, drops duplicate hops from the counts and reports distinct clients per pattern; it falls back when the trace columns are absent. Historical rows have empty span_id, so it only helps for data written since the trace columns shipped (~08:09 UTC 2026-09-20).
  • features.Row gained LogicalErrors, DuplicateErrors, RootCauseErrors, DistinctClients, TopClientShare, BotErrorShare (additive). window-features reads optional clientIp/uaClass/isDuplicate/isRootCause/hasUpstream.
  • Trace view in the UI shows the resolved client (lib/trace-client.ts).
  • Not done: the collector still records the hop IP on issue samples at ingest; span name GET / for orchestrator spans is unchanged (changing it alters every orchestrator pattern signature); HasUpstream is loaded but not yet a triage rule; trace propagation is missing in cart/checkout/profile/scheduling (roots only).

Commands

set -a; . ./.env; set +a

# replay the spike rule over a range (Needle off)
go run ./cmd/spike-alerts -since 2026-09-19T12:00:00Z -until 2026-09-19T20:30:00Z -baseline-windows 12
# add -show-muted to see probes / stale / lifecycle patterns too

# who would be blocked
go run ./cmd/bad-actors -since 2026-09-19T08:00:00Z -until 2026-09-19T20:30:00Z

# score the rules against labelled rows
go run ./cmd/triage-eval -input cmd/needle-sidecar/eval/holdout.jsonl

# Needle eval (needs the sidecar running; venv was in scratch and is gone)
pip install cactus-needle   # python 3.12 recommended
NEEDLE_PORT=5005 NEEDLE_HOST=127.0.0.1 python cmd/needle-sidecar/server.py
python cmd/needle-sidecar/eval/run_eval.py

Backfill in small windows only (10 to 30 minutes over the LAN); one-hour windows through a kubectl port-forward hung and dropped.

Git state

Everything is uncommitted. Files changed by this work: internal/{storage, normalization,features,triage,honeypot,blocker}, cmd/{spike-alerts, triage-eval,bad-actors,normalized-backfill,server,needle-sidecar}, k8s/{clickhouse,deployment}.yaml, .gitignore.

Not from this work; check before committing: AGENTS.md, internal/collector/{collector_test,issue_extract}.go, otel-ui/src/** (including a deleted GaugeMeter.tsx). Also tracked __pycache__/*.pyc files under cmd/needle-sidecar show as modified; they should be untracked and gitignored.

IP details (MaxMind)

The issues view's source IP opens a dialog (GET /api/ip/{ip}, authed) with country/region/city, timezone, ASN and network. Lookups are local. Enable by mounting GeoLite2 files and setting OTEL_GEOIP_CITY_DB and/or OTEL_GEOIP_ASN_DB (the Dockerfile copies db/*.mmdb into the image and sets both); without them the dialog says so. Private/loopback addresses are labelled instead of looked up.