go-otel/docs/STATUS.md

Status: error triage, spike detection, IP blocking (2026-09-19)

Pick-up notes. Nothing below is committed, deployed or applied to the cluster unless stated. Greenfield, single user: radical changes are fine.

Access

• ClickHouse (otel instance) is now a LoadBalancer: monitoring/clickhouse at 10.10.3.45 (native 9000, HTTP 8123). Set in k8s/clickhouse.yaml and patched live. docs/ML.md still says "keep ClusterIP"; that is superseded.
• Credentials are in .env (gitignored, mode 600). Load with set -a; . ./.env; set +a. clickhouse-client is installed locally.
• .env deliberately does not enable Needle; pass -needle URL explicitly.

What exists now

Piece	Where	State
Normalized rows written on ingest	internal/storage/normalized.go, Append in clickhouse.go	Code done, tests pass. Running collector is still the old build, so nothing new is written until deployed.
Live schema migration	EnsureNormalizedSchema (runs at collector start)	Applied once by hand via backfill; issue_fingerprint etc. now exist.
404/410 are not errors	isErrorStatus in internal/normalization/normalizer.go	Done. Rows stored before this still count 404s as errors.
Backfill	cmd/normalized-backfill	Ran 2026-09-19 12:00 to ~21:00 UTC. Queries need FINAL (duplicates until merge).
Spike rule	internal/features/spike.go	Excess-error rule, trimmed-mean baseline, Suppressor (cooldown, 2x escalation). Tested.
Replay CLI	cmd/spike-alerts	Replays a range; mutes non-actionable categories; Needle optional and off.
Cause categories	internal/triage	Deterministic rules, first match wins. Tested.
Rules scoring	cmd/triage-eval	Scores a labelled JSONL; also prepares rows for labelling.
Offender report	cmd/bad-actors	Read-only. Ranks IPs from raw nginx spans, marks [BLOCK] / [INTERNAL].
Tracker	internal/honeypot	Now needs distinct paths (default 3), never tracks internal/reserved IPs, uses triage.IsProbePath.
Blocker	internal/blocker	Writes ingress-nginx block-cidrs. Dry-run by default. Tested with a fake client. Wired into cmd/server.
RBAC + env	k8s/deployment.yaml	Role limited to the one ConfigMap; blocker env set to enabled + dry-run. Not applied.

Categories: scanner_probe, dependency_unreachable, client_credential_issue, stale_reference, service_defect, capacity_saturation, expected_lifecycle, needs_review. They are cause-level verdicts, not restatements of status.

Findings worth remembering

• Needle scored 10% on the 40-row golden set (about chance for 8 classes). Reversing the tool order changed its answers wholesale (15%, 34/40 service_defect), so it picks by position, not content. Confidence is uncorrelated with correctness. Decision: Needle stays dormant. Intended later use: a gate that decides whether a larger model should act on an issue (affects anyone? fixable by a smart model?). Do not use it for categorization.
• Rules scored 39/40 on the golden set, but that is a fit (written after seeing the rows). A fresh 60-row holdout is in cmd/needle-sidecar/eval/holdout.jsonl (from 2026-09-18, unseen); it still needs your expected labels. Known probe-like paths that still land in stale_reference: /ssl/server.key, /_debugbar/open, /debug/pprof, /actuator/*, /graphql, /server-info, /.bash_profile, /settings.py, /__env.js.
• The top error sources were noise: 404 probes and a 401 poller (/api/repos/available, /api/controller/instances, about 700 per hour, counted twice: nginx and the orchestrator).
• IP data: 44 public IPs account for 63% of probe requests (12.5 h window). 34% show a peer of 127.0.0.1 because net.peer.ip is the TCP peer behind the PROXY-protocol hop, so the tracker cannot see them. The ingress has enable-real-ip=true, forwarded-for-header=proxy_protocol, proxy-real-ip-cidr=127.0.0.1/32, so block-cidrs itself works on the real client address.
• Internal addresses are a hard stop in both tracker and blocker (honeypot.Untrackable): loopback, RFC1918, ULA, link-local, multicast, unspecified, IPv4-mapped, CGNAT 100.64/10 (Tailscale), 192.0.0.0/24, 198.18/15, 240/4. The allowlist only adds exclusions.

Before enabling enforcement

1. Deploy the new build (dry-run on). Watch GET /api/blocker/status for a day; compare against go run ./cmd/bad-actors -since ... -until ....
2. Put your own public IP(s) in OTEL_BLOCKER_ALLOW. A blocked IP gets 403, so you would lock yourself out of the UI. Recovery: kubectl edit cm ingress-nginx-controller -n ingress-nginx (remove entries; managed ones are listed in the otel-debug.k6n.net/blocked annotation).
3. Then set OTEL_BLOCKER_DRY_RUN=false.
4. Do not kubectl apply the whole k8s/deployment.yaml blindly: the dry run showed it would create a new otel-sourcemaps PVC not present in the cluster.

Next steps (suggested order)

1. Deploy the collector so normalized rows stay current; verify with SELECT max(timestamp) FROM otel.normalized_events.
2. Label holdout.jsonl (fill expected), run go run ./cmd/triage-eval -input cmd/needle-sidecar/eval/holdout.jsonl, fix the rule families that miss. Then re-export a fresh holdout so scoring stays honest.
3. Warm-up guard in DetectSpikes: require baseline data before alerting (the first window after data begins alerts on everything as NEW).
4. Real client IP in nginx spans, to close the 127.0.0.1 gap (inspect the ingress OTel config first; not looked at yet).
5. Alert delivery: run DetectSpikes on a timer in the collector and post somewhere (UI panel / ntfy / Slack). Category chips and same / different / noise + "mute pattern" in the UI, storing labels.
6. Reduce 401-poller noise: mute list or per-pattern rate limit; client_credential_issue is the caller's fault, likely a lower severity.
7. After 2+ weeks of data: time-of-day baseline, rollout proximity, then the logistic regression / MLP from docs/ML.md; the rule detector is what they must beat.

Commands

set -a; . ./.env; set +a

# replay the spike rule over a range (Needle off)
go run ./cmd/spike-alerts -since 2026-09-19T12:00:00Z -until 2026-09-19T20:30:00Z -baseline-windows 12
# add -show-muted to see probes / stale / lifecycle patterns too

# who would be blocked
go run ./cmd/bad-actors -since 2026-09-19T08:00:00Z -until 2026-09-19T20:30:00Z

# score the rules against labelled rows
go run ./cmd/triage-eval -input cmd/needle-sidecar/eval/holdout.jsonl

# Needle eval (needs the sidecar running; venv was in scratch and is gone)
pip install cactus-needle   # python 3.12 recommended
NEEDLE_PORT=5005 NEEDLE_HOST=127.0.0.1 python cmd/needle-sidecar/server.py
python cmd/needle-sidecar/eval/run_eval.py

Backfill in small windows only (10 to 30 minutes over the LAN); one-hour windows through a kubectl port-forward hung and dropped.

Git state

Everything is uncommitted. Files changed by this work: internal/{storage, normalization,features,triage,honeypot,blocker}, cmd/{spike-alerts, triage-eval,bad-actors,normalized-backfill,server,needle-sidecar}, k8s/{clickhouse,deployment}.yaml, .gitignore.

Not from this work; check before committing: AGENTS.md, internal/collector/{collector_test,issue_extract}.go, otel-ui/src/** (including a deleted GaugeMeter.tsx). Also tracked __pycache__/*.pyc files under cmd/needle-sidecar show as modified; they should be untracked and gitignored.