all the changes

docs/STATUS.md

@@ -0,0 +1,132 @@
+# Status: error triage, spike detection, IP blocking (2026-09-19)
+
+Pick-up notes. Nothing below is committed, deployed or applied to the cluster
+unless stated. Greenfield, single user: radical changes are fine.
+
+## Access
+
+- ClickHouse (otel instance) is now a **LoadBalancer**: `monitoring/clickhouse`
+  at `10.10.3.45` (native 9000, HTTP 8123). Set in `k8s/clickhouse.yaml` and
+  patched live. `docs/ML.md` still says "keep ClusterIP"; that is superseded.
+- Credentials are in `.env` (gitignored, mode 600). Load with
+  `set -a; . ./.env; set +a`. `clickhouse-client` is installed locally.
+- `.env` deliberately does not enable Needle; pass `-needle URL` explicitly.
+
+## What exists now
+
+| Piece | Where | State |
+| --- | --- | --- |
+| Normalized rows written on ingest | `internal/storage/normalized.go`, `Append` in `clickhouse.go` | Code done, tests pass. **Running collector is still the old build**, so nothing new is written until deployed. |
+| Live schema migration | `EnsureNormalizedSchema` (runs at collector start) | Applied once by hand via backfill; `issue_fingerprint` etc. now exist. |
+| 404/410 are not errors | `isErrorStatus` in `internal/normalization/normalizer.go` | Done. Rows stored before this still count 404s as errors. |
+| Backfill | `cmd/normalized-backfill` | Ran 2026-09-19 12:00 to ~21:00 UTC. Queries need `FINAL` (duplicates until merge). |
+| Spike rule | `internal/features/spike.go` | Excess-error rule, trimmed-mean baseline, `Suppressor` (cooldown, 2x escalation). Tested. |
+| Replay CLI | `cmd/spike-alerts` | Replays a range; mutes non-actionable categories; Needle optional and off. |
+| Cause categories | `internal/triage` | Deterministic rules, first match wins. Tested. |
+| Rules scoring | `cmd/triage-eval` | Scores a labelled JSONL; also prepares rows for labelling. |
+| Offender report | `cmd/bad-actors` | Read-only. Ranks IPs from raw nginx spans, marks `[BLOCK]` / `[INTERNAL]`. |
+| Tracker | `internal/honeypot` | Now needs distinct paths (default 3), never tracks internal/reserved IPs, uses `triage.IsProbePath`. |
+| Blocker | `internal/blocker` | Writes ingress-nginx `block-cidrs`. **Dry-run by default.** Tested with a fake client. Wired into `cmd/server`. |
+| RBAC + env | `k8s/deployment.yaml` | Role limited to the one ConfigMap; blocker env set to enabled + dry-run. Not applied. |
+
+Categories: `scanner_probe`, `dependency_unreachable`, `client_credential_issue`,
+`stale_reference`, `service_defect`, `capacity_saturation`, `expected_lifecycle`,
+`needs_review`. They are cause-level verdicts, not restatements of status.
+
+## Findings worth remembering
+
+- **Needle scored 10% on the 40-row golden set** (about chance for 8 classes).
+  Reversing the tool order changed its answers wholesale (15%, 34/40 `service_defect`),
+  so it picks by position, not content. Confidence is uncorrelated with
+  correctness. Decision: **Needle stays dormant.** Intended later use: a gate that
+  decides whether a larger model should act on an issue (affects anyone? fixable by
+  a smart model?). Do not use it for categorization.
+- **Rules scored 39/40 on the golden set, but that is a fit** (written after seeing
+  the rows). A fresh 60-row holdout is in `cmd/needle-sidecar/eval/holdout.jsonl`
+  (from 2026-09-18, unseen); it still needs your `expected` labels. Known
+  probe-like paths that still land in `stale_reference`: `/ssl/server.key`,
+  `/_debugbar/open`, `/debug/pprof`, `/actuator/*`, `/graphql`, `/server-info`,
+  `/.bash_profile`, `/settings.py`, `/__env.js`.
+- **The top error sources were noise:** 404 probes and a 401 poller
+  (`/api/repos/available`, `/api/controller/instances`, about 700 per hour, counted
+  twice: nginx and the orchestrator).
+- **IP data:** 44 public IPs account for 63% of probe requests (12.5 h window).
+  **34% show a peer of `127.0.0.1`** because `net.peer.ip` is the TCP peer behind
+  the PROXY-protocol hop, so the tracker cannot see them. The ingress has
+  `enable-real-ip=true`, `forwarded-for-header=proxy_protocol`,
+  `proxy-real-ip-cidr=127.0.0.1/32`, so `block-cidrs` itself works on the real
+  client address.
+- **Internal addresses are a hard stop** in both tracker and blocker
+  (`honeypot.Untrackable`): loopback, RFC1918, ULA, link-local, multicast,
+  unspecified, IPv4-mapped, CGNAT `100.64/10` (Tailscale), `192.0.0.0/24`,
+  `198.18/15`, `240/4`. The allowlist only adds exclusions.
+
+## Before enabling enforcement
+
+1. Deploy the new build (dry-run on). Watch `GET /api/blocker/status` for a day;
+   compare against `go run ./cmd/bad-actors -since ... -until ...`.
+2. Put your own public IP(s) in `OTEL_BLOCKER_ALLOW`. A blocked IP gets 403, so
+   you would lock yourself out of the UI. Recovery: `kubectl edit cm
+   ingress-nginx-controller -n ingress-nginx` (remove entries; managed ones are
+   listed in the `otel-debug.k6n.net/blocked` annotation).
+3. Then set `OTEL_BLOCKER_DRY_RUN=false`.
+4. Do not `kubectl apply` the whole `k8s/deployment.yaml` blindly: the dry run
+   showed it would create a new `otel-sourcemaps` PVC not present in the cluster.
+
+## Next steps (suggested order)
+
+1. Deploy the collector so normalized rows stay current; verify with
+   `SELECT max(timestamp) FROM otel.normalized_events`.
+2. Label `holdout.jsonl` (fill `expected`), run
+   `go run ./cmd/triage-eval -input cmd/needle-sidecar/eval/holdout.jsonl`, fix
+   the rule families that miss. Then re-export a fresh holdout so scoring stays
+   honest.
+3. Warm-up guard in `DetectSpikes`: require baseline data before alerting (the
+   first window after data begins alerts on everything as NEW).
+4. Real client IP in nginx spans, to close the 127.0.0.1 gap (inspect the
+   ingress OTel config first; not looked at yet).
+5. Alert delivery: run `DetectSpikes` on a timer in the collector and post
+   somewhere (UI panel / ntfy / Slack). Category chips and `same / different /
+   noise` + "mute pattern" in the UI, storing labels.
+6. Reduce 401-poller noise: mute list or per-pattern rate limit;
+   `client_credential_issue` is the caller's fault, likely a lower severity.
+7. After 2+ weeks of data: time-of-day baseline, rollout proximity, then the
+   logistic regression / MLP from `docs/ML.md`; the rule detector is what they
+   must beat.
+
+## Commands
+
+```bash
+set -a; . ./.env; set +a
+
+# replay the spike rule over a range (Needle off)
+go run ./cmd/spike-alerts -since 2026-09-19T12:00:00Z -until 2026-09-19T20:30:00Z -baseline-windows 12
+# add -show-muted to see probes / stale / lifecycle patterns too
+
+# who would be blocked
+go run ./cmd/bad-actors -since 2026-09-19T08:00:00Z -until 2026-09-19T20:30:00Z
+
+# score the rules against labelled rows
+go run ./cmd/triage-eval -input cmd/needle-sidecar/eval/holdout.jsonl
+
+# Needle eval (needs the sidecar running; venv was in scratch and is gone)
+pip install cactus-needle   # python 3.12 recommended
+NEEDLE_PORT=5005 NEEDLE_HOST=127.0.0.1 python cmd/needle-sidecar/server.py
+python cmd/needle-sidecar/eval/run_eval.py
+```
+
+Backfill in small windows only (10 to 30 minutes over the LAN); one-hour windows
+through a `kubectl port-forward` hung and dropped.
+
+## Git state
+
+Everything is uncommitted. Files changed by this work: `internal/{storage,
+normalization,features,triage,honeypot,blocker}`, `cmd/{spike-alerts,
+triage-eval,bad-actors,normalized-backfill,server,needle-sidecar}`,
+`k8s/{clickhouse,deployment}.yaml`, `.gitignore`.
+
+**Not from this work; check before committing:** `AGENTS.md`,
+`internal/collector/{collector_test,issue_extract}.go`, `otel-ui/src/**`
+(including a deleted `GaugeMeter.tsx`). Also tracked `__pycache__/*.pyc` files
+under `cmd/needle-sidecar` show as modified; they should be untracked and
+gitignored.