# Status: error triage, spike detection, IP blocking (2026-09-19)

Pick-up notes. Nothing below is committed, deployed or applied to the cluster
unless stated. Greenfield, single user: radical changes are fine.

## Access

- ClickHouse (otel instance) is now a **LoadBalancer**: `monitoring/clickhouse`
  at `10.10.3.45` (native 9000, HTTP 8123). Set in `k8s/clickhouse.yaml` and
  patched live. `docs/ML.md` still says "keep ClusterIP"; that is superseded.
- Credentials are in `.env` (gitignored, mode 600). Load with
  `set -a; . ./.env; set +a`. `clickhouse-client` is installed locally.
- `.env` deliberately does not enable Needle; pass `-needle URL` explicitly.

## What exists now

| Piece | Where | State |
| --- | --- | --- |
| Normalized rows written on ingest | `internal/storage/normalized.go`, `Append` in `clickhouse.go` | Code done, tests pass. **Running collector is still the old build**, so nothing new is written until deployed. |
| Live schema migration | `EnsureNormalizedSchema` (runs at collector start) | Applied once by hand via backfill; `issue_fingerprint` etc. now exist. |
| 404/410 are not errors | `isErrorStatus` in `internal/normalization/normalizer.go` | Done. Rows stored before this still count 404s as errors. |
| Backfill | `cmd/normalized-backfill` | Ran 2026-09-19 12:00 to ~21:00 UTC. Queries need `FINAL` (duplicates until merge). |
| Spike rule | `internal/features/spike.go` | Excess-error rule, trimmed-mean baseline, `Suppressor` (cooldown, 2x escalation). Tested. |
| Replay CLI | `cmd/spike-alerts` | Replays a range; mutes non-actionable categories; Needle optional and off. |
| Cause categories | `internal/triage` | Deterministic rules, first match wins. Tested. |
| Rules scoring | `cmd/triage-eval` | Scores a labelled JSONL; also prepares rows for labelling. |
| Offender report | `cmd/bad-actors` | Read-only. Ranks IPs from raw nginx spans, marks `[BLOCK]` / `[INTERNAL]`. |
| Tracker | `internal/honeypot` | Now needs distinct paths (default 3), never tracks internal/reserved IPs, uses `triage.IsProbePath`. |
| Blocker | `internal/blocker` | Writes ingress-nginx `block-cidrs`. **Dry-run by default.** Tested with a fake client. Wired into `cmd/server`. |
| RBAC + env | `k8s/deployment.yaml` | Role limited to the one ConfigMap; blocker env set to enabled + dry-run. Not applied. |

Categories: `scanner_probe`, `dependency_unreachable`, `client_credential_issue`,
`stale_reference`, `service_defect`, `capacity_saturation`, `expected_lifecycle`,
`needs_review`. They are cause-level verdicts, not restatements of status.

## Findings worth remembering

- **Needle scored 10% on the 40-row golden set** (about chance for 8 classes).
  Reversing the tool order changed its answers wholesale (15%, 34/40 `service_defect`),
  so it picks by position, not content. Confidence is uncorrelated with
  correctness. Decision: **Needle stays dormant.** Intended later use: a gate that
  decides whether a larger model should act on an issue (affects anyone? fixable by
  a smart model?). Do not use it for categorization.
- **Rules scored 39/40 on the golden set, but that is a fit** (written after seeing
  the rows). A fresh 60-row holdout is in `cmd/needle-sidecar/eval/holdout.jsonl`
  (from 2026-09-18, unseen); it still needs your `expected` labels. Known
  probe-like paths that still land in `stale_reference`: `/ssl/server.key`,
  `/_debugbar/open`, `/debug/pprof`, `/actuator/*`, `/graphql`, `/server-info`,
  `/.bash_profile`, `/settings.py`, `/__env.js`.
- **The top error sources were noise:** 404 probes and a 401 poller
  (`/api/repos/available`, `/api/controller/instances`, about 700 per hour, counted
  twice: nginx and the orchestrator).
- **IP data:** 44 public IPs account for 63% of probe requests (12.5 h window).
  **34% show a peer of `127.0.0.1`** because `net.peer.ip` is the TCP peer behind
  the PROXY-protocol hop, so the tracker cannot see them. The ingress has
  `enable-real-ip=true`, `forwarded-for-header=proxy_protocol`,
  `proxy-real-ip-cidr=127.0.0.1/32`, so `block-cidrs` itself works on the real
  client address.
- **Internal addresses are a hard stop** in both tracker and blocker
  (`honeypot.Untrackable`): loopback, RFC1918, ULA, link-local, multicast,
  unspecified, IPv4-mapped, CGNAT `100.64/10` (Tailscale), `192.0.0.0/24`,
  `198.18/15`, `240/4`. The allowlist only adds exclusions.

## Before enabling enforcement

1. Deploy the new build (dry-run on). Watch `GET /api/blocker/status` for a day;
   compare against `go run ./cmd/bad-actors -since ... -until ...`.
2. Put your own public IP(s) in `OTEL_BLOCKER_ALLOW`. A blocked IP gets 403, so
   you would lock yourself out of the UI. Recovery: `kubectl edit cm
   ingress-nginx-controller -n ingress-nginx` (remove entries; managed ones are
   listed in the `otel-debug.k6n.net/blocked` annotation).
3. Then set `OTEL_BLOCKER_DRY_RUN=false`.
4. Do not `kubectl apply` the whole `k8s/deployment.yaml` blindly: the dry run
   showed it would create a new `otel-sourcemaps` PVC not present in the cluster.

## Next steps (suggested order)

1. Deploy the collector so normalized rows stay current; verify with
   `SELECT max(timestamp) FROM otel.normalized_events`.
2. Label `holdout.jsonl` (fill `expected`), run
   `go run ./cmd/triage-eval -input cmd/needle-sidecar/eval/holdout.jsonl`, fix
   the rule families that miss. Then re-export a fresh holdout so scoring stays
   honest.
3. Warm-up guard in `DetectSpikes`: require baseline data before alerting (the
   first window after data begins alerts on everything as NEW).
4. Real client IP in nginx spans, to close the 127.0.0.1 gap (inspect the
   ingress OTel config first; not looked at yet).
5. Alert delivery: run `DetectSpikes` on a timer in the collector and post
   somewhere (UI panel / ntfy / Slack). Category chips and `same / different /
   noise` + "mute pattern" in the UI, storing labels.
6. Reduce 401-poller noise: mute list or per-pattern rate limit;
   `client_credential_issue` is the caller's fault, likely a lower severity.
7. After 2+ weeks of data: time-of-day baseline, rollout proximity, then the
   logistic regression / MLP from `docs/ML.md`; the rule detector is what they
   must beat.

## Commands

```bash
set -a; . ./.env; set +a

# replay the spike rule over a range (Needle off)
go run ./cmd/spike-alerts -since 2026-09-19T12:00:00Z -until 2026-09-19T20:30:00Z -baseline-windows 12
# add -show-muted to see probes / stale / lifecycle patterns too

# who would be blocked
go run ./cmd/bad-actors -since 2026-09-19T08:00:00Z -until 2026-09-19T20:30:00Z

# score the rules against labelled rows
go run ./cmd/triage-eval -input cmd/needle-sidecar/eval/holdout.jsonl

# Needle eval (needs the sidecar running; venv was in scratch and is gone)
pip install cactus-needle   # python 3.12 recommended
NEEDLE_PORT=5005 NEEDLE_HOST=127.0.0.1 python cmd/needle-sidecar/server.py
python cmd/needle-sidecar/eval/run_eval.py
```

Backfill in small windows only (10 to 30 minutes over the LAN); one-hour windows
through a `kubectl port-forward` hung and dropped.

## Git state

Everything is uncommitted. Files changed by this work: `internal/{storage,
normalization,features,triage,honeypot,blocker}`, `cmd/{spike-alerts,
triage-eval,bad-actors,normalized-backfill,server,needle-sidecar}`,
`k8s/{clickhouse,deployment}.yaml`, `.gitignore`.

**Not from this work; check before committing:** `AGENTS.md`,
`internal/collector/{collector_test,issue_extract}.go`, `otel-ui/src/**`
(including a deleted `GaugeMeter.tsx`). Also tracked `__pycache__/*.pyc` files
under `cmd/needle-sidecar` show as modified; they should be untracked and
gitignored.