ml

docs/ML.md

@@ -1,5 +1,86 @@
 # Error detection model
 
+## Current handoff (2026-09-18)
+
+The first iteration is intentionally explainable and deterministic. Raw OTLP
+events remain the source of truth; normalized rows are stored in
+`otel.normalized_events` with deterministic `event_id`, `exact_signature`, and
+`pattern_signature` values.
+
+Implemented commands:
+
+```bash
+# Read-only normalization summary for a bounded raw window.
+go run ./cmd/normalized-backfill \
+  -since 2026-09-18T17:00:00Z \
+  -until 2026-09-18T18:00:00Z
+
+# Explicit normalized-table write; use one-hour windows.
+go run ./cmd/normalized-backfill \
+  -since 2026-09-18T17:00:00Z \
+  -until 2026-09-18T18:00:00Z -write
+
+# Export a capped training sample.
+go run ./cmd/training-sample \
+  -since 2026-09-18T17:00:00Z \
+  -until 2026-09-18T22:00:00Z \
+  -max-per-pattern 10 -max-events 1000 \
+  -output /tmp/training-v001.jsonl
+
+# Train and score the explainable prototype model.
+go run ./cmd/issue-model -train /tmp/training-v001.jsonl \
+  -model /tmp/issue-model-v001.json
+go run ./cmd/issue-model -model /tmp/issue-model-v001.json \
+  -predict /tmp/sample.jsonl -min-score 0.75 -min-margin 0.05
+
+# Inspect rolling features (the five-hour sample is too short to label).
+go run ./cmd/window-features -input /tmp/training-v001.jsonl \
+  -window 5m -step 5m -future 15m \
+  -until 2026-09-18T22:00:00Z -include-unknown-labels
+```
+
+`window-features` emits one fixed-width row per service/release/window. It
+includes counts, error rate, span duration percentiles, kind and HTTP-status
+counts, unique trace and issue counts, frontend/backend error counts, and
+retry/timeout/cancellation counts. Its label is generated from the following
+fifteen minutes and a same-time-of-day trailing baseline. Rows without seven
+available baseline days or a complete future interval are withheld by default;
+pass `-include-unknown-labels` only for inspection, never as training labels.
+Supply the source export's exclusive `-until` watermark so a quiet final
+interval can be labelled without relying on the timestamp of its last event.
+
+For fast candidate/persistence analysis, override the observation and step
+durations explicitly (for example `-window 15s -step 5s`). Those short rows
+are not substitutes for the documented five-minute spike-prediction label.
+
+The current prototype was trained on 17:00–22:00 UTC and tested on a separate
+05:00–06:00 UTC sample. It achieved 45% top-1 accuracy against deterministic
+weak labels. This is a pipeline check, not a production model evaluation.
+
+The default similarity gate is provisional: score `>= 0.75` and top/runner-up
+margin `>= 0.05`. Recalibrate it after human labels are available; select the
+threshold for the desired precision, not by optimizing accuracy.
+
+Use 15 seconds with a 5-second step as the fast window for the observed
+five-second polling pattern. Add 30-second confirmation and five-minute trend
+windows. A single event is a candidate; persistence across windows makes it
+an issue.
+
+## Next session
+
+1. Export a clean five-hour JSONL sample and review 20–50 predictions.
+2. Add `same`, `different`, and `unknown/noise` labels in the UI.
+3. Add persistence and rollout proximity to the window aggregates; count,
+   error count/rate, unique traces, issue counts, latency, sources and status
+   classes are already exported.
+4. Re-evaluate the threshold on reviewed labels.
+5. Keep the model in shadow mode until it beats the deterministic baseline.
+
+Do not run an unrestricted backfill from a workstation port-forward. Use
+one-hour windows or an authenticated in-cluster job. The normalized table has
+some smoke-test rows; always select training data by an explicit timestamp
+window.
+
 ## Current decision
 
 Keep the original OTLP payloads as the source of truth, but derive a stable
@@ -40,6 +121,11 @@ PII-bearing fields such as raw messages, URLs, user agents, source IPs, and
 stack traces must not be model features. They can remain in the raw history and
 issue views subject to the existing retention and access controls.
 
+The normalized row now carries the canonical `issue_fingerprint` for errors,
+plus structured `is_retry`, `is_timeout`, and `is_cancellation` flags. The
+flags are derived from OTLP attributes and protocol status—not message text—so
+the corresponding aggregate features do not depend on raw user content.
+
 ## First aggregate feature row
 
 For each service/release/window, derive:
@@ -58,6 +144,15 @@ The model input should be numeric and fixed-width. Categorical service and
 release values can initially be filtered per service; one global model can be
 considered after there is enough multi-service data.
 
+## UI verification
+
+The **Issues** view has a **Test spike detector** button. It calls the
+authenticated `POST /api/ml/test-spike` endpoint, which creates no telemetry
+and writes nothing to storage. It runs a deterministic scenario through the
+same feature and label builder: seven daily 10% baseline windows followed by a
+future interval containing five errors in ten events. A passing result shows
+the baseline, future counts/rate, and the positive spike label in the UI.
+
 ## Finding similar errors and spans
 
 Similarity should be a separate concern from the primary issue fingerprint.