go-otel/docs/ML.md

# Error detection model

> Latest status, decisions and next steps: [STATUS.md](STATUS.md). Where they
> differ (ClickHouse is now a LoadBalancer; Needle is removed), STATUS.md wins.

## Current handoff (2026-09-18)

The first iteration is intentionally explainable and deterministic. Raw OTLP
events remain the source of truth; normalized rows are stored in
`otel.normalized_events` with deterministic `event_id`, `exact_signature`, and
`pattern_signature` values.

Implemented commands:

```bash
# Read-only normalization summary for a bounded raw window.
go run ./cmd/normalized-backfill \
  -since 2026-09-18T17:00:00Z \
  -until 2026-09-18T18:00:00Z

# Explicit normalized-table write; use one-hour windows.
go run ./cmd/normalized-backfill \
  -since 2026-09-18T17:00:00Z \
  -until 2026-09-18T18:00:00Z -write

# Export a capped training sample.
go run ./cmd/training-sample \
  -since 2026-09-18T17:00:00Z \
  -until 2026-09-18T22:00:00Z \
  -max-per-pattern 10 -max-events 1000 \
  -output /tmp/training-v001.jsonl

# Train and score the explainable prototype model.
go run ./cmd/issue-model -train /tmp/training-v001.jsonl \
  -model /tmp/issue-model-v001.json
go run ./cmd/issue-model -model /tmp/issue-model-v001.json \
  -predict /tmp/sample.jsonl -min-score 0.75 -min-margin 0.05

# Inspect rolling features (the five-hour sample is too short to label).
go run ./cmd/window-features -input /tmp/training-v001.jsonl \
  -window 5m -step 5m -future 15m \
  -until 2026-09-18T22:00:00Z -include-unknown-labels
```

`window-features` emits one fixed-width row per service/release/window. It
includes counts, error rate, span duration percentiles, kind and HTTP-status
counts, unique trace and issue counts, frontend/backend error counts, and
retry/timeout/cancellation counts. Its label is generated from the following
fifteen minutes and a same-time-of-day trailing baseline. Rows without seven
available baseline days or a complete future interval are withheld by default;
pass `-include-unknown-labels` only for inspection, never as training labels.
Supply the source export's exclusive `-until` watermark so a quiet final
interval can be labelled without relying on the timestamp of its last event.

For fast candidate/persistence analysis, override the observation and step
durations explicitly (for example `-window 15s -step 5s`). Those short rows
are not substitutes for the documented five-minute spike-prediction label.

The current prototype was trained on 17:00–22:00 UTC and tested on a separate
05:00–06:00 UTC sample. It achieved 45% top-1 accuracy against deterministic
weak labels. This is a pipeline check, not a production model evaluation.

The default similarity gate is provisional: score `>= 0.75` and top/runner-up
margin `>= 0.05`. Recalibrate it after human labels are available; select the
threshold for the desired precision, not by optimizing accuracy.

Use 15 seconds with a 5-second step as the fast window for the observed
five-second polling pattern. Add 30-second confirmation and five-minute trend
windows. A single event is a candidate; persistence across windows makes it
an issue.

## Next session

1. Export a clean five-hour JSONL sample and review 20–50 predictions.
2. Add `same`, `different`, and `unknown/noise` labels in the UI.
3. Add persistence and rollout proximity to the window aggregates; count,
   error count/rate, unique traces, issue counts, latency, sources and status
   classes are already exported.
4. Re-evaluate the threshold on reviewed labels.
5. Keep the model in shadow mode until it beats the deterministic baseline.

Do not run an unrestricted backfill from a workstation port-forward. Use
one-hour windows or an authenticated in-cluster job. The normalized table has
some smoke-test rows; always select training data by an explicit timestamp
window.

## Current decision

Keep the original OTLP payloads as the source of truth, but derive a stable
feature row for each service/release/time window. The model should not consume
raw OTLP JSON, and it does not need to classify every span or log message into
a semantic taxonomy first.

The first target is an **error-spike predictor**:

> Given the telemetry observed for a service in a five-minute window, predict
> whether the following fifteen-minute window contains an abnormal error rate.

This is more useful than classifying an event that the collector already knows
is an error.

## Normalized event contract

The extraction layer should reduce spans, logs, and metrics to these stable
dimensions before aggregation:

| Field | Meaning |
| --- | --- |
| `window_start` | UTC five-minute bucket |
| `service` | `service.name`, or `unknown` |
| `release` | `service.release`, or empty |
| `environment` | deployment environment, when present |
| `kind` | `span`, `log`, or `metric` |
| `is_error` | explicit error status, error HTTP/gRPC status, or error severity |
| `status_code` | HTTP status when present |
| `severity_number` | OTLP log severity, when present |
| `duration_ms` | span duration, when present |
| `route` | normalized HTTP route, when present |
| `source` | `frontend` or `backend` |
| `trace_id` | correlation identifier, when present |
| `issue_fingerprint` | existing issue fingerprint, when available |

PII-bearing fields such as raw messages, URLs, user agents, source IPs, and
stack traces must not be model features. They can remain in the raw history and
issue views subject to the existing retention and access controls.

The normalized row now carries the canonical `issue_fingerprint` for errors,
plus structured `is_retry`, `is_timeout`, and `is_cancellation` flags. The
flags are derived from OTLP attributes and protocol status—not message text—so
the corresponding aggregate features do not depend on raw user content.

## First aggregate feature row

For each service/release/window, derive:

- total event count and error count
- error rate
- p50/p95 span duration
- counts by kind (`span`, `log`, `metric`)
- counts by HTTP status class (`2xx`, `4xx`, `5xx`)
- unique issue count and newly observed issue count
- frontend and backend error counts
- retry, timeout, and cancellation counts where attributes identify them
- release age, if deployment time is available

The model input should be numeric and fixed-width. Categorical service and
release values can initially be filtered per service; one global model can be
considered after there is enough multi-service data.

## UI verification

The **Issues** view has a **Test spike detector** button. It calls the
authenticated `POST /api/ml/test-spike` endpoint, which creates no telemetry
and writes nothing to storage. It runs a deterministic scenario through the
same feature and label builder: seven daily 10% baseline windows followed by a
future interval containing five errors in ten events. A passing result shows
the baseline, future counts/rate, and the positive spike label in the UI.

## Finding similar errors and spans

Similarity should be a separate concern from the primary issue fingerprint.
Use deterministic grouping for the canonical issue ID because it is stable,
explainable, cheap at ingest time, and safe for alert counts. The current
`issues.NormalizeTitle` already removes common variable values such as UUIDs,
hex identifiers, and request numbers.

The recommended matching stages are:

1. **Canonical normalization**: lowercase and normalize IDs, numbers, quoted
   values, URLs, and whitespace in messages and span names.
2. **Structured signature**: combine service, event kind, normalized route or
   span name, exception type, HTTP/gRPC status class, and the top application
   stack frames. This avoids merging unrelated errors that happen to contain
   the same words.
3. **Exact fingerprint**: use the structured signature for the canonical issue
   ID and deterministic deduplication.
4. **Near-match suggestions**: compare only within a bounded candidate group,
   such as the same service and event kind. Token n-gram similarity, SimHash,
   or weighted field similarity can suggest related issues without changing
   their IDs.

For example, these should usually be suggested as similar but remain separate
until confirmed:

```text
checkout: database timeout after 2000ms
checkout: database timeout after 5000ms
```

They may represent the same underlying failure, but the timeout value could be
operationally important. A review UI can offer “merge” or “not related”; those
decisions become valuable labels later.

Embeddings or a neural similarity model are reasonable as a second phase for
semantic discovery, especially when messages use different wording. They
should not be the first grouping mechanism: they add model/version drift,
latency, explainability, and possible sensitive-data leakage. If introduced,
embed redacted normalized text and stack signatures, retrieve nearest
neighbors, and present the result as a suggestion rather than silently
merging issues.

For spans, the strongest similarity fields are usually the normalized span
name or route, service, status class, exception type, and stack-frame signature.
Trace IDs, span IDs, timestamps, and request-specific values should never
contribute to similarity.

## Label

For a row at time `t`, label it positive when the future interval `[t+5m,
t+20m)` satisfies both conditions:

1. error rate is at least three times the trailing seven-day baseline for that
   service and time-of-day; and
2. the future error count is at least 5.

The count floor prevents a single error in a very low-volume service from
becoming an incident. The baseline must be calculated using data before `t`
only.

## Baseline before neural network

Start with two baselines:

1. a trailing-rate/spike rule; and
2. logistic regression using the same aggregate features.

Then compare a small MLP (`32 -> 16 -> 1`, ReLU plus sigmoid) against them.
Use a chronological train/validation/test split, and report precision, recall,
PR-AUC, and false alerts per service. Accuracy is not meaningful for this
imbalanced target.

## Data readiness

The current cluster ClickHouse instance has approximately 168k entries across
18 services, but only about 13 hours of history at the time this document was
created. It is sufficient to test normalization and aggregation, but not to
train the seven-day-baseline detector. Let the normalized feature table collect
at least two weeks of data before evaluating a model; four weeks is preferable.

ClickHouse should remain a ClusterIP service while this work is developed. A
LoadBalancer would expose the receiver's OTLP and debug surfaces and is not
needed for model development; use a temporary `kubectl port-forward` or an
authenticated internal job for inspection and training data export.