more otel

docs/ML.md

@@ -0,0 +1,145 @@
+# Error detection model
+
+## Current decision
+
+Keep the original OTLP payloads as the source of truth, but derive a stable
+feature row for each service/release/time window. The model should not consume
+raw OTLP JSON, and it does not need to classify every span or log message into
+a semantic taxonomy first.
+
+The first target is an **error-spike predictor**:
+
+> Given the telemetry observed for a service in a five-minute window, predict
+> whether the following fifteen-minute window contains an abnormal error rate.
+
+This is more useful than classifying an event that the collector already knows
+is an error.
+
+## Normalized event contract
+
+The extraction layer should reduce spans, logs, and metrics to these stable
+dimensions before aggregation:
+
+| Field | Meaning |
+| --- | --- |
+| `window_start` | UTC five-minute bucket |
+| `service` | `service.name`, or `unknown` |
+| `release` | `service.release`, or empty |
+| `environment` | deployment environment, when present |
+| `kind` | `span`, `log`, or `metric` |
+| `is_error` | explicit error status, error HTTP/gRPC status, or error severity |
+| `status_code` | HTTP status when present |
+| `severity_number` | OTLP log severity, when present |
+| `duration_ms` | span duration, when present |
+| `route` | normalized HTTP route, when present |
+| `source` | `frontend` or `backend` |
+| `trace_id` | correlation identifier, when present |
+| `issue_fingerprint` | existing issue fingerprint, when available |
+
+PII-bearing fields such as raw messages, URLs, user agents, source IPs, and
+stack traces must not be model features. They can remain in the raw history and
+issue views subject to the existing retention and access controls.
+
+## First aggregate feature row
+
+For each service/release/window, derive:
+
+- total event count and error count
+- error rate
+- p50/p95 span duration
+- counts by kind (`span`, `log`, `metric`)
+- counts by HTTP status class (`2xx`, `4xx`, `5xx`)
+- unique issue count and newly observed issue count
+- frontend and backend error counts
+- retry, timeout, and cancellation counts where attributes identify them
+- release age, if deployment time is available
+
+The model input should be numeric and fixed-width. Categorical service and
+release values can initially be filtered per service; one global model can be
+considered after there is enough multi-service data.
+
+## Finding similar errors and spans
+
+Similarity should be a separate concern from the primary issue fingerprint.
+Use deterministic grouping for the canonical issue ID because it is stable,
+explainable, cheap at ingest time, and safe for alert counts. The current
+`issues.NormalizeTitle` already removes common variable values such as UUIDs,
+hex identifiers, and request numbers.
+
+The recommended matching stages are:
+
+1. **Canonical normalization**: lowercase and normalize IDs, numbers, quoted
+   values, URLs, and whitespace in messages and span names.
+2. **Structured signature**: combine service, event kind, normalized route or
+   span name, exception type, HTTP/gRPC status class, and the top application
+   stack frames. This avoids merging unrelated errors that happen to contain
+   the same words.
+3. **Exact fingerprint**: use the structured signature for the canonical issue
+   ID and deterministic deduplication.
+4. **Near-match suggestions**: compare only within a bounded candidate group,
+   such as the same service and event kind. Token n-gram similarity, SimHash,
+   or weighted field similarity can suggest related issues without changing
+   their IDs.
+
+For example, these should usually be suggested as similar but remain separate
+until confirmed:
+
+```text
+checkout: database timeout after 2000ms
+checkout: database timeout after 5000ms
+```
+
+They may represent the same underlying failure, but the timeout value could be
+operationally important. A review UI can offer “merge” or “not related”; those
+decisions become valuable labels later.
+
+Embeddings or a neural similarity model are reasonable as a second phase for
+semantic discovery, especially when messages use different wording. They
+should not be the first grouping mechanism: they add model/version drift,
+latency, explainability, and possible sensitive-data leakage. If introduced,
+embed redacted normalized text and stack signatures, retrieve nearest
+neighbors, and present the result as a suggestion rather than silently
+merging issues.
+
+For spans, the strongest similarity fields are usually the normalized span
+name or route, service, status class, exception type, and stack-frame signature.
+Trace IDs, span IDs, timestamps, and request-specific values should never
+contribute to similarity.
+
+## Label
+
+For a row at time `t`, label it positive when the future interval `[t+5m,
+t+20m)` satisfies both conditions:
+
+1. error rate is at least three times the trailing seven-day baseline for that
+   service and time-of-day; and
+2. the future error count is at least 5.
+
+The count floor prevents a single error in a very low-volume service from
+becoming an incident. The baseline must be calculated using data before `t`
+only.
+
+## Baseline before neural network
+
+Start with two baselines:
+
+1. a trailing-rate/spike rule; and
+2. logistic regression using the same aggregate features.
+
+Then compare a small MLP (`32 -> 16 -> 1`, ReLU plus sigmoid) against them.
+Use a chronological train/validation/test split, and report precision, recall,
+PR-AUC, and false alerts per service. Accuracy is not meaningful for this
+imbalanced target.
+
+## Data readiness
+
+The current cluster ClickHouse instance has approximately 168k entries across
+18 services, but only about 13 hours of history at the time this document was
+created. It is sufficient to test normalization and aggregation, but not to
+train the seven-day-baseline detector. Let the normalized feature table collect
+at least two weeks of data before evaluating a model; four weeks is preferable.
+
+ClickHouse should remain a ClusterIP service while this work is developed. A
+LoadBalancer would expose the receiver's OTLP and debug surfaces and is not
+needed for model development; use a temporary `kubectl port-forward` or an
+authenticated internal job for inspection and training data export.