# Error detection model ## Current decision Keep the original OTLP payloads as the source of truth, but derive a stable feature row for each service/release/time window. The model should not consume raw OTLP JSON, and it does not need to classify every span or log message into a semantic taxonomy first. The first target is an **error-spike predictor**: > Given the telemetry observed for a service in a five-minute window, predict > whether the following fifteen-minute window contains an abnormal error rate. This is more useful than classifying an event that the collector already knows is an error. ## Normalized event contract The extraction layer should reduce spans, logs, and metrics to these stable dimensions before aggregation: | Field | Meaning | | --- | --- | | `window_start` | UTC five-minute bucket | | `service` | `service.name`, or `unknown` | | `release` | `service.release`, or empty | | `environment` | deployment environment, when present | | `kind` | `span`, `log`, or `metric` | | `is_error` | explicit error status, error HTTP/gRPC status, or error severity | | `status_code` | HTTP status when present | | `severity_number` | OTLP log severity, when present | | `duration_ms` | span duration, when present | | `route` | normalized HTTP route, when present | | `source` | `frontend` or `backend` | | `trace_id` | correlation identifier, when present | | `issue_fingerprint` | existing issue fingerprint, when available | PII-bearing fields such as raw messages, URLs, user agents, source IPs, and stack traces must not be model features. They can remain in the raw history and issue views subject to the existing retention and access controls. ## First aggregate feature row For each service/release/window, derive: - total event count and error count - error rate - p50/p95 span duration - counts by kind (`span`, `log`, `metric`) - counts by HTTP status class (`2xx`, `4xx`, `5xx`) - unique issue count and newly observed issue count - frontend and backend error counts - retry, timeout, and cancellation counts where attributes identify them - release age, if deployment time is available The model input should be numeric and fixed-width. Categorical service and release values can initially be filtered per service; one global model can be considered after there is enough multi-service data. ## Finding similar errors and spans Similarity should be a separate concern from the primary issue fingerprint. Use deterministic grouping for the canonical issue ID because it is stable, explainable, cheap at ingest time, and safe for alert counts. The current `issues.NormalizeTitle` already removes common variable values such as UUIDs, hex identifiers, and request numbers. The recommended matching stages are: 1. **Canonical normalization**: lowercase and normalize IDs, numbers, quoted values, URLs, and whitespace in messages and span names. 2. **Structured signature**: combine service, event kind, normalized route or span name, exception type, HTTP/gRPC status class, and the top application stack frames. This avoids merging unrelated errors that happen to contain the same words. 3. **Exact fingerprint**: use the structured signature for the canonical issue ID and deterministic deduplication. 4. **Near-match suggestions**: compare only within a bounded candidate group, such as the same service and event kind. Token n-gram similarity, SimHash, or weighted field similarity can suggest related issues without changing their IDs. For example, these should usually be suggested as similar but remain separate until confirmed: ```text checkout: database timeout after 2000ms checkout: database timeout after 5000ms ``` They may represent the same underlying failure, but the timeout value could be operationally important. A review UI can offer “merge” or “not related”; those decisions become valuable labels later. Embeddings or a neural similarity model are reasonable as a second phase for semantic discovery, especially when messages use different wording. They should not be the first grouping mechanism: they add model/version drift, latency, explainability, and possible sensitive-data leakage. If introduced, embed redacted normalized text and stack signatures, retrieve nearest neighbors, and present the result as a suggestion rather than silently merging issues. For spans, the strongest similarity fields are usually the normalized span name or route, service, status class, exception type, and stack-frame signature. Trace IDs, span IDs, timestamps, and request-specific values should never contribute to similarity. ## Label For a row at time `t`, label it positive when the future interval `[t+5m, t+20m)` satisfies both conditions: 1. error rate is at least three times the trailing seven-day baseline for that service and time-of-day; and 2. the future error count is at least 5. The count floor prevents a single error in a very low-volume service from becoming an incident. The baseline must be calculated using data before `t` only. ## Baseline before neural network Start with two baselines: 1. a trailing-rate/spike rule; and 2. logistic regression using the same aggregate features. Then compare a small MLP (`32 -> 16 -> 1`, ReLU plus sigmoid) against them. Use a chronological train/validation/test split, and report precision, recall, PR-AUC, and false alerts per service. Accuracy is not meaningful for this imbalanced target. ## Data readiness The current cluster ClickHouse instance has approximately 168k entries across 18 services, but only about 13 hours of history at the time this document was created. It is sufficient to test normalization and aggregation, but not to train the seven-day-baseline detector. Let the normalized feature table collect at least two weeks of data before evaluating a model; four weeks is preferable. ClickHouse should remain a ClusterIP service while this work is developed. A LoadBalancer would expose the receiver's OTLP and debug surfaces and is not needed for model development; use a temporary `kubectl port-forward` or an authenticated internal job for inspection and training data export.