go-otel/docs/ML.md

Error detection model

Current decision

Keep the original OTLP payloads as the source of truth, but derive a stable feature row for each service/release/time window. The model should not consume raw OTLP JSON, and it does not need to classify every span or log message into a semantic taxonomy first.

The first target is an error-spike predictor:

Given the telemetry observed for a service in a five-minute window, predict whether the following fifteen-minute window contains an abnormal error rate.

This is more useful than classifying an event that the collector already knows is an error.

Normalized event contract

The extraction layer should reduce spans, logs, and metrics to these stable dimensions before aggregation:

Field	Meaning
window_start	UTC five-minute bucket
service	service.name, or unknown
release	service.release, or empty
environment	deployment environment, when present
kind	span, log, or metric
is_error	explicit error status, error HTTP/gRPC status, or error severity
status_code	HTTP status when present
severity_number	OTLP log severity, when present
duration_ms	span duration, when present
route	normalized HTTP route, when present
source	frontend or backend
trace_id	correlation identifier, when present
issue_fingerprint	existing issue fingerprint, when available

PII-bearing fields such as raw messages, URLs, user agents, source IPs, and stack traces must not be model features. They can remain in the raw history and issue views subject to the existing retention and access controls.

First aggregate feature row

For each service/release/window, derive:

• total event count and error count
• error rate
• p50/p95 span duration
• counts by kind (span, log, metric)
• counts by HTTP status class (2xx, 4xx, 5xx)
• unique issue count and newly observed issue count
• frontend and backend error counts
• retry, timeout, and cancellation counts where attributes identify them
• release age, if deployment time is available

The model input should be numeric and fixed-width. Categorical service and release values can initially be filtered per service; one global model can be considered after there is enough multi-service data.

Finding similar errors and spans

Similarity should be a separate concern from the primary issue fingerprint. Use deterministic grouping for the canonical issue ID because it is stable, explainable, cheap at ingest time, and safe for alert counts. The current issues.NormalizeTitle already removes common variable values such as UUIDs, hex identifiers, and request numbers.

The recommended matching stages are:

1. Canonical normalization: lowercase and normalize IDs, numbers, quoted values, URLs, and whitespace in messages and span names.
2. Structured signature: combine service, event kind, normalized route or span name, exception type, HTTP/gRPC status class, and the top application stack frames. This avoids merging unrelated errors that happen to contain the same words.
3. Exact fingerprint: use the structured signature for the canonical issue ID and deterministic deduplication.
4. Near-match suggestions: compare only within a bounded candidate group, such as the same service and event kind. Token n-gram similarity, SimHash, or weighted field similarity can suggest related issues without changing their IDs.

For example, these should usually be suggested as similar but remain separate until confirmed:

checkout: database timeout after 2000ms
checkout: database timeout after 5000ms

They may represent the same underlying failure, but the timeout value could be operationally important. A review UI can offer “merge” or “not related”; those decisions become valuable labels later.

Embeddings or a neural similarity model are reasonable as a second phase for semantic discovery, especially when messages use different wording. They should not be the first grouping mechanism: they add model/version drift, latency, explainability, and possible sensitive-data leakage. If introduced, embed redacted normalized text and stack signatures, retrieve nearest neighbors, and present the result as a suggestion rather than silently merging issues.

For spans, the strongest similarity fields are usually the normalized span name or route, service, status class, exception type, and stack-frame signature. Trace IDs, span IDs, timestamps, and request-specific values should never contribute to similarity.

Label

For a row at time t, label it positive when the future interval [t+5m, t+20m) satisfies both conditions:

1. error rate is at least three times the trailing seven-day baseline for that service and time-of-day; and
2. the future error count is at least 5.

The count floor prevents a single error in a very low-volume service from becoming an incident. The baseline must be calculated using data before t only.

Baseline before neural network

Start with two baselines:

1. a trailing-rate/spike rule; and
2. logistic regression using the same aggregate features.

Then compare a small MLP (32 -> 16 -> 1, ReLU plus sigmoid) against them. Use a chronological train/validation/test split, and report precision, recall, PR-AUC, and false alerts per service. Accuracy is not meaningful for this imbalanced target.

Data readiness

The current cluster ClickHouse instance has approximately 168k entries across 18 services, but only about 13 hours of history at the time this document was created. It is sufficient to test normalization and aggregation, but not to train the seven-day-baseline detector. Let the normalized feature table collect at least two weeks of data before evaluating a model; four weeks is preferable.

ClickHouse should remain a ClusterIP service while this work is developed. A LoadBalancer would expose the receiver's OTLP and debug surfaces and is not needed for model development; use a temporary kubectl port-forward or an authenticated internal job for inspection and training data export.