146 lines
6.0 KiB
Markdown
146 lines
6.0 KiB
Markdown
# Error detection model
|
|
|
|
## Current decision
|
|
|
|
Keep the original OTLP payloads as the source of truth, but derive a stable
|
|
feature row for each service/release/time window. The model should not consume
|
|
raw OTLP JSON, and it does not need to classify every span or log message into
|
|
a semantic taxonomy first.
|
|
|
|
The first target is an **error-spike predictor**:
|
|
|
|
> Given the telemetry observed for a service in a five-minute window, predict
|
|
> whether the following fifteen-minute window contains an abnormal error rate.
|
|
|
|
This is more useful than classifying an event that the collector already knows
|
|
is an error.
|
|
|
|
## Normalized event contract
|
|
|
|
The extraction layer should reduce spans, logs, and metrics to these stable
|
|
dimensions before aggregation:
|
|
|
|
| Field | Meaning |
|
|
| --- | --- |
|
|
| `window_start` | UTC five-minute bucket |
|
|
| `service` | `service.name`, or `unknown` |
|
|
| `release` | `service.release`, or empty |
|
|
| `environment` | deployment environment, when present |
|
|
| `kind` | `span`, `log`, or `metric` |
|
|
| `is_error` | explicit error status, error HTTP/gRPC status, or error severity |
|
|
| `status_code` | HTTP status when present |
|
|
| `severity_number` | OTLP log severity, when present |
|
|
| `duration_ms` | span duration, when present |
|
|
| `route` | normalized HTTP route, when present |
|
|
| `source` | `frontend` or `backend` |
|
|
| `trace_id` | correlation identifier, when present |
|
|
| `issue_fingerprint` | existing issue fingerprint, when available |
|
|
|
|
PII-bearing fields such as raw messages, URLs, user agents, source IPs, and
|
|
stack traces must not be model features. They can remain in the raw history and
|
|
issue views subject to the existing retention and access controls.
|
|
|
|
## First aggregate feature row
|
|
|
|
For each service/release/window, derive:
|
|
|
|
- total event count and error count
|
|
- error rate
|
|
- p50/p95 span duration
|
|
- counts by kind (`span`, `log`, `metric`)
|
|
- counts by HTTP status class (`2xx`, `4xx`, `5xx`)
|
|
- unique issue count and newly observed issue count
|
|
- frontend and backend error counts
|
|
- retry, timeout, and cancellation counts where attributes identify them
|
|
- release age, if deployment time is available
|
|
|
|
The model input should be numeric and fixed-width. Categorical service and
|
|
release values can initially be filtered per service; one global model can be
|
|
considered after there is enough multi-service data.
|
|
|
|
## Finding similar errors and spans
|
|
|
|
Similarity should be a separate concern from the primary issue fingerprint.
|
|
Use deterministic grouping for the canonical issue ID because it is stable,
|
|
explainable, cheap at ingest time, and safe for alert counts. The current
|
|
`issues.NormalizeTitle` already removes common variable values such as UUIDs,
|
|
hex identifiers, and request numbers.
|
|
|
|
The recommended matching stages are:
|
|
|
|
1. **Canonical normalization**: lowercase and normalize IDs, numbers, quoted
|
|
values, URLs, and whitespace in messages and span names.
|
|
2. **Structured signature**: combine service, event kind, normalized route or
|
|
span name, exception type, HTTP/gRPC status class, and the top application
|
|
stack frames. This avoids merging unrelated errors that happen to contain
|
|
the same words.
|
|
3. **Exact fingerprint**: use the structured signature for the canonical issue
|
|
ID and deterministic deduplication.
|
|
4. **Near-match suggestions**: compare only within a bounded candidate group,
|
|
such as the same service and event kind. Token n-gram similarity, SimHash,
|
|
or weighted field similarity can suggest related issues without changing
|
|
their IDs.
|
|
|
|
For example, these should usually be suggested as similar but remain separate
|
|
until confirmed:
|
|
|
|
```text
|
|
checkout: database timeout after 2000ms
|
|
checkout: database timeout after 5000ms
|
|
```
|
|
|
|
They may represent the same underlying failure, but the timeout value could be
|
|
operationally important. A review UI can offer “merge” or “not related”; those
|
|
decisions become valuable labels later.
|
|
|
|
Embeddings or a neural similarity model are reasonable as a second phase for
|
|
semantic discovery, especially when messages use different wording. They
|
|
should not be the first grouping mechanism: they add model/version drift,
|
|
latency, explainability, and possible sensitive-data leakage. If introduced,
|
|
embed redacted normalized text and stack signatures, retrieve nearest
|
|
neighbors, and present the result as a suggestion rather than silently
|
|
merging issues.
|
|
|
|
For spans, the strongest similarity fields are usually the normalized span
|
|
name or route, service, status class, exception type, and stack-frame signature.
|
|
Trace IDs, span IDs, timestamps, and request-specific values should never
|
|
contribute to similarity.
|
|
|
|
## Label
|
|
|
|
For a row at time `t`, label it positive when the future interval `[t+5m,
|
|
t+20m)` satisfies both conditions:
|
|
|
|
1. error rate is at least three times the trailing seven-day baseline for that
|
|
service and time-of-day; and
|
|
2. the future error count is at least 5.
|
|
|
|
The count floor prevents a single error in a very low-volume service from
|
|
becoming an incident. The baseline must be calculated using data before `t`
|
|
only.
|
|
|
|
## Baseline before neural network
|
|
|
|
Start with two baselines:
|
|
|
|
1. a trailing-rate/spike rule; and
|
|
2. logistic regression using the same aggregate features.
|
|
|
|
Then compare a small MLP (`32 -> 16 -> 1`, ReLU plus sigmoid) against them.
|
|
Use a chronological train/validation/test split, and report precision, recall,
|
|
PR-AUC, and false alerts per service. Accuracy is not meaningful for this
|
|
imbalanced target.
|
|
|
|
## Data readiness
|
|
|
|
The current cluster ClickHouse instance has approximately 168k entries across
|
|
18 services, but only about 13 hours of history at the time this document was
|
|
created. It is sufficient to test normalization and aggregation, but not to
|
|
train the seven-day-baseline detector. Let the normalized feature table collect
|
|
at least two weeks of data before evaluating a model; four weeks is preferable.
|
|
|
|
ClickHouse should remain a ClusterIP service while this work is developed. A
|
|
LoadBalancer would expose the receiver's OTLP and debug surfaces and is not
|
|
needed for model development; use a temporary `kubectl port-forward` or an
|
|
authenticated internal job for inspection and training data export.
|