Commit Graph
1 Commits
Author SHA1 Message Date
developer 8d71ef06c6 laya: incident decision engine over alert + pod state
Build and Publish / Metadata (push) Successful in 9s
Build and Publish / BuildAndDeployAmd64 (push) Successful in 2m29s
Adds internal/laya, a client and safety-gated decision engine for the
Laya decision API (https://laya.k6n.net). The API's OpenAPI document
leaves every response schema empty, so the wire format was pinned by
probing the live service: questions are typed (choice/score/noul) and
carry "criteria" as an object for a choice and an array for a score.

An alert is rendered into a compact factual state and asked four typed
questions: which action (restart/ignore/notify/escalate), severity and
how urgent. The state always includes the pods' last events - restart
count, readiness, last termination reason and exit code, log tail, and
any overlapping rollout - because a 503 burst from a pod being replaced
is otherwise indistinguishable from an outage.

The model proposes, the policy disposes. Every gate records why it
changed an answer: an executable-action allowlist, a probability and
lead-over-runner-up threshold, no restart while pods are crash-looping
(the kubelet is already doing it) or while a rollout is replacing every
pod (the 503s are the rollout's own), never ignoring a new error or a
high-priority spike without a human's mute, an hourly restart budget
per deployment, and a plain notification whenever the model is
unreachable. The margin, not the raw probability, is the useful signal:
label probabilities are diffuse and a correct answer rarely exceeds 0.5.

topology.PodLog now carries the pod's lifecycle facts alongside the log
tail. PodsNotReady/CrashLooping helpers build on them.

Verified end to end against the live API: a crash loop is withheld as
"already crash-looping", a rollout's 503 burst is held back at a 2%
margin, and crawler 404s are ignored at 13%. LAYA_URL=... go test
./internal/laya -run TestLiveModel replays it.
2026-09-25 20:17:47 +00:00