more otel

agents_example.md

@@ -4,6 +4,8 @@ This document provides instructions for AI agents on how to utilize the OTEL Col
 
 ## Base Configuration
 - **Base URL**: `https://otel.k6n.net`
+- **Authentication**: set the service token in `OTEL_AGENT_TOKEN` and send it
+  as `X-Otel-Agent-Token: $OTEL_AGENT_TOKEN` on every protected debug request.
 - **Endpoints**:
     - `GET /api/history/search?q={query}&type={type}&since={ns}&until={ns}&limit={limit}`: Search all historical telemetry data.
         - **Default Limit**: 50 (max 100 recommended for context).
@@ -16,8 +18,11 @@ This document provides instructions for AI agents on how to utilize the OTEL Col
         - **Ideal for Automation**: Use this after triggering an action to wait for the exact result.
         - **Rate limited**: capped per client IP; back off if you receive `429`.
     - `GET /api/traces/{traceId}`: Fetch every stored entry (spans and correlated logs) referencing a specific trace id in one call. Prefer this over substring-searching for a trace id.
-    - `GET /api/issues?limit={limit}`: List deduplicated error issues (from error-status spans and error/fatal-severity logs), most recently seen first. Use this to triage what's currently broken before diving into raw search.
-    - `GET /api/issues/{id}`: Fetch a single issue's detail, including sample occurrences with trace ids you can feed into `/api/traces/{traceId}`.
+    - `GET /api/issues`: List deduplicated error issues (from error-status spans and error/fatal-severity logs), most recently seen first. Supported filters are `limit` (default 100, max 200), `cursor`, `service`, `kind`, `status`, `q` (title substring), `sourceIp`, `route`, `since`, and `until` (the time bounds are unix nanoseconds). Results remain a JSON array for compatibility; when another page exists, use the `X-Next-Cursor` response header or the `Link: ...; rel=next` header in the next request. The cursor is opaque and must be reused unchanged. Use this to triage what's currently broken before diving into raw search.
+    - `GET /api/issues/{id}`: Fetch a single issue's detail, including sample occurrences with trace ids you can feed into `/api/traces/{traceId}`. `PATCH` the same path with `{"status":"resolved"|"ignored"|"open"}` to update triage state (a resolved issue that fires again is auto-reopened).
+    - `GET /api/stats/errors?hours=72&service=`: Hourly (daily when `hours > 336`) total/error entry counts, ClickHouse-backed. Responds `501` when the server runs on file storage. Use for "when did errors spike?" and trend questions.
+    - `GET /api/stats/releases?hours=72&service=`: Per-release error-count series (`{"series":{"v1.3":[...],"v1.4":[...]}}`) to compare error rates before/after a deploy. Only entries carrying a `service.release` attribute appear.
+    - `GET /api/stats/services`: Distinct services present in the stats rollup.
     - `GET /api/history`: Fetch recent history of all telemetry (searches the full persistent store, not just recent in-memory entries).
     - `GET /api/history/traces`: Fetch recent traces.
     - `GET /api/history/logs`: Fetch recent logs.
@@ -31,7 +36,8 @@ When debugging an issue, start by searching for relevant keywords in the logs an
 **Example Request:**
 ```bash
 # Search for recent "ERROR" occurrences (default limit 50, latest first)
-curl "https://otel.k6n.net/api/history/search?q=error"
+curl "https://otel.k6n.net/api/history/search?q=error" \
+  -H "X-Otel-Agent-Token: $OTEL_AGENT_TOKEN"
 ```
 
 ### 2. Waiting for a Signal
@@ -40,7 +46,8 @@ Use the `wait` endpoint to synchronize your debugging flow. This is perfect for
 **Example Request:**
 ```bash
 # Wait up to 60s for a specific traceId to show up in the collector
-curl "https://otel.k6n.net/api/history/wait?q=5682138e4a9b4629852899d45e542456&timeout=60"
+curl "https://otel.k6n.net/api/history/wait?q=5682138e4a9b4629852899d45e542456&timeout=60" \
+  -H "X-Otel-Agent-Token: $OTEL_AGENT_TOKEN"
 ```
 
 ### 3. Identifying Trace Chains
@@ -49,7 +56,8 @@ If you find a log entry with a `traceId`, use the dedicated trace endpoint to fe
 **Example Request:**
 ```bash
 # Fetch all activity related to a specific trace
-curl "https://otel.k6n.net/api/traces/5682138e4a9b4629852899d45e542456"
+curl "https://otel.k6n.net/api/traces/5682138e4a9b4629852899d45e542456" \
+  -H "X-Otel-Agent-Token: $OTEL_AGENT_TOKEN"
 ```
 
 ### 3. Fetching Recent Telemetry
@@ -57,7 +65,8 @@ To see the most recent activity without specific keywords, use the search endpoi
 
 **Example Request:**
 ```bash
-curl "https://otel.k6n.net/api/history/search?limit=20"
+curl "https://otel.k6n.net/api/history/search?limit=20" \
+  -H "X-Otel-Agent-Token: $OTEL_AGENT_TOKEN"
 ```
 
 ### 4. Data Format