go-otel/agents_example.md

# AI Agent Debugging with OTEL Collector

This document provides instructions for AI agents on how to utilize the OTEL Collector's debugging features to search and fetch telemetry data (traces, logs, and metrics). It is also served as-is at `{Base URL}/agents_example.md`, so an agent can fetch it directly before making any API calls.

## Base Configuration
- **Base URL**: `https://otel.k6n.net`
- **Authentication**: set the service token in `OTEL_AGENT_TOKEN` and send it
  as `X-Otel-Agent-Token: $OTEL_AGENT_TOKEN` on every protected debug request.
- **Endpoints**:
    - `GET /api/history/search?q={query}&type={type}&since={ns}&until={ns}&limit={limit}`: Search all historical telemetry data.
        - **Default Limit**: 50 (max 100 recommended for context).
        - **Ordering**: Latest entries first.
        - **Match Scope**: Case-insensitive substring match on raw JSON (covers service name, span body, attributes, etc.), combinable with structured filters.
        - **type**: optional, one of `traces`, `metrics`, `logs`.
        - **since / until**: optional, unix nanosecond timestamps to bound the time range.
    - `GET /api/history/wait?q={query}&type={type}&timeout={seconds}`: **Wait** for a specific signal to arrive.
        - **Blocking**: This call blocks until a match is found or the timeout (default 30s) is reached.
        - **Ideal for Automation**: Use this after triggering an action to wait for the exact result.
        - **Rate limited**: capped per client IP; back off if you receive `429`.
    - `GET /api/traces/{traceId}`: Fetch every stored entry (spans and correlated logs) referencing a specific trace id in one call. Prefer this over substring-searching for a trace id.
    - `GET /api/issues`: List deduplicated error issues (from error-status spans and error/fatal-severity logs), most recently seen first. Supported filters are `limit` (default 100, max 200), `cursor`, `service`, `kind`, `status`, `q` (title substring), `sourceIp`, `route`, `since`, and `until` (the time bounds are unix nanoseconds). Results remain a JSON array for compatibility; when another page exists, use the `X-Next-Cursor` response header or the `Link: ...; rel=next` header in the next request. The cursor is opaque and must be reused unchanged. Use this to triage what's currently broken before diving into raw search.
    - `GET /api/issues/{id}`: Fetch a single issue's detail, including sample occurrences with trace ids you can feed into `/api/traces/{traceId}`. `PATCH` the same path with `{"status":"resolved"|"ignored"|"open"}` to update triage state (a resolved issue that fires again is auto-reopened).
    - `GET /api/stats/errors?hours=72&service=`: Hourly (daily when `hours > 336`) total/error entry counts, ClickHouse-backed. Responds `501` when the server runs on file storage. Use for "when did errors spike?" and trend questions.
    - `GET /api/stats/releases?hours=72&service=`: Per-release error-count series (`{"series":{"v1.3":[...],"v1.4":[...]}}`) to compare error rates before/after a deploy. Only entries carrying a `service.release` attribute appear.
    - `GET /api/stats/services`: Distinct services present in the stats rollup.
    - `GET /api/history`: Fetch recent history of all telemetry (searches the full persistent store, not just recent in-memory entries).
    - `GET /api/history/traces`: Fetch recent traces.
    - `GET /api/history/logs`: Fetch recent logs.
    - `GET /api/history/metrics`: Fetch recent metrics.

## Agent Instructions

### 1. Searching for Context
When debugging an issue, start by searching for relevant keywords in the logs and traces. This helps locate specific errors or trace IDs associated with a failure. Since results are **latest first**, you will always see the most recent events related to your search.

**Example Request:**
```bash
# Search for recent "ERROR" occurrences (default limit 50, latest first)
curl "https://otel.k6n.net/api/history/search?q=error" \
  -H "X-Otel-Agent-Token: $OTEL_AGENT_TOKEN"
```

### 2. Waiting for a Signal
Use the `wait` endpoint to synchronize your debugging flow. This is perfect for verifying that a specific event occurred after you triggered an action.

**Example Request:**
```bash
# Wait up to 60s for a specific traceId to show up in the collector
curl "https://otel.k6n.net/api/history/wait?q=5682138e4a9b4629852899d45e542456&timeout=60" \
  -H "X-Otel-Agent-Token: $OTEL_AGENT_TOKEN"
```

### 3. Identifying Trace Chains
If you find a log entry with a `traceId`, use the dedicated trace endpoint to fetch all spans and logs associated with that specific trace and understand the full request flow.

**Example Request:**
```bash
# Fetch all activity related to a specific trace
curl "https://otel.k6n.net/api/traces/5682138e4a9b4629852899d45e542456" \
  -H "X-Otel-Agent-Token: $OTEL_AGENT_TOKEN"
```

### 3. Fetching Recent Telemetry
To see the most recent activity without specific keywords, use the search endpoint with an empty query.

**Example Request:**
```bash
curl "https://otel.k6n.net/api/history/search?limit=20" \
  -H "X-Otel-Agent-Token: $OTEL_AGENT_TOKEN"
```

### 4. Data Format
The returned data is a JSON array of `HistoryEntry` objects:
```json
[
  {
    "type": "logs",
    "timestamp": 1712132978000000000,
    "data": { ... OTEL Log Records ... }
  },
  {
    "type": "traces",
    "timestamp": 1712132979000000000,
    "data": { ... OTEL Resource Spans ... }
  }
]
```

## Best Practices for AI Agents
- **Iterative Search**: Start with a broad search (e.g., service name) and narrow down using specific attributes or error messages found in the results.
- **Limit results**: Always use the `limit` parameter to manage context window size (default is 100).
- **Correlate Signals**: Look for timestamps that coincide across logs and metrics to identify resource constraints or timing issues.