developerandClaude Sonnet 5
f77b89a779
- Fix /api/history/{type} to search the full persistent store instead of
only the bounded in-memory ring buffer, so filtered history isn't
silently truncated.
- Add structured type/since/until filters to /api/history/search and
/api/history/wait, alongside the existing free-text query.
- Add GET /api/traces/{traceId} to fetch all entries for a trace in one
call instead of relying on substring search.
- Rate-limit /api/history/wait per client IP.
- Add docs/PLAN.md tracking the phased improvement plan; update README
and agents_example.md for the new/changed endpoints.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
82 lines
3.8 KiB
Markdown
82 lines
3.8 KiB
Markdown
# AI Agent Debugging with OTEL Collector
|
|
|
|
This document provides instructions for AI agents on how to utilize the OTEL Collector's debugging features to search and fetch telemetry data (traces, logs, and metrics).
|
|
|
|
## Base Configuration
|
|
- **Base URL**: `https://otel.k6n.net`
|
|
- **Endpoints**:
|
|
- `GET /api/history/search?q={query}&type={type}&since={ns}&until={ns}&limit={limit}`: Search all historical telemetry data.
|
|
- **Default Limit**: 50 (max 100 recommended for context).
|
|
- **Ordering**: Latest entries first.
|
|
- **Match Scope**: Case-insensitive substring match on raw JSON (covers service name, span body, attributes, etc.), combinable with structured filters.
|
|
- **type**: optional, one of `traces`, `metrics`, `logs`.
|
|
- **since / until**: optional, unix nanosecond timestamps to bound the time range.
|
|
- `GET /api/history/wait?q={query}&type={type}&timeout={seconds}`: **Wait** for a specific signal to arrive.
|
|
- **Blocking**: This call blocks until a match is found or the timeout (default 30s) is reached.
|
|
- **Ideal for Automation**: Use this after triggering an action to wait for the exact result.
|
|
- **Rate limited**: capped per client IP; back off if you receive `429`.
|
|
- `GET /api/traces/{traceId}`: Fetch every stored entry (spans and correlated logs) referencing a specific trace id in one call. Prefer this over substring-searching for a trace id.
|
|
- `GET /api/history`: Fetch recent history of all telemetry (searches the full persistent store, not just recent in-memory entries).
|
|
- `GET /api/history/traces`: Fetch recent traces.
|
|
- `GET /api/history/logs`: Fetch recent logs.
|
|
- `GET /api/history/metrics`: Fetch recent metrics.
|
|
|
|
## Agent Instructions
|
|
|
|
### 1. Searching for Context
|
|
When debugging an issue, start by searching for relevant keywords in the logs and traces. This helps locate specific errors or trace IDs associated with a failure. Since results are **latest first**, you will always see the most recent events related to your search.
|
|
|
|
**Example Request:**
|
|
```bash
|
|
# Search for recent "ERROR" occurrences (default limit 50, latest first)
|
|
curl "https://otel.k6n.net/api/history/search?q=error"
|
|
```
|
|
|
|
### 2. Waiting for a Signal
|
|
Use the `wait` endpoint to synchronize your debugging flow. This is perfect for verifying that a specific event occurred after you triggered an action.
|
|
|
|
**Example Request:**
|
|
```bash
|
|
# Wait up to 60s for a specific traceId to show up in the collector
|
|
curl "https://otel.k6n.net/api/history/wait?q=5682138e4a9b4629852899d45e542456&timeout=60"
|
|
```
|
|
|
|
### 3. Identifying Trace Chains
|
|
If you find a log entry with a `traceId`, use the dedicated trace endpoint to fetch all spans and logs associated with that specific trace and understand the full request flow.
|
|
|
|
**Example Request:**
|
|
```bash
|
|
# Fetch all activity related to a specific trace
|
|
curl "https://otel.k6n.net/api/traces/5682138e4a9b4629852899d45e542456"
|
|
```
|
|
|
|
### 3. Fetching Recent Telemetry
|
|
To see the most recent activity without specific keywords, use the search endpoint with an empty query.
|
|
|
|
**Example Request:**
|
|
```bash
|
|
curl "https://otel.k6n.net/api/history/search?limit=20"
|
|
```
|
|
|
|
### 4. Data Format
|
|
The returned data is a JSON array of `HistoryEntry` objects:
|
|
```json
|
|
[
|
|
{
|
|
"type": "logs",
|
|
"timestamp": 1712132978000000000,
|
|
"data": { ... OTEL Log Records ... }
|
|
},
|
|
{
|
|
"type": "traces",
|
|
"timestamp": 1712132979000000000,
|
|
"data": { ... OTEL Resource Spans ... }
|
|
}
|
|
]
|
|
```
|
|
|
|
## Best Practices for AI Agents
|
|
- **Iterative Search**: Start with a broad search (e.g., service name) and narrow down using specific attributes or error messages found in the results.
|
|
- **Limit results**: Always use the `limit` parameter to manage context window size (default is 100).
|
|
- **Correlate Signals**: Look for timestamps that coincide across logs and metrics to identify resource constraints or timing issues.
|