78 lines
3.3 KiB
Markdown
78 lines
3.3 KiB
Markdown
# AI Agent Debugging with OTEL Collector
|
|
|
|
This document provides instructions for AI agents on how to utilize the OTEL Collector's debugging features to search and fetch telemetry data (traces, logs, and metrics).
|
|
|
|
## Base Configuration
|
|
- **Base URL**: `https://otel.k6n.net`
|
|
- **Endpoints**:
|
|
- `GET /api/history/search?q={query}&limit={limit}`: Search all historical telemetry data.
|
|
- **Default Limit**: 50 (max 100 recommended for context).
|
|
- **Ordering**: Latest entries first.
|
|
- **Match Scope**: Case-insensitive substring match on raw JSON (covers service name, span body, attributes, etc.).
|
|
- `GET /api/history/wait?q={query}&timeout={seconds}`: **Wait** for a specific signal to arrive.
|
|
- **Blocking**: This call blocks until a match is found or the timeout (default 30s) is reached.
|
|
- **Ideal for Automation**: Use this after triggering an action to wait for the exact result.
|
|
- `GET /api/history`: Fetch the recent in-memory history of all telemetry.
|
|
- `GET /api/history/traces`: Fetch recent traces.
|
|
- `GET /api/history/logs`: Fetch recent logs.
|
|
- `GET /api/history/metrics`: Fetch recent metrics.
|
|
|
|
## Agent Instructions
|
|
|
|
### 1. Searching for Context
|
|
When debugging an issue, start by searching for relevant keywords in the logs and traces. This helps locate specific errors or trace IDs associated with a failure. Since results are **latest first**, you will always see the most recent events related to your search.
|
|
|
|
**Example Request:**
|
|
```bash
|
|
# Search for recent "ERROR" occurrences (default limit 50, latest first)
|
|
curl "https://otel.k6n.net/api/history/search?q=error"
|
|
```
|
|
|
|
### 2. Waiting for a Signal
|
|
Use the `wait` endpoint to synchronize your debugging flow. This is perfect for verifying that a specific event occurred after you triggered an action.
|
|
|
|
**Example Request:**
|
|
```bash
|
|
# Wait up to 60s for a specific traceId to show up in the collector
|
|
curl "https://otel.k6n.net/api/history/wait?q=5682138e4a9b4629852899d45e542456&timeout=60"
|
|
```
|
|
|
|
### 3. Identifying Trace Chains
|
|
If you find a log entry with a `traceId`, use the search endpoint to find all spans and logs associated with that specific trace to understand the full request flow.
|
|
|
|
**Example Request:**
|
|
```bash
|
|
# Search for all activity related to a specific trace
|
|
curl "https://otel.k6n.net/api/history/search?q=5682138e4a9b4629852899d45e542456&limit=100"
|
|
```
|
|
|
|
### 3. Fetching Recent Telemetry
|
|
To see the most recent activity without specific keywords, use the search endpoint with an empty query.
|
|
|
|
**Example Request:**
|
|
```bash
|
|
curl "https://otel.k6n.net/api/history/search?limit=20"
|
|
```
|
|
|
|
### 4. Data Format
|
|
The returned data is a JSON array of `HistoryEntry` objects:
|
|
```json
|
|
[
|
|
{
|
|
"type": "logs",
|
|
"timestamp": 1712132978000000000,
|
|
"data": { ... OTEL Log Records ... }
|
|
},
|
|
{
|
|
"type": "traces",
|
|
"timestamp": 1712132979000000000,
|
|
"data": { ... OTEL Resource Spans ... }
|
|
}
|
|
]
|
|
```
|
|
|
|
## Best Practices for AI Agents
|
|
- **Iterative Search**: Start with a broad search (e.g., service name) and narrow down using specific attributes or error messages found in the results.
|
|
- **Limit results**: Always use the `limit` parameter to manage context window size (default is 100).
|
|
- **Correlate Signals**: Look for timestamps that coincide across logs and metrics to identify resource constraints or timing issues.
|