# AI Agent Debugging with OTEL Collector

This document provides instructions for AI agents on how to utilize the OTEL Collector's debugging features to search and fetch telemetry data (traces, logs, and metrics).

## Base Configuration
- **Base URL**: `https://otel.k6n.net`
- **Endpoints**:
    - `GET /api/history/search?q={query}&limit={limit}`: Search all historical telemetry data.
        - **Default Limit**: 50 (max 100 recommended for context).
        - **Ordering**: Latest entries first.
        - **Match Scope**: Case-insensitive substring match on raw JSON (covers service name, span body, attributes, etc.).
    - `GET /api/history/wait?q={query}&timeout={seconds}`: **Wait** for a specific signal to arrive.
        - **Blocking**: This call blocks until a match is found or the timeout (default 30s) is reached.
        - **Ideal for Automation**: Use this after triggering an action to wait for the exact result.
    - `GET /api/history`: Fetch the recent in-memory history of all telemetry.
    - `GET /api/history/traces`: Fetch recent traces.
    - `GET /api/history/logs`: Fetch recent logs.
    - `GET /api/history/metrics`: Fetch recent metrics.

## Agent Instructions

### 1. Searching for Context
When debugging an issue, start by searching for relevant keywords in the logs and traces. This helps locate specific errors or trace IDs associated with a failure. Since results are **latest first**, you will always see the most recent events related to your search.

**Example Request:**
```bash
# Search for recent "ERROR" occurrences (default limit 50, latest first)
curl "https://otel.k6n.net/api/history/search?q=error"
```

### 2. Waiting for a Signal
Use the `wait` endpoint to synchronize your debugging flow. This is perfect for verifying that a specific event occurred after you triggered an action.

**Example Request:**
```bash
# Wait up to 60s for a specific traceId to show up in the collector
curl "https://otel.k6n.net/api/history/wait?q=5682138e4a9b4629852899d45e542456&timeout=60"
```

### 3. Identifying Trace Chains
If you find a log entry with a `traceId`, use the search endpoint to find all spans and logs associated with that specific trace to understand the full request flow.

**Example Request:**
```bash
# Search for all activity related to a specific trace
curl "https://otel.k6n.net/api/history/search?q=5682138e4a9b4629852899d45e542456&limit=100"
```

### 3. Fetching Recent Telemetry
To see the most recent activity without specific keywords, use the search endpoint with an empty query.

**Example Request:**
```bash
curl "https://otel.k6n.net/api/history/search?limit=20"
```

### 4. Data Format
The returned data is a JSON array of `HistoryEntry` objects:
```json
[
  {
    "type": "logs",
    "timestamp": 1712132978000000000,
    "data": { ... OTEL Log Records ... }
  },
  {
    "type": "traces",
    "timestamp": 1712132979000000000,
    "data": { ... OTEL Resource Spans ... }
  }
]
```

## Best Practices for AI Agents
- **Iterative Search**: Start with a broad search (e.g., service name) and narrow down using specific attributes or error messages found in the results.
- **Limit results**: Always use the `limit` parameter to manage context window size (default is 100).
- **Correlate Signals**: Look for timestamps that coincide across logs and metrics to identify resource constraints or timing issues.