developerandClaude Sonnet 5
0d274ff89c
- Add internal/issues.Tracker: fingerprints (kind, service, title) into
deduplicated Issues with count/firstSeen/lastSeen/affected services and
bounded sample occurrences, evicting the least-recently-seen issue once
a cap is reached.
- Add internal/collector/issue_extract.go to pull issues out of ingested
OTLP data: spans with an error status (preferring exception.type/message
from an "exception" event, falling back to the span status message), and
logs at ERROR/FATAL severity (by severity number or text).
- Wire extraction into Collector.ConsumeTraces/ConsumeLogs and expose
GET /api/issues and GET /api/issues/{id}.
- Fix a pre-existing data race in the collector test's mockStore (unsynced
access to entries while addToHistory appends asynchronously), found via
`go test -race` while adding the new issue-tracking tests.
- Update docs/PLAN.md, README.md and agents_example.md for the new API.
The UI "Issues" tab itself lives in the separate frontend project that
builds this server's dist/ bundle; this commit only covers the backend.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
84 lines
4.2 KiB
Markdown
84 lines
4.2 KiB
Markdown
# AI Agent Debugging with OTEL Collector
|
|
|
|
This document provides instructions for AI agents on how to utilize the OTEL Collector's debugging features to search and fetch telemetry data (traces, logs, and metrics).
|
|
|
|
## Base Configuration
|
|
- **Base URL**: `https://otel.k6n.net`
|
|
- **Endpoints**:
|
|
- `GET /api/history/search?q={query}&type={type}&since={ns}&until={ns}&limit={limit}`: Search all historical telemetry data.
|
|
- **Default Limit**: 50 (max 100 recommended for context).
|
|
- **Ordering**: Latest entries first.
|
|
- **Match Scope**: Case-insensitive substring match on raw JSON (covers service name, span body, attributes, etc.), combinable with structured filters.
|
|
- **type**: optional, one of `traces`, `metrics`, `logs`.
|
|
- **since / until**: optional, unix nanosecond timestamps to bound the time range.
|
|
- `GET /api/history/wait?q={query}&type={type}&timeout={seconds}`: **Wait** for a specific signal to arrive.
|
|
- **Blocking**: This call blocks until a match is found or the timeout (default 30s) is reached.
|
|
- **Ideal for Automation**: Use this after triggering an action to wait for the exact result.
|
|
- **Rate limited**: capped per client IP; back off if you receive `429`.
|
|
- `GET /api/traces/{traceId}`: Fetch every stored entry (spans and correlated logs) referencing a specific trace id in one call. Prefer this over substring-searching for a trace id.
|
|
- `GET /api/issues?limit={limit}`: List deduplicated error issues (from error-status spans and error/fatal-severity logs), most recently seen first. Use this to triage what's currently broken before diving into raw search.
|
|
- `GET /api/issues/{id}`: Fetch a single issue's detail, including sample occurrences with trace ids you can feed into `/api/traces/{traceId}`.
|
|
- `GET /api/history`: Fetch recent history of all telemetry (searches the full persistent store, not just recent in-memory entries).
|
|
- `GET /api/history/traces`: Fetch recent traces.
|
|
- `GET /api/history/logs`: Fetch recent logs.
|
|
- `GET /api/history/metrics`: Fetch recent metrics.
|
|
|
|
## Agent Instructions
|
|
|
|
### 1. Searching for Context
|
|
When debugging an issue, start by searching for relevant keywords in the logs and traces. This helps locate specific errors or trace IDs associated with a failure. Since results are **latest first**, you will always see the most recent events related to your search.
|
|
|
|
**Example Request:**
|
|
```bash
|
|
# Search for recent "ERROR" occurrences (default limit 50, latest first)
|
|
curl "https://otel.k6n.net/api/history/search?q=error"
|
|
```
|
|
|
|
### 2. Waiting for a Signal
|
|
Use the `wait` endpoint to synchronize your debugging flow. This is perfect for verifying that a specific event occurred after you triggered an action.
|
|
|
|
**Example Request:**
|
|
```bash
|
|
# Wait up to 60s for a specific traceId to show up in the collector
|
|
curl "https://otel.k6n.net/api/history/wait?q=5682138e4a9b4629852899d45e542456&timeout=60"
|
|
```
|
|
|
|
### 3. Identifying Trace Chains
|
|
If you find a log entry with a `traceId`, use the dedicated trace endpoint to fetch all spans and logs associated with that specific trace and understand the full request flow.
|
|
|
|
**Example Request:**
|
|
```bash
|
|
# Fetch all activity related to a specific trace
|
|
curl "https://otel.k6n.net/api/traces/5682138e4a9b4629852899d45e542456"
|
|
```
|
|
|
|
### 3. Fetching Recent Telemetry
|
|
To see the most recent activity without specific keywords, use the search endpoint with an empty query.
|
|
|
|
**Example Request:**
|
|
```bash
|
|
curl "https://otel.k6n.net/api/history/search?limit=20"
|
|
```
|
|
|
|
### 4. Data Format
|
|
The returned data is a JSON array of `HistoryEntry` objects:
|
|
```json
|
|
[
|
|
{
|
|
"type": "logs",
|
|
"timestamp": 1712132978000000000,
|
|
"data": { ... OTEL Log Records ... }
|
|
},
|
|
{
|
|
"type": "traces",
|
|
"timestamp": 1712132979000000000,
|
|
"data": { ... OTEL Resource Spans ... }
|
|
}
|
|
]
|
|
```
|
|
|
|
## Best Practices for AI Agents
|
|
- **Iterative Search**: Start with a broad search (e.g., service name) and narrow down using specific attributes or error messages found in the results.
|
|
- **Limit results**: Always use the `limit` parameter to manage context window size (default is 100).
|
|
- **Correlate Signals**: Look for timestamps that coincide across logs and metrics to identify resource constraints or timing issues.
|