192 lines
7.4 KiB
Markdown
192 lines
7.4 KiB
Markdown
# Go OpenAPI Compatibility Layer
|
|
|
|
A Go library and CLI tool that provides compatibility with OpenAI API, enabling interactive sessions with local LLMs via llama.cpp.
|
|
|
|
## Overview
|
|
|
|
This project bridges the gap between OpenAI's API and local LLM inference engines (llama.cpp). It provides:
|
|
|
|
- **OpenAPI-compatible client** - Full support for ChatCompletion API
|
|
- **Streaming responses** - Real-time token streaming like OpenAI's API
|
|
- **Interactive CLI** - Session-based interaction with system prompts
|
|
- **Tool support** - Custom tools for reasoning (abstraction analysis, instruction updates)
|
|
- **Standard tools** - Built-in shell and file system tools
|
|
- **Extensible registry** - Easy addition of custom tools
|
|
|
|
## Features
|
|
|
|
### Core Functionality
|
|
|
|
- **Local LLM Integration**: Works with GGUF model files loaded via llama.cpp
|
|
- **System Prompts**: Configurable context for consistent responses
|
|
- **Streaming Output**: Chunked responses for real-time interaction
|
|
- **Reasoning Support**: Built-in reasoning content field for CoT models
|
|
|
|
### CLI Tools
|
|
|
|
| Tool | Description |
|
|
|------|-------------|
|
|
| `analyze_abstraction` | Analyzes code patterns (Decorator/Strategy) |
|
|
| `update_instructions` | Updates the system prompt for the session |
|
|
| `run_command` | Execute a shell command |
|
|
| `read_file` | Read the contents of a file |
|
|
| `write_file` | Write content to a file |
|
|
| `list_dir` | List contents of a directory |
|
|
| `replace_content` | Replace a block of text in a file |
|
|
|
|
### Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ CLI Interface │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ Session Manager │
|
|
│ - System prompts │
|
|
│ - Session state │
|
|
│ - Hook system (BeforeToolCall, OnReasoning, etc.) │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ Registry System │
|
|
│ - MapRegistry (custom tools) │
|
|
│ - ShellRegistry (command execution) │
|
|
│ - FileRegistry (filesystem operations) │
|
|
│ - CompositeRegistry (tool composition) │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ OpenAPI Client │
|
|
│ - ChatCompletion API implementation │
|
|
│ - Request/Response handling │
|
|
│ - Streaming support │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ llama.cpp (llama-cpp-go) │
|
|
│ - GGUF model loading │
|
|
│ - LLM inference │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### Configuration
|
|
|
|
Environment variables:
|
|
|
|
| Variable | Description | Default |
|
|
|----------|-------------|---------|
|
|
| `OPENAI_BASE_URL` | API endpoint | `http://10.10.11.33:8082/v1/` |
|
|
| `OPENAI_API_KEY` | API key for authentication | (empty) |
|
|
|
|
Model configuration (default):
|
|
```
|
|
Jackrong/Qwen3.5-4B-Claude-4.6-Opus-Reasoning-Distilled-GGUF
|
|
```
|
|
|
|
## Usage
|
|
|
|
### Quick Start
|
|
|
|
```bash
|
|
# Run the CLI (uses default settings)
|
|
./main.go
|
|
|
|
# Run with custom model
|
|
OPENAI_BASE_URL=http://localhost:8080/v1/ \
|
|
OPENAI_API_KEY=your-api-key \
|
|
./main.go
|
|
```
|
|
|
|
### CLI Interaction
|
|
|
|
```bash
|
|
$ ./main.go
|
|
Interactive Architect Session (BaseURL: http://10.10.11.33:8082/v1/)
|
|
Type your message and press Enter. Type 'exit' to quit.
|
|
|
|
User > What is the purpose of this project?
|
|
Assistant > This is an OpenAPI compatibility layer that enables...
|
|
|
|
User > Analyze my current package structure
|
|
Assistant > [Tool] Executing: analyze_abstraction({"name":"...", ...})...
|
|
[Tool] Returning results...
|
|
|
|
User > exit
|
|
```
|
|
|
|
### Custom Tool Registration
|
|
|
|
```go
|
|
registry := openapi.NewMapRegistry()
|
|
|
|
// Register a custom tool
|
|
registry.Register("my_custom_tool", "Describe the given function",
|
|
map[string]any{
|
|
"type": "object",
|
|
"properties": map[string]any{
|
|
"code": map[string]any{"type": "string"},
|
|
},
|
|
},
|
|
func(ctx context.Context, s *openapi.Session, args json.RawMessage) (any, error) {
|
|
var params struct{ Code string }
|
|
json.Unmarshal(args, ¶ms)
|
|
// Your tool logic here
|
|
return "Analysis complete.", nil
|
|
})
|
|
```
|
|
|
|
## Architecture Details
|
|
|
|
### Session Hooks
|
|
|
|
The session supports a powerful hook system for intercepting and modifying various events:
|
|
|
|
```go
|
|
session.SetHooks(openapi.SessionHooks{
|
|
BeforeToolCall: func(ctx context.Context, s *openapi.Session, tc *openapi.ToolCall) error {
|
|
// Called before each tool is invoked
|
|
return nil
|
|
},
|
|
OnReasoning: func(ctx context.Context, s *openapi.Session, reasoning string) error {
|
|
// Called for each reasoning step
|
|
return nil
|
|
},
|
|
OnDone: func(ctx context.Context, s *openapi.Session) error {
|
|
// Called when the session completes
|
|
return nil
|
|
},
|
|
OnError: func(ctx context.Context, s *openapi.Session, err error) {
|
|
// Called on errors
|
|
return nil
|
|
},
|
|
})
|
|
```
|
|
|
|
### Registry Pattern
|
|
|
|
The composite registry allows combining multiple tool sources:
|
|
|
|
```go
|
|
composite := &openapi.CompositeRegistry{}
|
|
|
|
// Base registry for custom tools
|
|
baseRegistry := openapi.NewMapRegistry()
|
|
composite.Add(baseRegistry)
|
|
|
|
// Add standard tools
|
|
composite.Add(openapi.NewShellRegistry())
|
|
composite.Add(openapi.NewFileRegistry("."))
|
|
|
|
// Apply to session
|
|
session.SetRegistry(composite)
|
|
```
|
|
|
|
## License
|
|
|
|
MIT
|