Go OpenAPI Compatibility Layer

A Go library and CLI tool that provides compatibility with OpenAI API, enabling interactive sessions with local LLMs via llama.cpp.

Overview

This project bridges the gap between OpenAI's API and local LLM inference engines (llama.cpp). It provides:

• OpenAPI-compatible client - Full support for ChatCompletion API
• Streaming responses - Real-time token streaming like OpenAI's API
• Interactive CLI - Session-based interaction with system prompts
• Tool support - Custom tools for reasoning (abstraction analysis, instruction updates)
• Standard tools - Built-in shell and file system tools
• Extensible registry - Easy addition of custom tools

Features

Core Functionality

• Local LLM Integration: Works with GGUF model files loaded via llama.cpp
• System Prompts: Configurable context for consistent responses
• Streaming Output: Chunked responses for real-time interaction
• Reasoning Support: Built-in reasoning content field for CoT models

CLI Tools

Tool	Description
analyze_abstraction	Analyzes code patterns (Decorator/Strategy)
update_instructions	Updates the system prompt for the session
run_command	Execute a shell command
read_file	Read the contents of a file
write_file	Write content to a file
list_dir	List contents of a directory
replace_content	Replace a block of text in a file

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    CLI Interface                             │
└─────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│                    Session Manager                           │
│  - System prompts                                           │
│  - Session state                                           │
│  - Hook system (BeforeToolCall, OnReasoning, etc.)         │
└─────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│                 Registry System                              │
│  - MapRegistry (custom tools)                              │
│  - ShellRegistry (command execution)                       │
│  - FileRegistry (filesystem operations)                    │
│  - CompositeRegistry (tool composition)                    │
└─────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│                  OpenAPI Client                             │
│  - ChatCompletion API implementation                       │
│  - Request/Response handling                               │
│  - Streaming support                                       │
└─────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│           llama.cpp (llama-cpp-go)                         │
│  - GGUF model loading                                     │
│  - LLM inference                                          │
└─────────────────────────────────────────────────────────────┘

Configuration

Environment variables:

Variable	Description	Default
OPENAI_BASE_URL	API endpoint	http://10.10.11.33:8082/v1/
OPENAI_API_KEY	API key for authentication	(empty)

Model configuration (default):

Jackrong/Qwen3.5-4B-Claude-4.6-Opus-Reasoning-Distilled-GGUF

Usage

Quick Start

# Run the CLI (uses default settings)
./main.go

# Run with custom model
OPENAI_BASE_URL=http://localhost:8080/v1/ \
OPENAI_API_KEY=your-api-key \
./main.go

CLI Interaction

$ ./main.go
Interactive Architect Session (BaseURL: http://10.10.11.33:8082/v1/)
Type your message and press Enter. Type 'exit' to quit.

User > What is the purpose of this project?
Assistant > This is an OpenAPI compatibility layer that enables...

User > Analyze my current package structure
Assistant > [Tool] Executing: analyze_abstraction({"name":"...", ...})...
[Tool] Returning results...

User > exit

Custom Tool Registration

registry := openapi.NewMapRegistry()

// Register a custom tool
registry.Register("my_custom_tool", "Describe the given function",
    map[string]any{
        "type": "object",
        "properties": map[string]any{
            "code": map[string]any{"type": "string"},
        },
    },
    func(ctx context.Context, s *openapi.Session, args json.RawMessage) (any, error) {
        var params struct{ Code string }
        json.Unmarshal(args, &params)
        // Your tool logic here
        return "Analysis complete.", nil
    })

Architecture Details

Session Hooks

The session supports a powerful hook system for intercepting and modifying various events:

session.SetHooks(openapi.SessionHooks{
    BeforeToolCall: func(ctx context.Context, s *openapi.Session, tc *openapi.ToolCall) error {
        // Called before each tool is invoked
        return nil
    },
    OnReasoning: func(ctx context.Context, s *openapi.Session, reasoning string) error {
        // Called for each reasoning step
        return nil
    },
    OnDone: func(ctx context.Context, s *openapi.Session) error {
        // Called when the session completes
        return nil
    },
    OnError: func(ctx context.Context, s *openapi.Session, err error) {
        // Called on errors
        return nil
    },
})

Registry Pattern

The composite registry allows combining multiple tool sources:

composite := &openapi.CompositeRegistry{}

// Base registry for custom tools
baseRegistry := openapi.NewMapRegistry()
composite.Add(baseRegistry)

// Add standard tools
composite.Add(openapi.NewShellRegistry())
composite.Add(openapi.NewFileRegistry("."))

// Apply to session
session.SetRegistry(composite)

License

MIT