Multi-Provider LLM Support in One UI: How CachiBot Handles It

One of the hardest parts of building an AI agent is choosing a model. OpenAI’s GPT-4 is great at reasoning. Anthropic’s Claude excels at long context. Local models via Ollama keep your data private. But each provider has its own API, its own quirks, and its own pricing.

In CachiBot, I solved this with a unified provider abstraction. Here’s how it works.

The provider problem

If you hardcode your agent to OpenAI, switching to Anthropic means rewriting API calls, parsing different response shapes, and retesting everything. Multiply that by three or four providers and you have a maintenance nightmare.

A cleaner approach is to define a common interface that every provider implements:

class LLMProvider:
    def chat(self, messages: list, tools: list = None) -> LLMResponse:
        raise NotImplementedError

    def embed(self, texts: list) -> list:
        raise NotImplementedError

Then each provider — OpenAI, Anthropic, Ollama — implements that interface.

The CachiBot provider layer

CachiBot’s provider layer has three parts:

1. Provider classes — one per LLM service
2. A provider registry — maps names like "openai" or "anthropic" to classes
3. A unified response format — every provider returns the same structure

Example: OpenAI provider

from openai import OpenAI

class OpenAIProvider(LLMProvider):
    def __init__(self, api_key: str, model: str = "gpt-4o"):
        self.client = OpenAI(api_key=api_key)
        self.model = model

    def chat(self, messages, tools=None):
        response = self.client.chat.completions.create(
            model=self.model,
            messages=messages,
            tools=tools,
        )
        return LLMResponse(
            content=response.choices[0].message.content,
            tool_calls=self._extract_tool_calls(response),
            usage=response.usage,
        )

Example: Anthropic provider

import anthropic

class AnthropicProvider(LLMProvider):
    def __init__(self, api_key: str, model: str = "claude-3-5-sonnet"):
        self.client = anthropic.Anthropic(api_key=api_key)
        self.model = model

    def chat(self, messages, tools=None):
        response = self.client.messages.create(
            model=self.model,
            messages=messages,
            tools=tools,
            max_tokens=4096,
        )
        return LLMResponse(
            content=self._extract_text(response),
            tool_calls=self._extract_tool_calls(response),
            usage=self._extract_usage(response),
        )

The agent code never calls OpenAI or Anthropic directly. It calls provider.chat().

Tool calling across providers

Tool calling (function calling) is where providers differ most. OpenAI uses a specific JSON schema. Anthropic has its own format. Local models might not support tools at all.

CachiBot normalizes tool definitions:

tool = {
    "name": "read_file",
    "description": "Read the contents of a file",
    "parameters": {
        "type": "object",
        "properties": {
            "path": {"type": "string"}
        },
        "required": ["path"]
    }
}

Each provider converts this into its own native format. The agent receives a normalized ToolCall object regardless of which model generated it.

Switching providers at runtime

In the CachiBot dashboard, you can pick a provider from a dropdown:

Model: [OpenAI GPT-4o ▼]
       [Anthropic Claude 3.5 Sonnet]
       [Ollama llama3.1]

The agent engine instantiates the right provider class from the registry and passes it in. No code changes needed.

This is useful for:

• Cost optimization: Use cheaper models for simple tasks.
• Privacy: Route sensitive tasks to local models.
• Redundancy: Fall back to another provider if one is down.
• Benchmarking: Compare outputs side by side.

Local models with Ollama

For fully private deployments, CachiBot supports Ollama:

class OllamaProvider(LLMProvider):
    def __init__(self, base_url: str = "http://localhost:11434", model: str = "llama3.1"):
        self.base_url = base_url
        self.model = model

    def chat(self, messages, tools=None):
        # Convert to Ollama's chat format
        ...

Local models are slower and less capable for complex reasoning, but they’re perfect for tasks where data must stay on-premise.

Lessons learned

1. Don’t trust provider schemas to stay stable. Abstract early.
2. Normalize usage metrics. OpenAI returns tokens; Anthropic returns input/output tokens. CachiBot normalizes both into a common Usage object.
3. Handle streaming carefully. Each provider streams differently. CachiBot’s streaming layer unifies the event format.
4. Test with real prompts. Provider behavior varies more than the docs suggest.

Explore the code

The provider system is in cachibot/providers/: github.com/jhd3197/CachiBot

If you’re building with multiple LLMs, this pattern will save you time.

Related posts:

• CachiBot: A Sandboxed AI Agent Dashboard You Can Actually Trust