Skip to content

ADR-0002: Native unified LLM adapter over LangChain/LlamaIndex

  • Status: Accepted
  • Date: 2026-06-04
  • Deciders: AOP core

Context

The platform must speak to local LLMs (Ollama, and later LM Studio / llama.cpp) and cloud providers (OpenAI, Anthropic, Gemini, Mistral) through a single interface, so that switching a model is a configuration change, not a code change. The original brief suggested a "LangChain / LlamaIndex abstraction layer."

Constraints that shape the decision:

  1. Zero-mock, production-ready. Every call path must be real and debuggable.
  2. Async-first. The orchestrator (Phase 3) is heavily concurrent; we need first-class asyncio and streaming with precise token accounting.
  3. Thin dependency surface. Fewer transitive dependencies means fewer security/maintenance liabilities and faster, more reliable installs — especially on bleeding-edge interpreters (the dev box runs Python 3.14, where parts of the LangChain dependency tree lag on wheels).
  4. Control over the wire. MCP tool-calling (Phase 3), per-agent memory (Phase 2), and observability hooks need direct access to request/response shapes, not a lowest-common-denominator abstraction.

Decision

Build a native, thin adapter layer (aop.llm) directly over each provider's HTTP API using a single shared httpx.AsyncClient per adapter:

  • One ABC, LLMAdapter, with chat, stream_chat, embed, list_models, health, plus lifecycle (aclose, async context manager).
  • Provider-agnostic Pydantic contracts (Message, ChatResponse, ChatChunk, EmbeddingResponse, ModelInfo, HealthStatus, Usage).
  • A registry (@register) + factory so adding a provider is one decorated class.
  • A typed error hierarchy (ProviderNotConfigured, ProviderHTTPError, FeatureNotSupported, UnknownProvider) — failures are loud and actionable, never silently mocked.

We deliberately do not wrap LangChain or LlamaIndex at the core layer.

Consequences

  • Positive: minimal, auditable dependencies (httpx, pydantic); installs cleanly on Python 3.14; full control of streaming, usage, and headers.
  • Positive: a stable internal contract decoupled from any third-party framework's churn.
  • Negative: we re-implement provider quirks (Anthropic's system field and required max_tokens; Gemini's contents/parts; OpenAI-compatible SSE). Mitigated by per-provider adapters with focused tests.
  • Negative: we forgo LangChain's prebuilt integrations (loaders, chains). Where those are genuinely useful (e.g. document loaders in Phase 1, eval helpers in Phase 5), we may adopt them at the edges as optional dependencies — but they will be wrapped behind our own interfaces, never leaked into the core contract.

Alternatives considered

  • LangChain ChatModel abstraction: rich ecosystem, but heavy transitive deps, frequent breaking changes, and an abstraction that obscures token usage and streaming details we need. Rejected for the core; reconsidered at edges.
  • LiteLLM: good provider coverage via an OpenAI-shaped facade, but it flattens provider-native features and adds a proxy-shaped dependency. Our OpenAI-compatible base already covers OpenAI + Mistral with ~30 lines each.