Skip to content

ADR-0005: RAG retrieval, re-ranking, and memory layering

  • Status: Accepted
  • Date: 2026-06-04
  • Deciders: AOP core

Context

Phase 2 turns the Phase-1 indexes into a retrieval system and adds conversation memory. Requirements: hybrid dense+sparse retrieval with fusion, re-ranking (cross-encoder or Cohere with a local fallback), per-agent scoped memory, and conversation memory combining a sliding window, summary compression, and an episodic long-term store.

Decisions

1. Hybrid retrieval = dense ∥ sparse, fused with RRF

HybridRetriever queries Qdrant (dense) and the persistent BM25 index (sparse) concurrently and merges them with Reciprocal Rank Fusion (Σ w/(k+rank)). RRF needs no score normalisation across the two very different scales, is parameter-light, and is a strong default.

2. Re-ranking: Cohere when keyed, else a local cross-encoder

build_reranker() returns CohereReranker (Cohere /v2/rerank via httpx — no SDK dependency) when AOP_COHERE_API_KEY is set, otherwise CrossEncoderReranker (a local transformers cross-encoder, ms-marco-MiniLM-L-6-v2). A cross-encoder scores each (query, passage) pair jointly — far more precise than the bi-encoder similarity used for first-stage retrieval. The local fallback means real re-ranking works with zero cloud setup.

3. Per-agent isolation by collection namespacing

Agents share one backend; isolation is by physical collection name (<namespace>__<collection>, and mem__<namespace> for episodic memory). No cross-tenant leakage, no per-agent infrastructure. Searching a not-yet-created namespace returns empty rather than erroring.

4. Three-tier conversation memory

  • Durable log in Postgres (full turn history, survives restarts).
  • Working window — most recent turns within a token budget.
  • Summary compression — turns evicted from the window are folded into a running LLM summary (stored with a covered watermark so we only summarise newly-evicted turns, not the whole history each time).
  • Episodic recall — every turn is embedded into Qdrant and recalled by semantic similarity to the current query.

ConversationMemory.context(query) assembles summary + recalled + recent into a bounded message list for the next model call.

5. Grounded, cited generation

RAGPipeline.answer() retrieves → re-ranks → builds a numbered-context prompt → calls the LLM, returning the answer plus the exact source chunks ("citations"). Generation is auditable, not a black box.

Consequences

  • Positive: strong retrieval out of the box with no cloud keys; clean per-agent isolation; durable, compressible, recallable memory; grounded answers. Reuses Phase-1 components wholesale.
  • Cost: the local cross-encoder downloads ~80 MB on first use; episodic memory embeds every turn (one embed call per turn).
  • Caveats / Phase-3 hand-off:
  • Token budgeting for the window uses a char-based approximation (≈4 chars/ token) rather than a real tokenizer — adequate for budgeting, not exact.
  • Summary compression is linear in newly-evicted turns; very long sessions will accumulate a large summary (acceptable; revisit with hierarchical summaries if needed).
  • The orchestrator (Phase 3) will own multi-agent routing and tool use on top of this retrieval + memory substrate.