Skip to content

ADR-0004: Data-infrastructure engine choices

  • Status: Accepted
  • Date: 2026-06-04
  • Deciders: AOP core

Context

Phase 1 ingests every source type in the brief (CSV, JSON, Parquet, SQL, NoSQL, cloud warehouses, message brokers, and unstructured PDF/HTML/transcripts), chunks documents, embeds them, and indexes them for retrieval — all local-first, no mocks, on a Python 3.14 interpreter.

Decisions

1. DuckDB is the file engine and the local warehouse

CSV/JSON/Parquet are read through DuckDB's native read_csv_auto / read_json_auto / read_parquet. This avoids a direct pyarrow dependency for file parsing, unifies "tabular ingestion" and "analytical warehouse" on one engine, and gets globs/compression for free. DuckDB is embedded (no service).

2. Qdrant for vectors, rank_bm25 for keywords

Qdrant (already in the compose stack) is the dense store, one collection per namespace for per-agent isolation. A file-persisted BM25 index is built alongside so Phase 2 can fuse dense + sparse with RRF. Phase 1 search is dense-only; the BM25 index is populated but not yet fused.

3. Two embedding backends behind one interface

OllamaEmbedder (pooled vectors, the default) and HFEmbedder (local token-level hidden states). The HF backend exists specifically to enable true late chunking — the Ollama embeddings API returns one pooled vector per input and cannot expose token-level vectors.

4. RAPTOR with numpy KMeans (not UMAP+GMM)

RAPTOR is implemented as recursive KMeans clustering over normalised embeddings plus local-LLM summarisation per cluster, building a multi-level tree whose nodes (leaves + summaries) are all indexed. We use a small deterministic numpy KMeans rather than the paper's UMAP+GMM stack to keep dependencies minimal and results reproducible.

5. Cloud connectors are real but credential-gated

BigQuery and Snowflake connectors open live connections (no simulation). Without credentials they raise SourceNotConfigured with the exact settings required; their integration tests skip (like the live Ollama tests) rather than mock. Their client libraries are optional extras (.[bigquery], .[snowflake]).

Consequences

  • Positive: minimal, 3.14-verified dependency set; one engine for files + warehouse; honest late chunking; reproducible RAPTOR.

Hardening (resolved 2026-06-04)

The Phase-1 caveats below were addressed before moving on:

  • Late chunking — no truncation. Documents longer than the model context are now processed with overlapping sliding windows (token vectors averaged across windows), not truncated. The context length is configurable (AOP_HF_MAX_TOKENS) and capped at the model's own max; long-context models (e.g. nomic-embed-text-v1.5, 8192 ctx, via AOP_HF_EMBEDDING_MODEL + AOP_HF_TRUST_REMOTE_CODE) are supported.
  • BM25 — no rebuild per search. The built BM25Okapi model is pickled on add and loaded once per search, cached in-process and invalidated by file mtime. Building happens only on write.
  • Record path — no in-memory cap. Records stream to the warehouse and index in batches of AOP_INGEST_BATCH_SIZE (default 1000), with schema-evolving appends (new columns are added via ALTER TABLE) and incremental quality accumulation. Memory is bounded regardless of dataset size.

Remaining for Phase 2

  • Fused dense+sparse retrieval (RRF) and re-ranking build on the dense store and the now-persistent BM25 index.