Runbook: Data Ingestion (Phase 1)¶
Operating the aop.data pipeline: ingest sources, query the warehouse, search,
and stream.
Prerequisites¶
docker compose up -d # Qdrant, Postgres, Redis, Redpanda, Mongo
ollama serve # llama3.2:3b + nomic-embed-text
aop diagnose # all data-source rows should be OK
Ingest¶
| Source | Command |
|---|---|
| Text/Markdown | aop ingest notes.md -c docs |
aop ingest paper.pdf -c docs |
|
| HTML (file/URL) | aop ingest https://site/page -t html -c web |
| Transcript | aop ingest talk.srt -c media |
| CSV/JSON/Parquet | aop ingest data.csv -c rows --table data |
| SQL | aop ingest orders -t sql --query "SELECT * FROM orders" --table orders |
| MongoDB | aop ingest db.coll -t mongo --table coll |
| BigQuery | aop ingest proj.ds.tbl -t bigquery --table tbl |
| Snowflake | aop ingest DB.SCH.TBL -t snowflake --table tbl |
Chunking strategy: --strategy recursive|sentence_window|late|raptor.
Embedding backend: --embed-backend ollama|hf (use hf with late).
Retrieve / inspect¶
aop search "your question" -c docs -k 5
aop collections
aop warehouse tables
aop warehouse query "SELECT * FROM data LIMIT 10"
Streaming¶
# terminal A — start a consumer that indexes into collection 'live'
aop stream consume --collection live --idle-timeout 30
# terminal B — publish documents
aop stream produce "first streamed document" --topic aop.documents
Troubleshooting¶
| Symptom | Cause | Fix |
|---|---|---|
search returns nothing |
wrong/empty collection, or backend mismatch | check aop collections; pass the same --embed-backend used at ingest. |
| late chunking errors on import | embeddings extra missing |
pip install -e ".[embeddings]". |
SourceNotConfigured: bigquery/snowflake |
no credentials | set the AOP_* vars in .env (see .env.example). |
| Qdrant dimension error on upsert | mixing embedding backends/strategies in one collection | use one collection per (strategy, backend); dims must match. |
| record cap warning in logs | source exceeded 50k rows | expected (Phase 1 in-memory cap); narrow with --limit or wait for Phase 2 batched writes. |
mongo/redpanda unreachable |
stack not up | docker compose up -d; verify with aop diagnose. |