Skip to content

ADR-0008: Fine-tuning control panel

  • Status: Accepted
  • Date: 2026-06-05
  • Deciders: AOP core

Context

Phase 5 adds per-agent fine-tuning: dataset building, training (LoRA/QLoRA/DPO/ RLHF), evaluation, and a versioned model registry — on a CPU-only host (torch 2.12.0+cpu, no CUDA).

Decisions

1. Honest hardware scoping

  • LoRA SFT and DPO run for real on CPU with a tiny base model (sshleifer/tiny-gpt2) + small dataset + few steps — enough to prove the full pipeline end-to-end (no mocks).
  • QLoRA (needs bitsandbytes/CUDA) and PPO-RLHF (needs a GPU) are implemented as launchers but fail loudly with HardwareUnavailable (a clear "requires a CUDA GPU" message) on a CPU-only host — never faked.

2. Training runs in an isolated subprocess

JobManager.submit() spawns python -m aop.finetune.worker with PYTHONUTF8=1. This (a) keeps heavy torch training off the API event loop, (b) makes runs killable/isolated, and (c) works around a TRL-on-Windows bug where bundled .jinja chat templates are read with the OS locale encoding (cp1252) instead of UTF-8 — UTF-8 mode fixes it. The worker publishes progress to the Phase-4 Redis event bus (run:ft-<job_id>), so CLI/API/UI stream it like agent runs.

3. TRL 1.x trainer API

SFTTrainer/DPOTrainer with SFTConfig/DPOConfig; use_cpu=True is set when no CUDA is present (transformers requires it for CPU training). LoRA target modules are inferred (c_attn for GPT-2-family).

4. Evaluation: real metrics + local LLM judge

ROUGE (rouge-score) and BERTScore (bert-score) are real metric libraries. RAGAS-style faithfulness + answer-relevancy are computed with the local Ollama LLM as judge — no heavy ragas dependency, and it reuses the platform's own model.

5. Versioned registry + A/B router

Checkpoints are versioned under models/registry/<name>/v<n>/ with meta.json (metrics + config). The registry lists/queries/diffs versions; ABRouter does weighted version selection for A/B inference routing.

6. Control panel UI

A /finetune Next.js route: launch form, live job progress over WS /ws/finetune/{id}, and the registry/jobs tables. Reuses the Phase-4 frontend.

Consequences

  • Positive: the entire fine-tuning lifecycle (dataset → train → register → evaluate) runs and is demonstrable locally; cloud-scale methods are present and GPU-gated honestly; training is isolated and streamable.
  • Caveats:
  • Only tiny models train in reasonable time on CPU; real models need a GPU (the launcher supports them — run off-box).
  • A failed run leaves an empty version directory; the registry ignores directories without meta.json, so listings stay clean.
  • The job registry is filesystem-based and single-host; multi-host scheduling is future work.