Engineering case study · RAG · 2026

Engineering a RAG system that can prove it improved

I built a Haystack-based hybrid retrieval layer for my personal knowledge system, connected it to AI coding tools through MCP, and treated retrieval quality as an engineering problem: establish a baseline, isolate changes, protect against overfitting and verify the result on unseen questions.

30/30

Hybrid Recall@5

0.950

Hybrid MRR

~210 ms

Median latency, top 5

92.63%

Backend test coverage

The system

Retrieval as infrastructure, not a prompt trick

The knowledge base contains technical notes, project decisions, earlier failures and reusable solutions. The retrieval layer scans Markdown deterministically, creates stable incremental chunks and stores both dense vectors and keyword-searchable text in PostgreSQL. A local manifest, locks and freshness guards make indexing observable and safe to operate.

Query-time retrieval combines exact technical language with semantic similarity. Weighted reciprocal-rank fusion merges the branches; source-aware scoring and path diversity keep a single long document from crowding out alternatives. MCP tools expose the resulting evidence to coding agents before they plan or modify a system.

Versioned knowledge

Markdown notes, project decisions, errors and reusable solutions

Deterministic indexing

Parsing, stable IDs, incremental chunks, manifest and freshness checks

Dual retrieval

Keyword/GIN and dense vector search in Supabase PostgreSQL with pgvector

Hybrid ranking

Haystack-based orchestration, weighted RRF, source factors and path diversity

Evidence for agents

MCP tools return compact passages, paths and headings before planning or coding

The language model never searches the raw vault by intuition. A deterministic retrieval service selects traceable evidence first; the agent receives source paths, headings and passages through a compact MCP contract.

First measurement

A relative win exposed an absolute quality problem

The initial A/B test compared a repository-level lexical search with the RAG contract on 15 real questions. Lexical search found the expected source in the top five for only 1/15 cases; the RAG system reached 7/15, with MRR 0.363 and median latency of 1.246 seconds. The relative gain was large, but 46.7% Recall@5 was not good enough for a system meant to guide engineering decisions.

Before optimizing, I turned those cases into development data, made result artifacts versionable and reran the current code. The reproduced baseline was 6/15 with MRR 0.350. I kept the historical 7/15 result in the record instead of hiding the discrepancy: the reproduced run, not the more flattering old number, became the baseline for every following ablation.

Second iteration

Five controlled steps from 6/15 to 15/15

I used Recall@5 as the primary acceptance metric and MRR to reveal whether the right source moved toward rank one. Each row below is an ablation: cumulative, measured against the same development set and retained only when the evidence justified it.

Experiment 1

6/15

MRR 0.350

Reproduced baseline

The existing task-context contract was rerun against the 15-case development set. This became the controlled starting point for optimization.

Experiment 2

7/15

MRR 0.367

Source diversity

Limiting task context to one chunk per source increased mean unique paths from 3.67 to 5.00 and reduced redundant evidence.

Experiment 3

11/15

MRR 0.489

Token-aware ranking

Exact identifiers were separated from ordinary words, while title and heading matches received explicit weights.

Experiment 4

13/15

MRR 0.711

Source-aware scoring

Curated knowledge sources were promoted and recurring low-value noise sources were demoted with transparent factors.

Experiment 5

15/15

MRR 0.819

Focused hybrid fusion

A focused keyword branch, controlled query expansion and weighted reciprocal-rank fusion completed the development-set improvement.

Why these changes worked

Diversity: retrieval quality is about useful sources, not repeated chunks from the same source. The task contract now selects one chunk per path.

Precision: identifiers, filenames and technical terms need exact matching; natural-language intent still benefits from dense embeddings.

Fusion: neither branch is trusted universally. Weighted RRF combines rankings without pretending that incomparable raw scores share one scale.

Independent validation

Freeze the system, then ask 30 new questions

A perfect development score is not proof of generalization. I therefore created 30 additional questions with 30 disjoint gold documents, froze the set before the final run and made no parameter changes after seeing the results. The questions covered retrieval knowledge, projects, operations and recurring workflows.

Hybrid and dense retrieval both reached 30/30 Recall@5; keyword retrieval reached 28/30. Hybrid MRR was 0.950, meaning the expected source usually appeared at rank one. All seven historically successful development cases remained successful, so the gain did not trade away known behavior.

Index scale: 1,055 local files · 1,056 remote sources · 13,405 chunks
Retrieval modes: Keyword 28/30 · Dense 30/30 · Hybrid 30/30 on holdout
Latency: 209.5 ms median · 1,265.6 ms p95 for top-5 context
Verification: 285 backend tests · build, import and operational checks passed

AI-assisted engineering

Use model capability where it has the highest leverage

I deliberately separated architectural judgment from repeatable implementation. A stronger reasoning model helped frame the experiment, plan the work packages and critique outcomes. A faster, lower-cost model implemented bounded changes and ran the prescribed checks. The interface between both was a written specification plus machine-readable evidence, not an informal handoff.

REASONING MODEL

Plan and challenge

Frame the failure modes and define measurable acceptance criteria.
Separate development data from the final holdout before tuning.
Review each experiment and decide the next bounded work package.

SPEC ⇄ EVIDENCE

LOWER-COST MODEL

Implement and verify

Implement one scoped change against explicit interfaces and tests.
Run the evaluation matrix and return structured results.
Escalate surprises instead of silently changing the experiment.

The expensive capability is concentrated where judgment has the highest leverage. Repeatable implementation work is delegated with narrow specifications, while tests and frozen measurements remain the common control loop.

Engineering judgment

What I would claim — and what I would not

This benchmark demonstrates reliable retrieval on a bounded, real-world knowledge base and a disciplined route from weak baseline to validated improvement. It does not claim universal RAG quality. Gold documents were selected manually, so question selection bias remains a limitation even though the holdout was frozen and disjoint.

Eight known scanner warnings were reviewed and did not block the measurement, but they remain visible operational debt. Retrieval also supplies evidence rather than truth: the downstream model must still reason, cite and handle a legitimate “no reliable match” result.

Interested in the system or the method?

I am happy to discuss architecture, evaluation design and applied AI engineering.

Contact me