Imperial College - Does Chunking Strategy Matter? Measuring RAG Quality on Regulatory Text

This is the third post in a mini-series on my Imperial College AI for Business programme. The capstone work developed two use cases that rely on RAG: the Cashflow Copilot's scenario simulation layer and an FCA complaints handling bot. Before building either, I wanted a systematic way to measure whether chunking strategy actually matters for answer quality. This framework is that measurement tool.

Everyone building RAG systems makes the same claim: "use better chunking and your retrieval improves." But how much? And does it actually affect answer quality, or just retrieval metrics? I built a framework to find out, using UK financial regulation as the stress test.

The result is rag-eval-framework: a configurable evaluation harness that runs ground-truth Q&A pairs through multiple RAG pipeline configurations, scores each answer with dual LLM judges, and outputs comparison tables. The headline finding: structure-aware chunking delivers 3× better retrieval precision than fixed-size splitting, but answer faithfulness barely moves.

The question

RAG pipelines have several configuration choices: how you chunk documents, which embedding model you use, what retrieval method you apply, and which generation model synthesises the answer. Each choice affects the others, and without measuring it's hard to know which ones actually matter for the output quality you care about. I wanted to find out.

I chose UK Financial Conduct Authority (FCA) Handbook text as the corpus. It aligns directly to both Imperial programme use cases (complaints handling and Cashflow Copilot both operate in the FCA regulatory domain), and regulatory data is a demanding stress test for RAG: it has explicit document hierarchy (modules, chapters, sections, provisions), dense cross-references between sections, and precise terminology where a wrong word changes the legal meaning. If your chunking strategy breaks a provision across two chunks, a compliance question might get half an answer.

The pipeline

The framework chunks the corpus several ways, indexes each version in its own Bedrock Knowledge Base, retrieves against a ground-truth Q&A set, generates answers with Claude, and scores each answer with two independent judges.

.n-gray{fill:#ecedef;stroke:#c0c2c6}.n-blue{fill:#E6F1FB;stroke:#185FA5}.n-teal{fill:#E1F5EE;stroke:#0F6E56} .d-title{font-family:'Source Sans Pro',sans-serif;font-size:13px;font-weight:600} .d-sub{font-family:'Source Sans Pro',sans-serif;font-size:11px;font-weight:400} .t-gray{fill:#3d3d3a}.ts-gray{fill:#73726c}.t-blue{fill:#0C447C}.ts-blue{fill:#185FA5}.t-teal{fill:#085041}.ts-teal{fill:#0F6E56} .arr{stroke:#c0c2c6;fill:none}.arr-teal{stroke:#0F6E56;fill:none} FCA Handbook 1,246 sections Chunking strategies fixed-size + structure-aware 4 Bedrock KBs S3 Vectors Retrieve + generate Claude Sonnet 4.6 Eval harness dual judges + 7 metrics Comparison tables 

The configurations

I tested four pipeline configurations: two chunking strategies crossed with two embedding models. The original plan included hybrid search for eight configurations; that got cut when S3 Vectors turned out not to support it (more on that below).

Fixed-size chunking is the baseline. Bedrock's default FIXED_SIZE strategy splits documents into equal-length windows with overlap. Simple, fast, and naive; it has no awareness of document structure.

Structure-aware chunking uses the FCA Handbook's provision hierarchy. Each chunk contains one or more complete regulatory provisions from the same section, packed up to a dual size cap (2,048 characters and 480 Cohere tokens). The chunker is custom Python that parses provision boundaries from the markdown and packs them into coherent units.

Titan Embeddings V2 and Cohere Embed English v3 are the two embedding models, both via Bedrock Knowledge Bases. Cohere has a 512-token hard limit that forced the token cap in the structure-aware chunker; it hard-fails on anything longer, which I found out during ingestion when entire sections silently disappeared.

The results

78 ground-truth Q&A pairs, each scored by two independent judges (Claude Haiku 4.5 and GPT-oss-120b), across all four configurations.

The precision gap is striking: structure-aware returns relevant chunks 38% of the time vs 13% for fixed-size. That's 3× less noise in the context window. Recall is uniformly high across all configs (96-99%); every configuration finds the relevant sections, and the difference is how much irrelevant material comes along for the ride.

The surprise is faithfulness. Despite 3× worse precision, fixed-size chunking produces answers that are just as well-grounded (~90% across the board, under 2% hallucination). Claude Sonnet 4.6 with structured outputs appears to filter out irrelevant chunks effectively and ground its answers in the relevant ones. The generation model compensates for retrieval noise.

This has a practical implication: if you only care about answer quality and not about efficiency (token cost, context window usage), fixed-size chunking might be good enough. But if you're paying per token or working near context limits, the 3× precision improvement matters a lot.

The eval framework

Measuring RAG quality properly required building a few things that don't exist off the shelf.

Ground-truth Q&A generation. 78 curated pairs generated by Claude Opus 4.6 and validated by GPT-oss-120b as an independent critic. Three types: single-module (answerable from one FCA section), cross-module (requiring 2-3 sections), and policy-regulation (spanning synthetic bank policies and their referenced FCA provisions). Each pair records the specific sections that should be retrieved, giving a ground truth for retrieval metrics.

Claim-level faithfulness scoring. Each judge decomposes the generated answer into individual factual claims and classifies each as GROUNDED, PARTIALLY_GROUNDED, or UNGROUNDED against the retrieved chunks. This avoids the score-clustering problem of holistic 1-5 ratings; "87% of claims grounded" is more interpretable than "3.8 out of 5".

Dual judges from different providers. Using Claude to judge Claude's output risks self-enhancement bias. The dual-judge design (Claude Haiku and GPT-oss-120b) with inter-judge agreement metrics makes bias transparent. Agreement was around 77%, with divergence mostly at the GROUNDED/PARTIALLY_GROUNDED boundary.

What I built along the way

The evaluation framework is the point of the project, but building it required a full RAG pipeline as the thing being evaluated: an FCA scraper that pulls the full Handbook via their API into a 28MB JSONL file (1,246 sections); a synthetic policy generator that creates 20 realistic bank compliance policies referencing specific FCA provisions, giving the eval a real-world dimension beyond raw regulatory text; and a setup script that creates and syncs four Bedrock Knowledge Bases with metadata sidecars for section-level filtering.

Generation runs through the Bedrock KB Retrieve API and Claude Sonnet 4.6 with native structured outputs. Passing a Pydantic-derived JSON schema via outputConfig.textFormat constrains Claude's response at the grammar level, so there are no parsing hacks; the first call compiles the schema (around 30s), then it's cached for 24 hours.

The eval runner itself is a CLI that iterates Q&A pairs across configs, calls the pipeline, runs dual judges, computes seven metrics (precision, recall, faithfulness, hallucination rate, citation accuracy, latency, cost), and outputs comparison tables. It checkpoints after every single question, because losing a multi-hour run to a crash partway through is not acceptable; a lesson learned the hard way.

Things that didn't work

Hybrid search on S3 Vectors. The original plan was an eight-configuration matrix, adding hybrid search (dense plus BM25 via Reciprocal Rank Fusion). I chose S3 Vectors as the vector store because it's fully managed with zero provisioning. It turns out hybrid search throws a ValidationException on S3 Vectors-backed KBs. The docs mention this obliquely, but I had to hit the error to confirm it. The configuration matrix dropped from eight to four.

GPT-4o as a judge. The original design used GPT-4o as the secondary judge. By the time I built the eval framework, GPT-4o was retired on Bedrock and the replacements use the Responses API rather than the Converse API. GPT-oss-120b works with Converse, but it's a reasoning model that returns reasoningContent blocks instead of plain text; the response parser needed to handle both formats.

Per-config checkpointing. My first attempt at incremental saves wrote results after each config completed. When the run crashed at question 59, about 40 minutes of work vanished. The fix was per-question checkpointing with a separate checkpoint file and resume support. Should have built it that way from the start.

What I'd do differently

Build the checkpointing and resume logic into the eval runner from day one. Any long-running batch process needs it. It's boring infrastructure work, but it's the difference between a framework you can actually use and one that works in theory.

What's next

The broader interest is in making RAG evaluation systematic rather than vibes-based. The framework is designed to be reusable: point it at a different corpus, generate new Q&A pairs, and run the same eval. The chunking strategies would need adapting for non-hierarchical documents, but the eval harness, judge design, and metrics computation are domain-agnostic.

The next step is to clone this framework and apply it to the Module 1 use case: an FCA complaints handling bot using the same regulatory corpus as its knowledge base.

The source code is on GitHub.