Imperial College - Building the Complaints Handling Bot and Evaulating Output

This is the fourth post in a mini-series on my Imperial College AI for Business programme. The story so far: a reflection on the programme, the Cashflow Copilot concept that came out of the capstone, and the RAG evaluation framework I built to measure whether chunking strategy actually matters. This post covers the build of the bot itself.

The bot

The FCA complaints handling assistant takes a complaint scenario, retrieves the relevant FCA Handbook provisions, and drafts a compliant response grounded in that regulatory text. It cites the specific provisions it used and flags cases that need human review. The corpus is the FCA Handbook (primarily the DISP sourcebook for dispute resolution, plus CONC for consumer-credit complaints and the conduct modules) alongside a set of synthetic bank compliance policies, so a response can be grounded in both the regulator's rules and the bank's internal policy.

The interface is a FastAPI and HTMX web app that a complaints handler could actually use. Generation runs through Claude Sonnet 4.6 with native structured outputs.

If you read the first post, you'll know I pivoted away from this use case during the programme. In Module 4 I set the complaints bot aside in favour of Cashflow Copilot: the complaints idea was a solid internal-efficiency play, but I wanted something more ambitious. I came back to it here because it's the cleaner technical demonstration of the eval framework. Cashflow Copilot is a three-layer product; the complaints bot is a single, focused RAG task on regulated text. Commercially I still favour Cashflow Copilot. Technically, the complaints bot is the better instrument.

What I reused and what was rebuilt

The project is a clone of the eval framework, but a self-contained one: its own S3 bucket, its own Knowledge Bases, its own corpus, nothing shared with the framework's live infrastructure.

.n-gray{fill:#ecedef;stroke:#c0c2c6}.n-blue{fill:#E6F1FB;stroke:#185FA5}.n-teal{fill:#E1F5EE;stroke:#0F6E56} .d-title{font-family:'Source Sans Pro',sans-serif;font-size:13px;font-weight:600} .d-sub{font-family:'Source Sans Pro',sans-serif;font-size:11px;font-weight:400} .t-gray{fill:#3d3d3a}.ts-gray{fill:#73726c}.t-blue{fill:#0C447C}.ts-blue{fill:#185FA5}.t-teal{fill:#085041}.ts-teal{fill:#0F6E56} .arr{stroke:#c0c2c6;fill:none}.arr-teal{stroke:#0F6E56;fill:none} rag-eval-framework the measurement tool · post 16 FCA complaints bot retrieve · draft · cite · flag Drafting eval structure vs fixed Rebuilt for the drafting task corpus +DISP/CONC · two-dimension rubric · FastAPI UI 

Reused: the FCA scraper, the structure-aware chunker, the Bedrock Knowledge Base setup, the retrieval pipeline, and the dual-judge harness. The approach and most of the code carried straight over.

Rebuilt: three things. The corpus, because the foundation didn't include DISP and a complaints bot cannot function without it; the scraper now pulls DISP and CONC and the corpus is rebuilt fresh. The generation layer, because the bot drafts responses rather than answering questions. And the judge rubric, which is where the real work was.

Why drafting broke the rubric

The Q&A judge decomposes an answer into factual claims and grades each one's grounding against the retrieved chunks. That works when the whole answer is supposed to come from the source. But a compliant complaint response is mostly not drawn from the Handbook: the acknowledgement, the empathy, the summary of the investigation, the next steps. Run the old rubric unchanged and all of that legitimate content scores as ungrounded and tanks faithfulness for entirely the wrong reason.

So I split the scoring into two dimensions. Regulatory grounding asks whether the regulatory claims are correct and cited to the right provision, with no invented rules. Required-elements compliance asks whether the draft contains the things a compliant reply must contain: referral rights to the Financial Ombudsman Service, the relevant time limits, fair-treatment language, expressed in compliant terms. The first dimension is where citing the correct provision matters; the second is where the draft's quality as a response matters. Keeping them separate is what makes the eval honest about a task where most of a good answer is legitimately unsourced.

What the eval found

The eval framework post landed on a counterintuitive result for factual Q&A: structure-aware chunking delivered 3× the retrieval precision of fixed-size splitting, but answer faithfulness barely moved; both sat around 90% grounded. Claude Sonnet 4.6 was good enough at ignoring irrelevant chunks that the generation model compensated for noisy retrieval.

On the drafting task, that finding breaks. Here are the two configurations (fixed-size and structure-aware chunking, both on Titan embeddings) across 35 complaint scenarios, each scored by two independent judges.

Structure-aware chunking wins on every metric, which it did not in the Q&A run. But the number that tells the story is the retrieval-citation gap. It's the difference between how much of the expected rule text was actually retrieved and how many of the expected provisions the bot cited. A negative gap means the bot cited provisions whose rule text it had not retrieved; it was citing from the model's memory rather than from grounded context. Fixed-size chunking sits at -12.4%. Structure-aware closes it to -0.5%: what it cites is what it retrieved.

That's the mechanism. When a complaint draft has to cite the correct DISP rule, the retrieval has to actually surface that rule's text. Fixed-size chunking retrieves the right provision's text only 11.4% of the time, so the model is repeatedly forced to cite a provision it can name but wasn't given; an ungrounded citation dressed up as a grounded one. Structure-aware chunking surfaces the rule text 32.6% of the time, so its citations have something to stand on.

The grounding score tells the same story. In the Q&A run, fixed-size chunking scored 89% grounded; the generation model papered over the retrieval noise. On the drafting task, the same config drops to 72%. The compensation that held up for answering doesn't hold up for drafting, because drafting demands provenance, not just plausibility. Citing the correct provision is a stricter bar than stating a correct fact, and it's far more sensitive to retrieval precision.

The read on the numbers

The absolute scores are modest. Rubric coverage of 55%, section precision of 14%, citation precision of 31% are a long way below the Q&A run's numbers. That's expected, and it's worth being clear about why rather than dressing it up. Drafting is a harder task, and these metrics are deliberately stricter: a citation only counts if it's the correct provision, and coverage measures specific required compliance elements, not general plausibility.

More to the point, absolute scores from LLM judges are judge-dependent and don't travel well between setups; what transfers is the delta between configurations measured by the same judges. So the finding here is not "the bot scored 55%". The finding is that the gap between structure-aware and fixed-size chunking, which was invisible in faithfulness terms for Q&A, is real and consistent for drafting, and the retrieval-citation gap explains why. The value is in the comparison and the diagnostic, not the leaderboard number.

Hardening it as if it were real

One thread from the programme that stuck with me was Module 6's framing of ethics as architecture: safeguards as structural design decisions, not a compliance checklist bolted on afterwards. The bot runs entirely on synthetic data, but I took that idea seriously and added a data-protection and security layer as if it handled real complainant PII. Personal data is pseudonymised before anything reaches the model, the audit trail records every interaction without storing any PII, processing is pinned to EU regions, and the known gaps are documented rather than hidden. The details, including a data-flow diagram and a control mapping across GDPR, FCA/PRA, DORA and the EU AI Act, are in the repository.

Closing the bot out

Course to concept to measurement tool to working bot. The eval framework set out to make RAG evaluation systematic rather than vibes-based, and the most useful thing it did here was prove that a finding doesn't transfer: the Q&A conclusion that chunking barely affects answer quality is simply false for drafting, and only measurement revealed that. Clone the framework, swap the corpus, rework the rubric for the task, re-run; the answer changed. That's the whole argument for building the measurement tool in the first place. If RAG evaluation is task-specific, you need a way to re-measure each time, not a rule of thumb carried over from the last task.

The complaints bot and the eval framework are both on GitHub.