CASE 11
10 weeks
Audit questions drafted in minutes, with receipts.
A regulated financial services firm answered audits with Cmd-F across PDFs, producing inconsistent answers depending on which officer did the searching. We built hybrid semantic search over every regulatory and internal policy document, with a drafting interface where the citation trail is the deliverable.
THE PROBLEM
Compliance officers answered audits by searching PDFs with Cmd-F. Every audit was a week of copy-paste, and answers were inconsistent between officers.
Regulatory texts, internal policies and prior audit responses lived in three separate shares with no common search. Officers relied on memory of where a control was documented.
Two officers answering the same question cited different sources — sometimes a superseded policy version — which auditors noticed and which turned a routine question into a finding.
Prior audit responses were the most valuable asset in the building and the least accessible, buried in email threads and named things like final_v3_FINAL.docx.
CONSTRAINTS WE WORKED UNDER
WHAT WE DID
OCR + semantic chunking pipeline for every regulatory document and internal policy.
Hybrid search (BM25 + dense vectors) so keyword-exact queries and meaning-based queries both work.
Draft-an-answer UI where the officer picks the question and gets a cited draft to edit.
Every draft stored with citations so the audit trail is the answer itself.
ARCHITECTURE
01
OCR where needed, section-aware chunking, and an explicit version graph per document. Every chunk knows whether it belongs to the current or a superseded version, and superseded content is retrievable only when the query asks for history.
02
BM25 and dense vectors over the same chunks, fused with reciprocal rank fusion and reranked. Control codes and regulation identifiers hit exactly; 'what do we do about vendor risk' hits semantically. Neither mode alone was adequate in testing.
03
The officer selects an audit question, the system proposes an answer assembled from retrieved passages with citations attached to each assertion, and the officer edits in place. Edits are tracked separately from generated text.
04
Approved answers are stored with their citations and become retrievable themselves, so the next audit starts from the last one instead of from the corpus. This turned out to be the highest-value component.
05
SSO, per-document access rules mirroring their share permissions, and an append-only log of every query, retrieval and approval for the auditors' benefit as much as their own.
KEY DECISIONS
WHY
The first evaluation made it obvious: dense-only retrieval failed on control codes, and BM25-only failed on paraphrased policy questions. Every real query set contained both.
TRADE-OFF
More moving parts and a fusion step to tune. Measured on their own question set, it was not close.
WHY
Auditors ask what the policy was in 2022. Deleting history to keep the index clean would have removed answers they are legally required to produce.
TRADE-OFF
Ingestion complexity and a retrieval filter that must never be forgotten — so it is enforced at the query layer, not left to the caller.
WHY
Compliance answers carry personal accountability. Officers were never going to sign something they had not shaped, and building for auto-answer would have produced a tool nobody used.
TRADE-OFF
Less automation on paper. In practice the drafting step is where the week of work disappeared.
THE HARD PARTS
Naive text extraction interleaved columns and destroyed tables, producing chunks that read as nonsense. Layout-aware extraction with table handling was three days of unglamorous work that improved answer quality more than any retrieval tuning afterwards.
Early on, a draft cited a policy replaced in 2023. That single incident set the design rule: version filtering enforced at the query layer, with historical retrieval requiring an explicit intent from the officer.
The answer library was originally a nice-to-have. Once officers started drafting from previously approved answers, inter-officer consistency stopped being a training problem and became a default.
HOW IT RAN
01
Inventoried documents, measured extraction quality, and collected a real question set from the last three audits to evaluate against.
02
Layout-aware extraction, OCR, section-aware chunking, and the document version graph.
03
BM25 and dense indexes, fusion and reranking, tuned against the audit question set until the correct source ranked first consistently.
04
Draft-and-approve workflow, citation rendering, the approved-answer library, then rollout across a live audit.
OUTCOME
Days → hours
Time to produce an audit response
Consistent
Two officers, same question, same citations
Full history
Superseded versions retrievable, never citable as current
Every draft
Stored with its citation trail as audit evidence
Audit responses that used to take days now take hours, and two officers generating the same answer produce the same citations.
The answer library compounds. Each audit leaves the corpus better than it found it, which means the fourth audit is meaningfully cheaper than the first.
The unexpected outcome was gap detection: questions where retrieval found nothing turned out to mark genuine documentation gaps, which the team now works through deliberately.
STACK
BACKEND
RETRIEVAL
INGESTION
GOVERNANCE
HANDOVER
