CASE 04
10 weeks
Cut first-response time and auto-resolved a meaningful share of tickets.
A multi-brand retailer was drowning in support volume that grew faster than the team could hire. We built a triage and answering layer that classifies every inbound ticket, answers the routine ones with cited product and policy passages, and routes the rest to the right human with a pre-drafted reply attached.
THE PROBLEM
Support volume outgrew the team. First-response time regularly exceeded a day, and routing to the right agent was handled manually by a lead who had become a bottleneck.
Every ticket landed in one shared inbox. The support lead read each one, guessed which of six queues it belonged to, and dragged it across — roughly two and a half hours of a senior person's day spent on triage rather than on the escalations only they could handle.
Answer quality varied by agent tenure. Returns policy differed by brand and by country, and the authoritative version lived in a PDF that had been superseded twice. Agents quoting the wrong policy was a routine and expensive occurrence.
Two previous attempts at a canned-response macro system had failed. Macros went stale, nobody maintained them, and agents stopped opening the panel.
CONSTRAINTS WE WORKED UNDER
WHAT WE DID
Built a triage classifier fine-tuned on twelve months of anonymized tickets.
Stood up a RAG layer over the product catalog and policy docs so agents (and the model) pull exact passages with citations.
Designed an escalation router: confident auto-responses ship, edge cases route to the right human with context pre-drafted.
Shipped an agent console in Next.js with live queues, SLA timers, and a feedback loop that retrains the classifier weekly.
ARCHITECTURE
01
Helpdesk webhooks land on a FastAPI endpoint, get normalized into a common ticket shape regardless of channel (email, web form, chat), and are queued in Redis. The queue is the backpressure valve — peak season floods the queue, not the model provider.
02
A fine-tuned small model assigns intent, brand, urgency and language, returning a confidence score per label. Anything under threshold falls through to a general queue instead of guessing — a wrong confident route costs more than an unrouted ticket.
03
Product catalog, policy documents and prior resolved tickets chunked and indexed in pgvector, filtered at query time by brand and country so a UK policy can never be cited to a Greek customer. Hybrid BM25 + vector retrieval, because SKU codes are keyword problems, not semantic ones.
04
Above the auto-send threshold, a cited answer ships. Below it, the same draft is attached to the ticket for a human to approve or rewrite. Every ticket takes one of two paths — nothing sits in an ambiguous middle state.
05
A Next.js console with live queues, SLA countdowns, the retrieved passages beside the draft, and one-click accept / edit / reject. Rejections are captured with a reason code and become next week's training data.
KEY DECISIONS
WHY
Routing is a narrow, high-volume, low-creativity task with twelve months of labelled examples sitting in the helpdesk. Fine-tuning made it cheaper per call and dramatically faster.
TRADE-OFF
It needs periodic retraining as the catalog changes, so we built the retraining loop as a first-class part of the system rather than a script someone would forget.
WHY
The cost asymmetry is brutal: a good auto-answer saves four minutes, a bad one costs a customer relationship and a supervisor's afternoon.
TRADE-OFF
Auto-resolution rate is lower than the demo-day number would have been. We tuned the threshold down gradually over the first month using real rejection data instead of setting it optimistically on day one.
WHY
The team already lived in it, reporting was built on it, and replacing it would have turned a ten-week project into a migration.
TRADE-OFF
Two-way sync latency and their API rate limits became our problem, so the Redis queue and idempotent writes carry more weight than they otherwise would.
THE HARD PARTS
Retrieval kept surfacing two plausible refund windows because both documents were live. No model fixes that. We built a document-precedence field into the index and made the client nominate one authoritative version per brand — a two-hour meeting that fixed more answer errors than a week of prompt work would have.
Greek customers writing product names in English broke naive language detection and sent tickets to the wrong queue. Language detection moved to the classifier, which sees the whole ticket in context, and retrieval became language-aware rather than language-locked.
The first console version showed the draft but hid the sources behind a click. Agents did not trust it. Putting the cited passages directly beside the draft — visible without interaction — took acceptance from grudging to routine within a week.
HOW IT RAN
01
Pulled and anonymized twelve months of tickets, hand-labelled a 500-ticket golden set with the support lead, and defined what 'correct' meant per intent before writing production code.
02
Fine-tuned and benchmarked the classifier against the golden set, built the ingestion and chunking pipeline, and tuned hybrid retrieval until citation accuracy stopped improving.
03
Shipped the agent console and ran the whole system in shadow — drafting answers nobody sent — for two weeks, comparing model output against what agents actually wrote.
04
Enabled auto-send on the two safest intents first, widened it as the rejection data allowed, then documentation, runbook and a training session with their in-house developers.
OUTCOME
~60%
Lower first-response time
2.5 hrs
Daily triage time returned to the support lead
2 weeks
Shadow mode before a single auto-reply shipped
100%
Auto-answers with a resolvable citation
Enforced structurally: no source, no send.
First-response time dropped by roughly 60%. A meaningful portion of tickets now resolve without a human touch, and the lead is back to doing lead work.
The auto-resolution rate climbed steadily after launch — not because the model improved, but because the rejection reasons agents logged told us exactly which intents were safe to widen. That loop is still running without us.
The unplanned win was the citation requirement. Because every answer resolves to a passage, the client discovered which policy documents were genuinely ambiguous and fixed the documents. Support quality improved for the human-written answers too.
STACK
BACKEND
AI
FRONTEND
OPS
HANDOVER
