How this evaluation was run
The same five steps a model factory runs for any workflow. Written for an AI or servicing team that wants to check the work.
The loop
A model factory picks the model for each workflow on evidence. Observability shows what every model did on every message. Observe feeds the next Evaluate: messages that go wrong become new test cases.
Why this workflow
Aditya Birla Capital already runs AI in servicing at scale. With many use cases live, the next questions are which model each workflow needs, what it costs per message, and how to see what each model did.
Aditya Birla Capital's Q4 FY2025-26 investor presentation lists Gen AI email bots servicing 40% of customer email, 26 AI voice bots across the customer lifecycle, and Gen AI Service Assist for contact centre agents.
Microsoft India's AI First Movers page says Aditya Birla Capital is integrating Azure OpenAI Services and building solutions including a Sales Pitch Assistant and Service Assist.
1. Evaluate
The task
For each servicing message, a model returns every request it contains (one or more of ten request types), the product it concerns and an urgency flag (fraud, complaint, hospitalisation claim).
| Area | Request type | Includes | Messages |
|---|---|---|---|
| Loans | EMI or payment status | EMI paid but not showing, EMI due amount or date, bounce or late-payment charges, overdue amount. | 44 |
| Loans | Foreclosure or part-prepayment | Foreclosure quote or process, part-prepayment, closure charges. | 41 |
| Loans | Statement or interest certificate | Account statement, repayment schedule, interest certificate, NOC or closure letter. | 42 |
| Loans | Mandate or NACH change | Change or re-register NACH or e-NACH bank account, cancel a mandate, change the EMI debit date. | 41 |
| Insurance | Premium due or renewal | Premium due date or amount, renewal, grace period, revival of a lapsed policy. | 42 |
| Insurance | Claim status | Claim intimation or status, pending documents, cashless pre-authorisation, settlement amount. | 41 |
| Insurance | Policy change | Nominee change, policy document copy, address change for a specific policy. | 44 |
| Common | KYC or contact update | Mobile, email or address on the customer record, PAN or Aadhaar re-KYC, name correction. | 39 |
| Common | Complaint or escalation | Explicit complaint, escalation request, regulator or court threat, agent harassment. An angry tone alone is not a complaint. | 40 |
| Common | Suspected fraud | Loan or policy taken without consent, unauthorised debit, OTP shared with a fraudster, phishing. | 51 |
The messages
- 300 synthetic messages: 150 in English and 150 in Hinglish (Roman script). 100 (33%) carry two or three requests.
- Channels: WhatsApp 121, Email 99, App ticket 80. Tones from neutral to angry, including ombudsman threats.
- Hard cases: Negation 14, Buried fraud 14, Product mix-up 12.
- The answer key was fixed first by a seeded plan; a separate model (Gemma 4 31B, not one of the models compared) then wrote one message per plan item. Every label was reviewed against the definitions before any model was scored; unclear ones were flagged, and the fixes were approved by our team.
- A separate set of 60 messages was kept for checking prompt and settings. The scored set was not used for tuning.
- No Aditya Birla Capital data, names or policies were used. Customer names come from a held-out list; account numbers are masked.
Scoring
- Fully right message: the set of request types matches the answer key exactly, and no fraud or complaint flag is missed. Product and urgency are scored separately.
- Accuracy per request type: the F1 score, high only when a model both catches the request type and does not tag it where it is absent. "Caught" is recall: the share of messages with that request type the model spotted.
- Cost: list price. Dvar portal prices in INR; frontier cost as reported by OpenRouter, converted at ₹84 per USD.
- Eval Console checks: five checks per reply. Only a missed fraud or a missed complaint is a must-pass check, as in the scorer; request types, urgency and product are scored but not must-pass. The console's quality gates use a 1-point margin and are not relaxed for any model.
- Automatic judge: Claude Sonnet 5.5 (not one of the compared models) scores today's model's replies on the same checks, so the console can show how often it agrees with the reviewed answer key. The answer key, not the judge, decides the scores.
- Latency: time to a full answer, measured from our test environment; median and slowest 5%.
- All numbers come from the harness output file, not typed by hand: results/servicing-intent-lending-insurance/merged-20261021T061351Z.json, run merged-20261021T061351Z.
2. Select
Six models ran with the same instructions, temperature 0, a 2048-token limit and 3 runs each. The open model shown on the other screens is the open model with the most messages fully right: gpt-oss-120b. A request type goes to it when its accuracy is no more than 2 points below frontier (Gemini 4 Argon).
| Model | Type | Input price, per 1M tokens | Output price, per 1M tokens |
|---|---|---|---|
| GLM 5.3 Flash | Open model run by Dvar in India | ₹10.00 | ₹40.00 |
| gpt-oss-120b | Open model run by Dvar in India | ₹10.00 | ₹40.00 |
| Nemotron 3.5 30B | Open model run by Dvar in India | ₹5.00 | ₹20.00 |
| DeepSeek V4 Flash | Open model run by Dvar in India | ₹10.00 | ₹20.00 |
| Gemini 4 Argon | Leading paid model | $2.00 | $10.00 |
| Gemini 3.5 Flash Lite | Leading paid model, low-cost tier | $0.30 | $2.50 |
| GLM 5.3 | Open model run by Dvar in India | ₹105.00 | ₹340.00 |
| Muse Glimmer 30B | Open model run by Dvar in India | ₹25.00 | ₹110.00 |
| Gemma 4 26B | Open model run by Dvar in India | ₹10.00 | ₹30.00 |
Frontier prices are shown in USD as listed; the benchmark uses the cost OpenRouter reports for each call where available.
3. Tune
No model was tuned for this demo. For the request types still on frontier, the next lever is to tune the open model on your own labelled servicing messages and re-run the same evaluation. Tuning is worth it only where the evaluation shows a gap, and it is checked against the same answer key.
4. Serve
The open-weight models are served on Dvar Inference, hosted in India. Frontier models were called through OpenRouter. The assistant on this site calls both live with the same instructions and a one-page sample policy sheet.
5. Observe
Every model call is recorded with its input, output, expected answer, pass or fail, latency and cost. The Evals tab shows all of them. In production the same traces become the quality monitor: new failures become new test cases for the next evaluation.
Limits of this test
- Synthetic messages are cleaner than real ones. Real emails have more noise, attachments and context from earlier messages.
- 300 messages give a useful signal but small differences between models (a point or two) are within noise.
- Latency was measured from one location at one time of day.
- Prices are list prices on the day of the run; negotiated or batch prices would change the totals for every model.