Concept demo

9 AI models, 300 servicing messages, one answer key

Each message was sent 3 times to every model, to check it answers the same way each time. Internal benchmark on synthetic messages.

9 models, ranked

Open model, Dvar IndiaLeading paid model95% bar

Share of the 300 messages each model got fully right (every request, product and urgent flag), and its cost per message at list price.

ModelFully rightPer message
Gemini 4 ArgonPaid99.2%meets bar₹0.1393
gpt-oss-120bOpen · Dvar India97.2%meets bar₹0.0162
GLM 5.3Open · Dvar India96.3%meets bar₹0.0875
Gemma 4 26BOpen · Dvar India95.9%meets bar₹0.0336
Nemotron 3.5 30BOpen · Dvar India94.4%below bar₹0.0349
GLM 5.3 FlashOpen · Dvar India92.3%below bar₹0.0396
DeepSeek V4 FlashOpen · Dvar India91.9%below bar₹0.0243
Gemini 3.5 Flash LitePaid89.6%below bar₹0.0247
Muse Glimmer 30BOpen · Dvar India86.6%below bar₹0.1042
Today

An open model clears the bar at a fraction of the cost

gpt-oss-120b, run by Dvar in India, got 97.2% fully right, above the 95% bar, and more than Gemini 3.5 Flash Lite (89.6%). It costs ₹0.0162 a message against ₹0.1393 for Gemini 4 Argon: 8.6× lower.

Next step

Fine-tune it for the 2 topics still under 95%

  • Policy change 32 messages90.6%
  • Suspected fraud 26 messagesThe fraud is always flagged; another part of the answer was sometimes wrong.94.9%

Overall it is above the bar, but these kinds of message are not yet. Training it further on Aditya Birla Capital's own servicing messages targets exactly these, and we re-test against the same 95% bar before anything goes live.

Head to head: Gemini 4 Argon vs gpt-oss-120b

P
Leading paid modelGemini 4 Argon
Fully right99.2%893 of 900 runs
Per message₹0.1393list price
Reply time3.4 smedian, p95 5.6 s
Fraud caught100.0%153 of 153 runs
Complaints caught100.0%120 of 120 runs
D
Open model run by Dvar in Indiagpt-oss-120b
Fully right97.2%875 of 900 runs
Per message₹0.0162list price
Reply time1.6 smedian, p95 2.9 s
Fraud caught100.0%153 of 153 runs
Complaints caught100.0%120 of 120 runs
CostOpen model costs ₹0.1231 less per message (88% lower)
QualityBoth above the 95% bar; paid model 2.0 points ahead
SpeedOpen model replies 1.8 s faster (median)

Loading test cases

Breakdown

Request typePaidOpen
EMI or payment status (44)100.0%98.1%
Foreclosure or part-prepayment (41)100.0%100.0%
Statement or interest certificate (42)100.0%98.0%
Mandate or NACH change (41)98.8%100.0%
Premium due or renewal (42)100.0%100.0%
Claim status (41)100.0%100.0%
Policy change (44)100.0%96.5%
KYC or contact update (39)98.3%97.5%
Complaint or escalation (40)100.0%98.8%
Suspected fraud (51)100.0%100.0%

How well each model spots each kind of request: it catches the request when it is there and does not add it when it is not. Green: the open model is within 2 points of the paid model. Amber: behind, a fine-tuning target.

Models Dvar runs in India

9 available · scroll for all
gpt-oss-120bSelected₹10 in / ₹40 out per 1M tokens
97.2%fully right
GLM 5.3₹105 in / ₹340 out per 1M tokens
96.3%fully right
Gemma 4 26B₹10 in / ₹30 out per 1M tokens
95.9%fully right
Nemotron 3.5 30B₹5 in / ₹20 out per 1M tokens
94.4%fully right
GLM 5.3 Flash₹10 in / ₹40 out per 1M tokens
92.3%fully right
DeepSeek V4 Flash₹10 in / ₹20 out per 1M tokens
91.9%fully right
Muse Glimmer 30B₹25 in / ₹110 out per 1M tokens
86.6%fully right
Gemma 4 31B₹10 in / ₹30 out per 1M tokens
Wrote the synthetic messages; not tested
Qwen 3.8 27B₹25 in / ₹210 out per 1M tokens
Not tested on this set

Live list from Dvar Inference, 2026-10-21. Prices as recorded from the Dvar portal in the benchmark config.