| Model | Quality | p99 latency | Cost per request | vs today | Result |
|---|---|---|---|---|---|
| Gemini 4 Argon Today's modelopenrouter/google/gemini-4-argon | 99.2% replayed on 300 gold rows | 13,459 ms benchmark replies on the same messages | ₹0.1393 benchmark replies on the same messages | — | Today |
| openai_compatible/dvar:gpt-oss-120b | 97.2% replayed on 300 gold rows | 3,793 ms replayed on 300 gold rows | ₹0.0162 replayed on 300 gold rows | −88% | Fails 1 check |
dvar:gpt-oss-120b falls below the minimum on Policy change: 0.91 vs 0.95 (32 gold rows); Suspected fraud: 0.949 vs 0.950 (26 gold rows). Every other gated topic is at or above its minimum. Weakest topic: Policy change 0.91 | |||||
| openai_compatible/dvar:deepseek-v4-flash-0731 | 91.9% replayed on 300 gold rows | 20,061 ms replayed on 300 gold rows | ₹0.0243 replayed on 300 gold rows | −83% | Fails 4 checks |
dvar:deepseek-v4-flash-0731 is fully right on 91.9% of gold rows, under the 95.0% quality bar; fails a critical check on 12 gold rows that today's model passes; falls below the minimum on EMI or payment status: 0.90 vs 0.95 (44 gold rows); Foreclosure or part prepayment: 0.93 vs 0.95 (41 gold rows); Statement or interest certificate: 0.93 vs 0.95 (33 gold rows); Mandate or NACH change: 0.92 vs 0.95 (20 gold rows); Premium due or renewal: 0.92 vs 0.95 (31 gold rows); Claim status: 0.94 vs 0.95 (41 gold rows); Policy change: 0.89 vs 0.95 (32 gold rows); Complaint or escalation: 0.83 vs 0.95 (12 gold rows); Suspected fraud: 0.92 vs 0.95 (26 gold rows). Every other gated topic is at or above its minimum; is too slow: p99 20,061 ms replayed on 300 gold rows, over the 10,000 ms limit. Weakest topic: Complaint or escalation 0.83 | |||||
| openrouter/google/gemini-3.5-flash-lite | 89.6% replayed on 300 gold rows | 1,910 ms replayed on 300 gold rows | ₹0.0247 replayed on 300 gold rows | −82% | Fails 2 checks |
google/gemini-3.5-flash-lite is fully right on 89.6% of gold rows, under the 95.0% quality bar; falls below the minimum on Foreclosure or part prepayment: 0.80 vs 0.95 (41 gold rows); Statement or interest certificate: 0.94 vs 0.95 (33 gold rows); Mandate or NACH change: 0.77 vs 0.95 (20 gold rows); Premium due or renewal: 0.94 vs 0.95 (31 gold rows); Claim status: 0.93 vs 0.95 (41 gold rows); Policy change: 0.78 vs 0.95 (32 gold rows); KYC or contact update: 0.92 vs 0.95 (20 gold rows); Suspected fraud: 0.94 vs 0.95 (26 gold rows). Every other gated topic is at or above its minimum. Weakest topic: Mandate or NACH change 0.77 | |||||
| openai_compatible/dvar:gemma-4-26b-a4b-it | 95.9% replayed on 300 gold rows | 20,982 ms replayed on 300 gold rows | ₹0.0336 replayed on 300 gold rows | −76% | Fails 3 checks |
dvar:gemma-4-26b-a4b-it fails a critical check on 1 gold row that today's model passes; falls below the minimum on Policy change: 0.85 vs 0.95 (32 gold rows). Every other gated topic is at or above its minimum; is too slow: p99 20,982 ms replayed on 300 gold rows, over the 10,000 ms limit. Weakest topic: Policy change 0.85 | |||||
| openai_compatible/dvar:nemotron-3.5-lightning-30b-a3b-bf16 | 94.4% replayed on 300 gold rows | 20,755 ms replayed on 300 gold rows | ₹0.0349 replayed on 300 gold rows | −75% | Fails 4 checks |
dvar:nemotron-3.5-lightning-30b-a3b-bf16 is fully right on 94.4% of gold rows, under the 95.0% quality bar; fails a critical check on 7 gold rows that today's model passes; falls below the minimum on Statement or interest certificate: 0.950 vs 0.950 (33 gold rows); Policy change: 0.82 vs 0.95 (32 gold rows); KYC or contact update: 0.93 vs 0.95 (20 gold rows). Every other gated topic is at or above its minimum; is too slow: p99 20,755 ms replayed on 300 gold rows, over the 10,000 ms limit. Weakest topic: Policy change 0.82 | |||||
| openai_compatible/dvar:glm-5.3-flash | 92.3% replayed on 300 gold rows | 21,869 ms replayed on 300 gold rows | ₹0.0396 replayed on 300 gold rows | −72% | Fails 4 checks |
dvar:glm-5.3-flash is fully right on 92.3% of gold rows, under the 95.0% quality bar; fails a critical check on 11 gold rows that today's model passes; falls below the minimum on EMI or payment status: 0.92 vs 0.95 (44 gold rows); Foreclosure or part prepayment: 0.93 vs 0.95 (41 gold rows); Statement or interest certificate: 0.92 vs 0.95 (33 gold rows); Mandate or NACH change: 0.87 vs 0.95 (20 gold rows); Premium due or renewal: 0.90 vs 0.95 (31 gold rows); Claim status: 0.92 vs 0.95 (41 gold rows); Policy change: 0.92 vs 0.95 (32 gold rows); Complaint or escalation: 0.92 vs 0.95 (12 gold rows). Every other gated topic is at or above its minimum; is too slow: p99 21,869 ms replayed on 300 gold rows, over the 10,000 ms limit. Weakest topic: Mandate or NACH change 0.87 | |||||
| openai_compatible/dvar:glm-5.3 | 96.3% replayed on 300 gold rows | 4,301 ms replayed on 300 gold rows | ₹0.0875 replayed on 300 gold rows | −37% | Fails 1 check |
dvar:glm-5.3 falls below the minimum on Foreclosure or part prepayment: 0.92 vs 0.95 (41 gold rows); KYC or contact update: 0.90 vs 0.95 (20 gold rows). Every other gated topic is at or above its minimum. Weakest topic: KYC or contact update 0.90 | |||||
| openai_compatible/dvar:muse-glimmer-30b | 86.6% replayed on 300 gold rows | 29,531 ms replayed on 300 gold rows | ₹0.1042 replayed on 300 gold rows | −25% | Fails 4 checks |
dvar:muse-glimmer-30b is fully right on 86.6% of gold rows, under the 95.0% quality bar; fails a critical check on 2 gold rows that today's model passes; falls below the minimum on Foreclosure or part prepayment: 0.87 vs 0.95 (41 gold rows); Statement or interest certificate: 0.86 vs 0.95 (33 gold rows); Mandate or NACH change: 0.78 vs 0.95 (20 gold rows); Claim status: 0.79 vs 0.95 (41 gold rows); Policy change: 0.77 vs 0.95 (32 gold rows); KYC or contact update: 0.75 vs 0.95 (20 gold rows); Suspected fraud: 0.94 vs 0.95 (26 gold rows). Every other gated topic is at or above its minimum; is too slow: p99 29,531 ms replayed on 300 gold rows, over the 10,000 ms limit. Weakest topic: KYC or contact update 0.75 | |||||
| Fine-tuned open modelnext step | Measured after tuning on your data. No numbers yet. | Talk to us | |||
Fine-tuned on your masked gold set. Masked rows only. Needs your data-sharing opt-in. | |||||
No candidate passes every check. Open a model's row for its reasons.
Prices are per request, at each provider's list price.
Judge: 0 of 5 checks trusted.
| Topic | Rows | Today · Gemini 4 Argon | Candidate · dvar:gpt-oss-120b | Candidate · dvar:deepseek-v4-flash-0731 | Candidate · google/gemini-3.5-flash-lite | Candidate · dvar:gemma-4-26b-a4b-it | Candidate · dvar:nemotron-3.5-lightning-30b-a3b-bf16 | Candidate · dvar:glm-5.3-flash | Candidate · dvar:glm-5.3 | Candidate · dvar:muse-glimmer-30b | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| EMI or payment status | 44 | 1.00 | 0.98 | −0.02 | 0.90 | −0.10 | 0.98 | −0.02 | 1.00 | 0.00 | 0.95 | −0.05 | 0.92 | −0.08 | 0.96 | −0.04 | 0.95 | −0.05 |
| Foreclosure or part prepayment | 41 | 1.00 | 0.98 | −0.02 | 0.93 | −0.07 | 0.80 | −0.20 | 0.97 | −0.03 | 0.95 | −0.05 | 0.93 | −0.07 | 0.92 | −0.08 | 0.87 | −0.13 |
| Claim status | 41 | 1.00 | 0.98 | −0.02 | 0.94 | −0.06 | 0.93 | −0.07 | 0.95 | −0.05 | 0.96 | −0.04 | 0.92 | −0.08 | 0.98 | −0.02 | 0.79 | −0.21 |
| Statement or interest certificate | 33 | 1.00 | 0.98 | −0.02 | 0.93 | −0.07 | 0.94 | −0.06 | 0.96 | −0.04 | 0.95 | −0.05 | 0.92 | −0.08 | 0.97 | −0.03 | 0.86 | −0.14 |
| Policy change | 32 | 0.96 | 0.91 | −0.05 | 0.89 | −0.07 | 0.78 | −0.18 | 0.85 | −0.11 | 0.82 | −0.14 | 0.92 | −0.04 | 0.99 | +0.03 | 0.77 | −0.19 |
| Premium due or renewal | 31 | 1.00 | 1.00 | 0.00 | 0.92 | −0.08 | 0.94 | −0.06 | 0.97 | −0.03 | 0.97 | −0.03 | 0.90 | −0.10 | 0.99 | −0.01 | 0.96 | −0.04 |
| Suspected fraud | 26 | 0.96 | 0.95 | −0.01 | 0.92 | −0.04 | 0.94 | −0.02 | 0.96 | 0.00 | 0.96 | 0.00 | 0.96 | 0.00 | 0.96 | 0.00 | 0.94 | −0.02 |
| Mandate or NACH change | 20 | 1.00 | 0.97 | −0.03 | 0.92 | −0.08 | 0.77 | −0.23 | 0.95 | −0.05 | 0.98 | −0.02 | 0.87 | −0.13 | 0.98 | −0.02 | 0.78 | −0.22 |
| KYC or contact update | 20 | 1.00 | 1.00 | 0.00 | 0.97 | −0.03 | 0.92 | −0.08 | 1.00 | 0.00 | 0.93 | −0.07 | 0.98 | −0.02 | 0.90 | −0.10 | 0.75 | −0.25 |
| Complaint or escalation | 12 | 1.00 | 1.00 | 0.00 | 0.83 | −0.17 | 0.97 | −0.03 | 1.00 | 0.00 | 1.00 | 0.00 | 0.92 | −0.08 | 1.00 | 0.00 | 1.00 | 0.00 |
| All topics, weighted by traffic | 300 | 0.99 | 0.97 | −0.02 | 0.92 | −0.07 | 0.90 | −0.09 | 0.96 | −0.03 | 0.94 | −0.05 | 0.92 | −0.07 | 0.96 | −0.03 | 0.87 | −0.12 |