| Model | Quality | p99 latency | Cost per request | vs today | Result |
|---|---|---|---|---|---|
| Gemini 3.8 Flash Stand-in for today's modelopenrouter/google/gemini-3.8-flash | 85.0% | 36,399 ms | ₹0.4219 | — | Today |
| DeepSeek V4 Flashopenai_compatible/dvar:deepseek-v4-flash-0731 | 88.8% | 12,419 ms | ₹0.0468 | −89% | Recommended |
| openai_compatible/dvar:glm-5.3-flash | 85.0% | 10,491 ms | ₹0.0579 | −86% | Fails 1 check |
dvar:glm-5.3-flash falls below the minimum on Settlement status: 0.85 vs 0.87 (48 gold rows); Delayed, on hold or failed: 0.85 vs 0.87 (26 gold rows). Every other gated topic is at or above its minimum. Weakest topic: Delayed, on hold or failed 0.85 | |||||
| openai_compatible/dvar:gemma-4-31b-it | 83.1% | 27,622 ms | ₹0.0718 | −83% | Fails 3 checks |
dvar:gemma-4-31b-it is fully right on 83.1% of gold rows, under the 84.0% quality bar; falls below the minimum on UTR and proof: 0.79 vs 0.84 (19 gold rows). Every other gated topic is at or above its minimum; is too slow: p99 27,622 ms replayed on 160 gold rows, over the 15,000 ms limit. Weakest topic: UTR and proof 0.79 | |||||
| openrouter/anthropic/claude-sonnet-5.5 | 87.5% | 19,510 ms | ₹0.6816 | +62% | Fails 2 checks |
anthropic/claude-sonnet-5.5 falls below the minimum on Account and plan changes: 0.67 vs 0.78 (12 gold rows). Every other gated topic is at or above its minimum; is too slow: p99 19,510 ms replayed on 160 gold rows, over the 15,000 ms limit. Weakest topic: Account and plan changes 0.67 | |||||
Post-trained Dvar model: next step, measured after training on the assistant's outcomes. No numbers yet. It goes through this same gate: Fine-tune. | |||||
DeepSeek V4 Flash passes every check: 3.8 points higher than today's model at 89% less cost.
Prices are per request, at each provider's list price.
Judge: 7 of 7 checks trusted.
| Topic | Rows | Today · Gemini 3.8 Flash | Candidate · DeepSeek V4 Flash | Candidate · GLM 5.3 Flash | Candidate · Gemma 4 31B | Candidate · Claude Sonnet 5.5 | ||||
|---|---|---|---|---|---|---|---|---|---|---|
| Settlement status | 48 | 0.92 | 0.92 | 0.00 | 0.85 | −0.07 | 0.92 | 0.00 | 0.94 | +0.02 |
| Delayed, on hold or failed | 26 | 0.92 | 0.88 | −0.04 | 0.85 | −0.07 | 0.92 | 0.00 | 1.00 | +0.08 |
| UTR and proof | 19 | 0.89 | 0.89 | 0.00 | 0.95 | +0.06 | 0.79 | −0.10 | 0.95 | +0.06 |
| Account and plan changes | 12 | 0.83 | 0.83 | 0.00 | 0.83 | 0.00 | 0.83 | 0.00 | 0.67 | −0.16 |
| Amount and deductions | 36 | 0.75 | 0.92 | +0.17 | 0.83 | +0.08 | 0.75 | 0.00 | 0.75 | 0.00 |
| Payouts | 13 | 0.69 | 0.69 | 0.00 | 0.85 | +0.16 | 0.77 | +0.08 | 0.85 | +0.16 |
| Instant settlement | 6 | 0.83 | 1.00 | +0.17 | 0.67 | −0.16 | 0.50 | −0.33 | 0.83 | 0.00 |
| All topics, weighted by traffic | 160 | 0.85 | 0.89 | +0.04 | 0.85 | 0.00 | 0.83 | −0.02 | 0.88 | +0.03 |