Evals · Summary
DeepSeek V4 Flash: as good, 9.4× cheaper
| Model | Resolved | p99 | Per 1M chats | vs today |
|---|---|---|---|---|
Gemini 3.8 Flash Today | 85.0% | 32.8 s | ₹4.2 lakh | – |
Claude Sonnet 5.5 Frontier | 87.5% | 18.9 s | ₹6.8 lakh | +₹2.6 lakh |
DeepSeek V4 FlashBest Open · Dvar | 88.8% | 12.3 s | ₹46,800 | −₹3.8 lakh |
GLM 5.3 Flash Open · Dvar | 85.0% | 10.2 s | ₹57,900 | −₹3.6 lakh |
Gemma 4 31B Open · Dvar | 83.1% | 36.1 s | ₹71,800 | −₹3.5 lakh |
By topic
| Topic | Gemini 3.8 Flash | Claude Sonnet 5.5 | DeepSeek V4 Flash | GLM 5.3 Flash | Gemma 4 31B |
|---|---|---|---|---|---|
| Settlement status | 92% | 94% | 92% | 85% | 92% |
| Delayed, on hold or failed | 92% | 100% | 88% | 85% | 92% |
| Amount and deductions | 75% | 75% | 92% | 83% | 75% |
| UTR and proof | 89% | 95% | 89% | 95% | 79% |
| Payouts | 69% | 85% | 69% | 85% | 77% |
| Instant settlement | 83% | 83% | 100% | 67% | 50% |
| Account and plan changes | 83% | 67% | 83% | 83% | 83% |