What is measured, written and assumed

Measured

• Run bench-paytm-v3m: 160 chats, temperature 0, prompt paytm-v3, the same prompt, tools and JSON tool protocol for every model.
• ₹: billed cost for frontier models, Dvar list price for open models.
• Replies cut off at the token limit were re-run once, for every model, with a larger limit (reasoning models count thinking inside it).
• Fallback rows: the open model's reply unless a critical check or the judge failed it; then today's model's. Both calls paid.
• Gateway overhead: p50 0.008 ms, p95 0.026 ms over 1,821 calls.
• Judge: openai/gpt-6.1-sol, 855 verdicts, every reply of every model; it never saw the gold, and it is not from the same provider as any frontier model compared.
• A pilot scores with human review.

Written for the demo

• A synthetic ledger: 40 merchants (QR and Soundbox shops, and online stores), 90 days, from seed 20261022. T+1 settlement, no fee on UPI, 2% on cards, a monthly Soundbox rental. No real account or phone number.
• A settlement policy sheet, illustrative for this demo, not Paytm's policy.
• The gold set: actions and facts computed by code; a model wrote only the merchant's wording. Test 396 chats, 160 benchmarked.

Assumed

• Today's model: Gemini 3.8 Flash.
• The bar (84.0%) and the 15 s p99 gate: to set with Paytm.

Choices the runs forced

• Every model used the same JSON tool protocol.
• The rules router is off.

Spend

₹810.21 across 7,101 calls.