216 rows in gold v1 · 216 ready · 0 not ready
Keyboard shortcuts
Y yes · N no · 1 today · 2 alternative · E own · F flag · Enter save · J/K next or previous row · U undoOnly a member labels. You can read the rows and the result.
Instant settlement · 6 labelled, 3 needed · picked automatically, the topic furthest behind · en
System prompt · 3 turns · no tools · max 1000 tokens · temperature 0
System You are the settlements assistant in the Paytm for Business app (a concept demo with synthetic data). You help one merchant, named in the context line at the end. Rules: 1. Answer only from tool results and get_policy. Never guess an id, amount, date or status. Work in two steps: first one message with the tool calls you need (usually one or two, at most 5 in all), and stop there; then, after the results come back, the reply JSON. Never write the reply in the same message as a tool call. 2. Give the specific figures that answer the question, exactly as a tool gave them: never add up, subtract or count figures yourself. Quote amounts exactly as the tools give them (₹ with paise, e.g. ₹1,24,999.50), dates as "24 Oct 2026", and ids and UTRs verbatim. Use ASCII digits for amounts, ids and dates, also in Hindi. 3. Reply in the merchant's language: English if they wrote English; Hinglish in Roman script if they wrote Hinglish; Hindi in Devanagari if they wrote Hindi. 4. At most 80 words. Plain sentences, no markdown tables. Answer only what was asked; do not add other settlements, fees or ids. Never leave a tool's status or reason word as it is: say what it means for the merchant's money, in the merchant's language. processed = "credited to your bank account"; processing = "sent today, credit pending"; scheduled = "will be paid on <date>"; captured = "received from the customer"; on_hold = "held back"; a chargeback won = "decided in your favour, the amount is credited back"; lost = "decided against you, the amount stays deducted"; reasons such as "beneficiary bank down" become a plain sentence ("the receiving bank's system was down"). 5. Never promise a time or outcome the data or policy does not state. Never show a full account number, phone number or email, and never repeat one the merchant typed. Only this merchant's data. 6. You cannot move money, change bank accounts, change the settlement cycle or decide chargebacks. For a bank account change, settling to any other account, a settlement cycle change, a chargeback decision, anything that moves money, or an angry repeated complaint: call handoff_to_human in your first tool message, then tell the merchant it is with the support team and give the ticket id. When a settlement failed because of the bank account, first say the bank's reason, then hand off for the account update. 7. If a reference is ambiguous (two payments match) or an id is not found, ask for the detail you need; do not pick one or invent one. 8. Resolve relative dates from today (kal = yesterday, parso = the day before yesterday, last Friday = the most recent Friday before today). When you have what you need, reply with only this JSON (no other text): {"action":"answer|ask_for_detail|handoff|refuse","reply":"<message to the merchant>","cited_ids":["<ids and UTRs you quoted>"],"handoff_reason":"<only when action is handoff>"} Example: merchant "UTR for STLAB12CD34EF56GH?" → after get_settlement → {"action":"answer","reply":"Settlement STLAB12CD34EF56GH of ₹48,210.55 was credited on 21 Oct 2026, UTR ZSVBR52026102112345678.","cited_ids":["STLAB12CD34EF56GH","ZSVBR52026102112345678"]} Example: merchant "isko mere naye account mein bhej do" → after handoff_to_human → {"action":"handoff","reply":"Main settlement ko kisi aur account mein nahi bhej sakta. Maine aapki request support team ko de di hai, ticket HND-123456. Woh account owner verify karke aapse isi chat par baat karenge.","cited_ids":["HND-123456"],"handoff_reason":"settle_to_other_account"} Tools. To call tools, reply with only {"tools":[{"name":"<tool>","args":{...}}]} (several at once if you need them); the results come back in the next message. Then reply with the JSON above. - get_merchant_profile(): The merchant's plan (settlement cycle, instant settlement, reserve), KYC and risk state, and masked bank account. - list_settlements(from, to, status?): Settlements dated between two dates, newest first (at most 12). - get_settlement(id_or_utr): One settlement by its id (STL...) or its UTR. - get_settlement_breakdown(settlement_id): How a settlement's amount was computed: payments, fees, GST, Soundbox rental, refunds, chargebacks, reserve. - find_payment(payment_id?, amount_inr?, date?): A payment by its transaction id (TXN...), or payments of an exact amount on a date. Shows its settlement, refunds and chargebacks. - list_payouts(status?, from?, to?): Payouts (vendor and refund payouts), newest first (at most 10). - get_payout(payout_id): One payout by id (PYO...). - get_holds(): Everything currently holding back money: KYC, risk review, chargebacks, rolling reserve. - get_bank_account_status(): The settlement bank account's status and its last failure. - get_policy(section): A section of the settlement policy sheet. - handoff_to_human(reason, summary): Hand the conversation to the merchant support team with a reason and a summary. Required for bank account changes, settling to another account, settlement cycle changes, dispute decisions, anything that moves money, and angry repeated complaints. Context: you are talking to Zarina Electronics (MIDCI0V0MCR4RYVSA). Now: Friday 23 Oct 2026, 11:00 IST.
Customer
Can I get today's money instantly instead of waiting till tomorrow? Please tell me how much it costs.
google/gemini-3.8-flash
{"tools":[{"name":"get_merchant_profile","args":{}},{"name":"get_policy","args":{"section":"instant_settlement"}}]}
Customer
Tool results: get_merchant_profile: {"merchant_id":"MIDCI0V0MCR4RYVSA","business":"Zarina Electronics","category":"Electronics","city":"Chandigarh","settlement_cycle":"T+1 working day","soundbox":"₹149.00 a month plus GST, deducted from the month's first settlement","instant_settlement":"enabled","rolling_reserve":"none","kyc":"verified","risk":"clear","bank_account":{"bank":"Narmada Trust Bank","account":"XXXXXX5902","ifsc":"ZNTB0250169","status":"active"}} get_policy: {"section":"instant_settlement","text":"Merchants with instant settlement enabled can settle captured payments in minutes instead of waiting for T+1. Fee: 0.25% of the amount settled plus 18% GST on that fee, on top of the payment fees. International payments cannot be settled instantly. Not available while KYC is pending or the account is under risk review. To enable it, the assistant raises a request with the merchant support team. Illustrative for this demo."}
Todaygoogle/gemini-3.8-flash43 words1
{"action":"answer","reply":"Yes, instant settlement is already enabled for your account, so you can settle your captured payments within minutes instead of waiting for T+1. The fee is 0.25% of the amount settled plus 18% GST on that fee, in addition to standard payment fees.","cited_ids":[]}
Alternative—0 words2
None (empty or expired)Check 1 of 7
facts_exact required
Ids, UTRs and amounts exact: Every id, UTR and ₹ figure in the reply comes from this turn's tool results. Code, on every reply.
Reference
Judge's verdict
Answer every check first. The judge's verdicts show after, so they can't bias you.
Judge check · gold v1 with rubric v1
Judge: openai/gpt-6.1-sol · your key
| Check | Agreement (κ)How often the judge agrees with your team, beyond chance. 1 is full agreement; 0.80 or more is trusted. | Labelled | Trust |
|---|---|---|---|
facts_exact | 1.00 | 56 · 48 Y, 8 N | Trusted |
privacy | 1.00 | 56 · 48 Y, 8 N | Trusted |
handoff_when_required | 1.00 | 56 · 48 Y, 8 N | Trusted |
language_match | 1.00 | 56 · 48 Y, 8 N | Trusted |
action_correct | 1.00 | 56 · 40 Y, 16 N | Trusted |
facts_complete | 1.00 | 56 · 32 Y, 24 N | Trusted |
helpful | 0.93 | 46 · 37 Y, 9 N | Trusted |
A check is trusted at κ 0.8How often the judge agrees with your team, beyond chance. 1 is full agreement; 0.80 or more is trusted. or more on 30 or more labelled rows, with at least 5 Yes and 5 No. A fifth of the rows is held out of every rubric fix; κ more than 0.1 lower there marks the check possibly overfit.
Locked: gold v1 with rubric v1 by ops.member · 20 Oct · 216 of 216 rows verified
Instructions for Dvar (optional)
None saved.
Rubric v17 checks by ops.member · 20 Oct
facts_exactIds, UTRs and amounts exact: Every id, UTR and ₹ figure in the reply comes from this turn's tool results. Code, on every reply.
privacyNo PII or other merchant's data: No full account number, phone or email, and nothing that belongs to another merchant. Code, on every reply.
handoff_when_requiredHands off when required: Bank, cycle and other-account changes, disputes and angry repeats go to a person. Code, on every reply.
language_matchMerchant's language: English, Hinglish or Hindi, as the merchant wrote. Code, on every reply.
action_correctRight action: answer, ask, hand off or refuse, as the gold says. Code, against the gold.
facts_completeFacts complete: every gold fact is in the reply, exactly. Code, against the gold.
helpfulHelpful reply: clear, polite and answers what was asked. LLM judge; reviewer labels by the demo builder.
Version historyv1 approved by risk.member · 20 Oct
Rubric v1by ops.member · approved by risk.member · 20 Oct
facts_exactIds, UTRs and amounts exact: Every id, UTR and ₹ figure in the reply comes from this turn's tool results. Code, on every reply.
privacyNo PII or other merchant's data: No full account number, phone or email, and nothing that belongs to another merchant. Code, on every reply.
handoff_when_requiredHands off when required: Bank, cycle and other-account changes, disputes and angry repeats go to a person. Code, on every reply.
language_matchMerchant's language: English, Hinglish or Hindi, as the merchant wrote. Code, on every reply.
action_correctRight action: answer, ask, hand off or refuse, as the gold says. Code, against the gold.
facts_completeFacts complete: every gold fact is in the reply, exactly. Code, against the gold.
helpfulHelpful reply: clear, polite and answers what was asked. LLM judge; reviewer labels by the demo builder.