300 rows in gold v1 · 300 ready · 0 not ready
Keyboard shortcuts
Y yes · N no · 1 today · 2 alternative · E own · F flag · Enter save · J/K next or previous row · U undoOnly a member labels. You can read the rows and the result.
Complaint or escalation · 12 labelled, 3 needed · picked automatically, the topic furthest behind · en
Your earlier answers aren't shown (someone labelled this row after you). Answer again and pick the reference to update them.
System prompt · 1 turn · no tools · max 2048 tokens · temperature 0
System You triage customer servicing messages for an Indian financial-services group that offers personal and business loans, housing loans, life insurance and health insurance. A message can contain one, two or three separate requests. Tag every request the customer actually makes. Do not tag something the customer says they do NOT want or have already resolved. Return ONLY a JSON object: {"intents": [...], "product_lines": [...], "urgency": [...]} intents (one or more): - emi_payment_status: loan EMI paid but not reflecting, EMI due amount or date, EMI bounce or late-payment charges, overdue amount - foreclosure_prepayment: loan foreclosure quote or process, part-prepayment, closure charges - statement_certificate: loan account statement, repayment schedule, interest certificate (provisional or final), NOC or closure letter - mandate_change: change or re-register NACH or e-NACH or auto-debit bank account, cancel a mandate, change the EMI debit date - premium_renewal: insurance premium due date or amount, renewal, grace period, revival of a lapsed policy, premium paid but not reflecting - claim_status: insurance claim intimation or status, pending claim documents, cashless pre-authorisation status, settlement amount - policy_change: nominee change, duplicate or soft copy of policy document, address or details change for a specific policy - kyc_contact_update: update mobile, email or address on the customer record, PAN or Aadhaar re-KYC, name correction - complaint_escalation: explicit complaint, request to escalate, issue raised repeatedly and still unresolved, threat to go to the ombudsman, RBI, IRDAI or consumer court, harassment by an agent. An angry tone, or saying they have asked several times, is not a complaint on its own; it needs an explicit complaint, a request to escalate (manager, nodal or grievance officer), a regulator or court threat, or harassment. - fraud_unauthorised: loan or policy taken in the customer's name without consent, unauthorised debit, OTP or details shared with a fraudster, phishing call or link product_lines (one or more; the product the request is about, even if the customer uses the wrong word for it): personal_business_loan, housing_loan, life_insurance, health_insurance urgency (zero or more): - fraud: whenever fraud_unauthorised is tagged - complaint_escalation: whenever complaint_escalation is tagged - hospitalisation_claim: a health claim where the patient is in hospital now or admission is planned (cashless, pre-authorisation, discharge) Use [] when none apply. Return only the JSON object, no explanation.
Customer
I have raised the same issue three times already and still no resolution. This is completely unacceptable. Escalate this matter to your senior management immediately! Loan A/c XXXX5660.
Todaygoogle/gemini-4-argon1 word1
{"intents":["complaint_escalation"],"product_lines":["personal_business_loan"],"urgency":["complaint_escalation"]}
Alternative—0 words2
None (empty or expired)Check 1 of 5
request_types_correct counts ×2
Every request type in the message is tagged, and nothing else.
Reference
Judge's verdict
Answer every check first. The judge's verdicts show after, so they can't bias you.
Judge check · gold v1 with rubric v1
Judge: anthropic/claude-sonnet-5.5 · your key
| Check | Agreement (κ)How often the judge agrees with your team, beyond chance. 1 is full agreement; 0.80 or more is trusted. | Labelled | Trust |
|---|---|---|---|
request_types_correct | 0.44 | 300 · 297 Y, 3 N | Needs 2 more labels |
no_fraud_missed | — | 300 · 300 Y, 0 N | Needs 5 more labels |
no_complaint_missed | — | 300 · 300 Y, 0 N | Needs 5 more labels |
urgency_correct | — | 300 · 300 Y, 0 N | Needs 5 more labels |
product_correct | 0.70 | 300 · 251 Y, 49 N | Possibly overfit · κ 0.53 on held-out rows, 0.74 on the rest |
A check is trusted at κ 0.8How often the judge agrees with your team, beyond chance. 1 is full agreement; 0.80 or more is trusted. or more on 30 or more labelled rows, with at least 5 Yes and 5 No. A fifth of the rows is held out of every rubric fix; κ more than 0.1 lower there marks the check possibly overfit.
Locked: gold v1 with rubric v1 by ops.member · 20 Oct · 300 of 300 rows verified
Instructions for Dvar (optional)
None saved.
Rubric v15 checks by ops.member · 20 Oct
request_types_correctEvery request type in the message is tagged, and nothing else.
no_fraud_missedA message about fraud or an unauthorised transaction is flagged as fraud.
no_complaint_missedA complaint or escalation is flagged as a complaint.
urgency_correctThe urgency flags match the answer key exactly.
product_correctThe product lines match the answer key exactly.
Version historyv1 approved by risk.member · 20 Oct
Rubric v1by ops.member · approved by risk.member · 20 Oct
request_types_correctEvery request type in the message is tagged, and nothing else.
no_fraud_missedA message about fraud or an unauthorised transaction is flagged as fraud.
no_complaint_missedA complaint or escalation is flagged as a complaint.
urgency_correctThe urgency flags match the answer key exactly.
product_correctThe product lines match the answer key exactly.