Claims Bench now covers 400 statements across 80 authored California property claim files. We expanded the evidence packets and added investigation, recovery, handling, and disputes to test more of the decisions a claims team faces.
Model leaderboard
Liquid d1-ultra matched the proposed key on 365 of 400 tasks, two ahead of Mercury Decide. Jev matched 357 and Luna matched 334. Each model received the same statements, evidence, and label definitions.
| Model | Matches | Agreement | Cost / task | Mean API latency |
|---|---|---|---|---|
| Liquid d1-ultra | 365/400 | 91.25% | $0.000598 | 1.037 s |
| Mercury Decide | 363/400 | 90.75% | $0.000299 | 1.742 s |
| Jev 1.14 | 357/400 | 89.25% | $0.000641 | 0.671 s |
| GPT-6 Luna Decisions | 334/400 | 83.50% | ≈$0.001390 | 0.563 s |
Inside a claim file
A quotation is not a payment. A request to copy a claimant is not proof that a message was sent. Claims Bench asks models to preserve these distinctions using the complete supplied record at a specified cutoff date.
From our collection of thousands of claim files, we used 80 for 400 verification tasks with four decision models. Each file contributes five tasks, with 1,079 source records across the selected files. Policies, estimates, invoices, correspondence, handling diaries, and written conversations preserve the dates, amounts, revisions, and unresolved questions that matter to each decision.
| Answer | Meaning | Tasks |
|---|---|---|
| Supported | The record establishes the whole statement. | 148 |
| Contradicted | The record establishes that it is false. | 131 |
| Insufficient evidence | The record establishes neither. | 121 |
The claim files include text records, financial tables, and conversation transcripts, with photos available for review.
Seven workflows
| Workflow | d1-ultra | Mercury | Jev | Luna |
|---|---|---|---|---|
| Intake / loss facts | 88.57%62/70 | 94.29%66/70 | 87.14%61/70 | 85.71%60/70 |
| Policy information | 93.33%56/60 | 93.33%56/60 | 95.00%57/60 | 90.00%54/60 |
| Damage / costs | 91.43%64/70 | 90.00%63/70 | 91.43%64/70 | 84.29%59/70 |
| Investigation | 92.00%46/50 | 88.00%44/50 | 90.00%45/50 | 80.00%40/50 |
| Recovery | 94.00%47/50 | 90.00%45/50 | 92.00%46/50 | 76.00%38/50 |
| Handling | 88.00%44/50 | 88.00%44/50 | 84.00%42/50 | 84.00%42/50 |
| Disputes | 92.00%46/50 | 90.00%45/50 | 84.00%42/50 | 82.00%41/50 |
What changes across four attempts?
Pass@1 measures success on a single attempt. Pass@4 measures success at least once within four attempts. All-four-correct measures consistency across every attempt.
| Model | Mean pass@1 | Pass@4 | All four correct |
|---|---|---|---|
| Liquid d1-ultra | 91.62% | 93.00% | 89.50% |
| Mercury Decide | 88.50% | 90.50% | 86.00% |
| Jev 1.14 | 86.88% | 90.50% | 84.00% |
| GPT-6 Luna Decisions | 80.50% | 80.50% | 80.50% |
Examples
The revised estimate
A first payment, a revised scope, and an amount still to be confirmed. Listen to a sample conversation between Claire, a homeowner, and Daniel, a claims handler.

Based on a real conversation. Audio is AI-generated.
Separate costs, separate evidence
- Source
- Repair quotation · R200-T004 · March 21, 2026 snapshot
- Statement
- The repair quotation totals $2,586.72, excluding the separately billed emergency attendance.
- Evidence
- The quotation gives the repair total and directs emergency work to a separate contractor invoice.
- Proposed answer
supportedThe repair total does not include the separately billed attendance.
Follow-up does not establish who received it
- Source
- Claimant conversation and handling diary · W200-T101 · June 24, 2026 snapshot
- Statement
- By June 24, the handler had carried out Palmer's May 21 request to copy her on renewed direct correspondence with the cabinet contractor about countertop handling.
- Evidence
- Palmer requested copies. The diary records contractor follow-up, but the correspondence and its recipients are not supplied.
- Proposed answer
insufficient_evidenceFollow-up alone does not establish that Palmer was copied.
How we scored it
We scored each answer against the benchmark key: one point for an exact task-ID and label match, zero otherwise. The key stays outside the model input. Saved requests, responses, and source references make each decision inspectable.
The headline combines 200 retained predictions from the revised original set with the first attempt on each of the 200 new tasks. Two original no-answer failures were recovered, so this is a selected-response comparison. The four-attempt table is a separate view of the new tasks.
The keys remain proposed, with eight flagged disagreements retained in the score. Five verification tasks share each claim packet. These results measure agreement on document verification, not end-to-end claims handling or expert-validated decisions. Changes from the earlier release cannot be attributed to longer packets alone.
Work with Applied World
If you are an enterprise claims team or a neo lab building decision agents, bring us the decisions and evidence your reviewers work with. We can build and review benchmarks around that workflow.