Research / Insurance benchmarks

Claims Bench

400 tasks testing what decision models can verify from an insurance claim file.

Applied World
400Verification tasks
80Property claim files
7Claim workflows
4Decision models

Claims Bench now covers 400 statements across 80 authored California property claim files. We expanded the evidence packets and added investigation, recovery, handling, and disputes to test more of the decisions a claims team faces.

Model leaderboard

Liquid d1-ultra matched the proposed key on 365 of 400 tasks, two ahead of Mercury Decide. Jev matched 357 and Luna matched 334. Each model received the same statements, evidence, and label definitions.

Model agreement and cost
Agreement with proposed keys (%)
80
84
88
92
96
$0.00
$0.40
$0.80
$1.20
$1.60
USD per 1,000 statement checks
Open circle: estimated uncached input cost for Luna. Filled circles: API-reported usage. Costs scaled from 400 requests.
ModelMatchesAgreementCost / taskMean API latency
Liquid d1-ultra365/40091.25%$0.0005981.037 s
Mercury Decide363/40090.75%$0.0002991.742 s
Jev 1.14357/40089.25%$0.0006410.671 s
GPT-6 Luna Decisions334/40083.50%≈$0.0013900.563 s

Inside a claim file

A quotation is not a payment. A request to copy a claimant is not proof that a message was sent. Claims Bench asks models to preserve these distinctions using the complete supplied record at a specified cutoff date.

From our collection of thousands of claim files, we used 80 for 400 verification tasks with four decision models. Each file contributes five tasks, with 1,079 source records across the selected files. Policies, estimates, invoices, correspondence, handling diaries, and written conversations preserve the dates, amounts, revisions, and unresolved questions that matter to each decision.

AnswerMeaningTasks
SupportedThe record establishes the whole statement.148
ContradictedThe record establishes that it is false.131
Insufficient evidenceThe record establishes neither.121
Always predicting the most common label gives 37% agreement.

The claim files include text records, financial tables, and conversation transcripts, with photos available for review.

Seven workflows

Workflowd1-ultraMercuryJevLuna
Intake / loss facts88.57%62/7094.29%66/7087.14%61/7085.71%60/70
Policy information93.33%56/6093.33%56/6095.00%57/6090.00%54/60
Damage / costs91.43%64/7090.00%63/7091.43%64/7084.29%59/70
Investigation92.00%46/5088.00%44/5090.00%45/5080.00%40/50
Recovery94.00%47/5090.00%45/5092.00%46/5076.00%38/50
Handling88.00%44/5088.00%44/5084.00%42/5084.00%42/50
Disputes92.00%46/5090.00%45/5084.00%42/5082.00%41/50
Exact-label agreement across the same 400 tasks. Category counts determine their weight in the overall score.

What changes across four attempts?

Pass@1 measures success on a single attempt. Pass@4 measures success at least once within four attempts. All-four-correct measures consistency across every attempt.

ModelMean pass@1Pass@4All four correct
Liquid d1-ultra91.62%93.00%89.50%
Mercury Decide88.50%90.50%86.00%
Jev 1.1486.88%90.50%84.00%
GPT-6 Luna Decisions80.50%80.50%80.50%
Additional 200 tasks only. One d1-ultra failed request remains unsuccessful. Separate calls can produce correlated outputs; independent sampling is not established. Pass@4 requires a key to identify the successful answer.

Examples

The revised estimate

A first payment, a revised scope, and an amount still to be confirmed. Listen to a sample conversation between Claire, a homeowner, and Daniel, a claims handler.

Illustrative inspection photograph of water damage at a kitchen cabinet base and floor beside a dishwasher.
Sample conversation0:00 / 1:02

Based on a real conversation. Audio is AI-generated.

Separate costs, separate evidence

Source
Repair quotation · R200-T004 · March 21, 2026 snapshot
Statement
The repair quotation totals $2,586.72, excluding the separately billed emergency attendance.
Evidence
The quotation gives the repair total and directs emergency work to a separate contractor invoice.
Proposed answer
supported The repair total does not include the separately billed attendance.

Follow-up does not establish who received it

Source
Claimant conversation and handling diary · W200-T101 · June 24, 2026 snapshot
Statement
By June 24, the handler had carried out Palmer's May 21 request to copy her on renewed direct correspondence with the cabinet contractor about countertop handling.
Evidence
Palmer requested copies. The diary records contractor follow-up, but the correspondence and its recipients are not supplied.
Proposed answer
insufficient_evidence Follow-up alone does not establish that Palmer was copied.

How we scored it

We scored each answer against the benchmark key: one point for an exact task-ID and label match, zero otherwise. The key stays outside the model input. Saved requests, responses, and source references make each decision inspectable.

The headline combines 200 retained predictions from the revised original set with the first attempt on each of the 200 new tasks. Two original no-answer failures were recovered, so this is a selected-response comparison. The four-attempt table is a separate view of the new tasks.

The keys remain proposed, with eight flagged disagreements retained in the score. Five verification tasks share each claim packet. These results measure agreement on document verification, not end-to-end claims handling or expert-validated decisions. Changes from the earlier release cannot be attributed to longer packets alone.

Work with Applied World

If you are an enterprise claims team or a neo lab building decision agents, bring us the decisions and evidence your reviewers work with. We can build and review benchmarks around that workflow.

Email hello@appliedworld.ai