OpenAI Agents Litigation Engine: Nuclear Verdict Risk Predictor
Surging litigation costs and social inflation create unexpected catastrophic losses for casualty insurers across civil jurisdictions. Build an OpenAI Agents SDK system to extract legal case factors, predict nuclear verdict probability, and maintain a Brier score below 0.12 on liability claims.
What you are building
The core problem, expected build, and operating context for this challenge.
Build an agent using the OpenAI Agents SDK to analyze casualty claim dockets and estimate high-severity litigation probability.
How work is evaluated
Evaluates nuclear verdict probability estimation calibration and accuracy.
Shared data for this challenge
Review public datasets and any private uploads tied to your build.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
brier_score_threshold
Ensures prediction Brier score is below 0.12.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
brier_score
Brier score measuring probability calibration accuracy. • target: 0.1 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Implement autonomous agent reasoning loop using OpenAI Agents SDK.
Extract litigation risk factors (venue jurisdiction, plaintiff counsel profile, injury severity).
Predict probability of a nuclear verdict (> $10M liability award).
Achieve a calibrated Brier score under 0.12 across test cases.
Reference links and supporting material
Dataset of 300 commercial liability civil court cases with actual final jury award values.
How this agent runs
Evaluates nuclear verdict probability estimation calibration and accuracy.
Challenge input
JSON containing case_id, venue_county, injury_type, defense_reserve_usd
OpenAI Agents SDK
Official OpenAI SDK for autonomous multi-tool agent orchestration.
Evaluated output
JSON containing nuclear_verdict_probability, key_drivers, recommended_reserve_usd
- Ensures prediction Brier score is below 0.12.
- Brier score measuring probability calibration accuracy. • target: 0.1 • range: 0-1
- Benchmark: Legal Risk Calibration Benchmark
- Brier Score target: 0.1
- 1 public reference case
- Python execution harness
- Python sandbox (unavailable on Versalist)
View technical recipe
Configured tools
- OpenAI · Required
- Llama 3.3 70B · Optional
- Azure OpenAI Service · Optional
Evaluation contract
- brier_score_threshold · Weight 1
- brier_score · Weight 1
Recipe state
This is a preview. The configuration can change before the evaluation recipe is locked.