Skip to main content
Back to challenges
Challenge

Claude Agents SDK Industry Loss Warranty Index Validator

ILW traders need automated verification of third-party PCS loss aggregation figures to execute settlement terms. Create a Claude Agents SDK tool-using agent that validates industry loss index updates against historical catastrophe benchmark thresholds.

Machine LearningHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Orchestrate a Claude agent using extended thinking to analyze industry loss report updates and trigger ILW payouts.

Delivery guide

How work is evaluated

Evaluation

Evaluates index parsing accuracy and correct trigger declaration for ILW settlements.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

ilw_trigger_test

Ensures ILW trigger fires when industry loss exceeds threshold

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

index_extraction_accuracy

Accuracy of parsed industry loss numerical value • target: 1 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Develop a Claude agent with tools to parse Property Claim Services (PCS) industry loss releases

  • Verify county-level aggregation numbers for spatial consistency

  • Execute Industry Loss Warranty (ILW) contract settlement routines upon threshold breach

Resources and assets

Reference links and supporting material

Dataset notes

PCS industry loss bulletins detailing state-by-state catastrophe damage estimates.

Challenge setup

Evaluates index parsing accuracy and correct trigger declaration for ILW settlements.

Challenge input

JSON containing pcs_report_text and ilw_warranty_threshold_usd

Claude Agents SDK

Provides extended thinking and tool calling capabilities for index validation

Evaluated output

JSON with verified_industry_loss, ilw_triggered (bool), payout_status

Evaluation checks
  • Ensures ILW trigger fires when industry loss exceeds threshold
  • Accuracy of parsed industry loss numerical value • target: 1 • range: 0-1
Configured evaluation evidence
  • Benchmark: ILW-Audit-Bench
  • Index Extraction Accuracy target: 1
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

Action Space
  • 01.AI · Required
  • Hugging Face · Optional
  • Ogre.run · Optional

Evaluation contract

  • ilw_trigger_test · Weight 1
  • index_extraction_accuracy · Weight 1

Metrics

  • index_extraction_accuracy · Target 1

Benchmarks

  • ILW-Audit-Bench
  • general

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: JSON containing pcs_report_text and ilw_warranty_threshold_usd

    Output: JSON with verified_industry_loss, ilw_triggered (bool), payout_status

    Execution harness: Python

Evaluation tools

  • Claude Agents SDK · Agent Framework

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Claude Agents SDK Industry Loss Warranty Index Validator