Skip to main content
Loading navigation
Back to challenges

Claude Agents SDK Industry Loss Warranty Index Validator

ILW traders need automated verification of third-party PCS loss aggregation figures to execute settlement terms. Create a Claude Agents SDK tool-using agent that validates industry loss index updates against historical catastrophe benchmark thresholds.

Machine LearningHosted by Vera
Sign in free to start

What you are building

Orchestrate a Claude agent using extended thinking to analyze industry loss report updates and trigger ILW payouts.

How work is evaluated

Evaluation

Evaluates index parsing accuracy and correct trigger declaration for ILW settlements.

Datasets

Public datasets, plus the private datasets you have access to.

How submissions are scored

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

ilw_trigger_test

Ensures ILW trigger fires when industry loss exceeds threshold

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

index_extraction_accuracy

Accuracy of parsed industry loss numerical value • target: 1 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

  • Develop a Claude agent with tools to parse Property Claim Services (PCS) industry loss releases

  • Verify county-level aggregation numbers for spatial consistency

  • Execute Industry Loss Warranty (ILW) contract settlement routines upon threshold breach

Resources

Dataset notes

PCS industry loss bulletins detailing state-by-state catastrophe damage estimates.

Challenge setup

Evaluates index parsing accuracy and correct trigger declaration for ILW settlements.

Challenge input

JSON containing pcs_report_text and ilw_warranty_threshold_usd

Claude Agents SDK

Provides extended thinking and tool calling capabilities for index validation

Evaluated output

JSON with verified_industry_loss, ilw_triggered (bool), payout_status

Evaluation checks
  • Ensures ILW trigger fires when industry loss exceeds threshold
  • Accuracy of parsed industry loss numerical value • target: 1 • range: 0-1
Configured evaluation evidence
  • Benchmark: ILW-Audit-Bench
  • Index Extraction Accuracy target: 1
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Shows the first task, up to three tools, and up to three evidence items.

View technical recipe

Configured tools

Action Space
  • 01.AI · Required
  • Hugging Face · Optional
  • Ogre.run · Optional

Evaluation contract

  • ilw_trigger_test · Weight 1
  • index_extraction_accuracy · Weight 1

Metrics

  • index_extraction_accuracy · Target 1

Benchmarks

  • ILW-Audit-Bench
  • general

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: JSON containing pcs_report_text and ilw_warranty_threshold_usd

    Output: JSON with verified_industry_loss, ilw_triggered (bool), payout_status

    Execution harness: Python

Evaluation tools

  • Claude Agents SDK · Agent Framework

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Claude Agents SDK Industry Loss Warranty Index Validator