Skip to main content
Loading navigation
Back to challenges

Type-Safe Climate Hazard Scoring Engine with Pydantic AI

Climate risk assessments require strict schema validation to prevent downstream rating errors. You will build a type-safe climate risk evaluation engine using Pydantic AI that ingests wildfire, flood, and extreme heat metrics for commercial assets. The system must enforce strong typing and output validated risk records with 100% schema compliance.

Machine LearningHosted by Vera
Sign in free to start

What you are building

Build a type-safe, validated geospatial climate hazard scoring engine using Pydantic AI.

How work is evaluated

Evaluation

Evaluates schema validation success rate and numeric accuracy of calculated combined climate hazard scores.

Datasets

Public datasets, plus the private datasets you have access to.

How submissions are scored

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

pydantic_validation_pass

Validates output adheres 100% to Pydantic schema without validation errors.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

schema_compliance_rate

Percentage of test outputs passing Pydantic validation • target: 1 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

  • Build strongly typed agents using Pydantic AI schema definitions.

  • Validate complex climate indicator inputs against custom Pydantic validators.

  • Produce risk outputs guaranteed to conform to structural schema types.

Resources

Dataset notes

Dataset of 500 spatial coordinates across North America with associated flood, storm surge, and wildfire risk attributes.

Challenge setup

Evaluates schema validation success rate and numeric accuracy of calculated combined climate hazard scores.

Challenge input

JSON object with latitude, longitude, flood_zone, wildfire_score, structural_value

Pydantic AI

Assigned framework for type-safe Python agent creation.

Evaluated output

JSON object conforming strictly to AssetRiskScore schema (composite_score, primary_hazard, error_bounds)

Evaluation checks
  • Validates output adheres 100% to Pydantic schema without validation errors.
  • Percentage of test outputs passing Pydantic validation • target: 1 • range: 0-1
Configured evaluation evidence
  • Benchmark: Type-Safe LLM Output Benchmark
  • Schema Compliance Rate target: 100%
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Shows the first task, up to three tools, and up to three evidence items.

View technical recipe

Configured tools

Action Space
  • Pydantic AI · Required
  • Llama 3.1 405B · Optional
Policy Serving
  • Together AI · Optional
Training Infra
  • Together AI · Optional
Orchestration
  • Pydantic AI · Required

Evaluation contract

  • pydantic_validation_pass · Weight 1
  • schema_compliance_rate · Weight 1

Metrics

  • schema_compliance_rate · Target 1

Benchmarks

  • Type-Safe LLM Output Benchmark
  • general

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: JSON object with latitude, longitude, flood_zone, wildfire_score, structural_value

    Output: JSON object conforming strictly to AssetRiskScore schema (composite_score, primary_hazard, error_bounds)

    Execution harness: Python

Evaluation tools

  • Pydantic AI · Type-Safe Agent Framework

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Type-Safe Climate Hazard Scoring Engine with Pydantic AI