Skip to main content
Back to challenges
Challenge

Jafurah Gas Field Expansion Risk Evaluation with Pydantic AI

Evaluating contractor bids for Saudi Aramco's Jafurah fifth expansion phase requires type-safe extraction of reservoir pressure ratings and execution timelines. Build a Pydantic AI agent that enforces strong type hints and Pydantic schema validation for bid technical submittals. Achieve 100% type validation compliance with zero unhandled schema errors.

Business OperationsHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Develop a type-safe Pydantic AI evaluation agent for Jafurah gas expansion contractor bids.

Delivery guide

How work is evaluated

Evaluation

Evaluates Pydantic AI agent schema enforcement and bid evaluation accuracy.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

Schema Validation Check

Verifies that output passes Pydantic data type validation without raising ValidationError.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

Schema Parsing Success Rate

Percentage of agent evaluations successfully parsed into Pydantic models. • target: 1 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Implement Pydantic AI `Agent` with structured response validation

  • Validate technical drilling parameters against Saudi Aramco safety specs

  • Inject external dependency tools for live cost-index lookups

  • Generate clean JSON evaluation reports guaranteed to match target schema

Resources and assets

Reference links and supporting material

Dataset notes

Contractor bid proposals narrative text for Saudi Aramco Jafurah unconventional gas field expansion.

Challenge setup

Evaluates Pydantic AI agent schema enforcement and bid evaluation accuracy.

Challenge input

Text prompt containing contractor proposal narrative

Pydantic AI

Assigned framework for structured validation.

Evaluated output

Validated JSON object adhering to JafurahBidEval Pydantic model

Evaluation checks
  • Verifies that output passes Pydantic data type validation without raising ValidationError.
  • Percentage of agent evaluations successfully parsed into Pydantic models. • target: 1 • range: 0-1
Configured evaluation evidence
  • Benchmark: PydanticBench
  • Schema Parsing Success Rate target: 100%
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

Action Space
  • Pydantic AI · Required
  • AutoML (H2O) · Optional
  • All Hands AI · Optional
Orchestration
  • Pydantic AI · Required

Evaluation contract

  • Schema Validation Check · Weight 1
  • Schema Parsing Success Rate · Weight 1

Metrics

  • Schema Parsing Success Rate · Target 1

Benchmarks

  • PydanticBench
  • general

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: Text prompt containing contractor proposal narrative

    Output: Validated JSON object adhering to JafurahBidEval Pydantic model

    Execution harness: Python

Evaluation tools

  • Pydantic AI · Type-Safe Agent Framework

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Jafurah Gas Field Expansion Risk Evaluation with Pydantic AI