Back to challenges
Challenge

Substation Procurement Risk Agent with OpenAI Agents SDK

Riyadh Expo infrastructure expansion requires real-time monitoring of electrical substation supply chains and equipment lead times. Build an autonomous agent using OpenAI Agents SDK and Hamming to audit supplier bids and issue risk alerts. Achieve an F1-score above 0.90 in flagging critical delivery delays.

Business OperationsHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Utilize OpenAI Agents SDK with function calling to evaluate electrical substation vendor proposals against project schedules.

Delivery guide

How work is evaluated

Evaluation

Evaluates procurement risk identification precision and recall on synthetic vendor proposals.

Datasets

Shared data for this challenge

Review public datasets and any private uploads tied to your build.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

tool_call_validity

Agent correctly invokes schedule lookup tools without hallucinated arguments

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

delay_detection_f1

F1 score for accurately predicting procurement schedule violations • target: 0.9 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Build dynamic multi-tool workflows with OpenAI Agents SDK

  • Integrate Hamming for continuous prompt and output evaluation

  • Analyze transformer lead times, procurement bottlenecks, and vendor reliability

  • Generate automated compliance reports for Riyadh Expo project managers

Resources and assets

Reference links and supporting material

Dataset notes

Dataset of 100 historical procurement proposals for high-voltage substation components in Saudi Arabia including lead times and audit findings.

How this agent runs

Evaluates procurement risk identification precision and recall on synthetic vendor proposals.

Challenge input

JSON payload containing vendor profile, lead time, and technical specs

OpenAI Agents SDK

Provides robust multi-agent primitives and tool orchestration for Python applications.

Hamming

Automates LLM task metrics tracking and systematic prompt evaluations.

Evaluated output

JSON payload containing risk score, flagged anomalies, and approval recommendation

Checks for
  • Agent correctly invokes schedule lookup tools without hallucinated arguments
  • F1 score for accurately predicting procurement schedule violations • target: 0.9 • range: 0-1
Proof of success
  • Benchmark: ProcureAgentEval
  • Delay Detection F1 target: 0.9
  • 1 public reference case
Runtime evidence
  • Python execution harness
  • Python sandbox (unavailable on Versalist)
View technical recipe

Configured tools

Action Space
  • OpenAI · Required
  • Hamming · Optional
  • Martian · Optional

Evaluation contract

  • tool_call_validity · Weight 1
  • delay_detection_f1 · Weight 1

Recipe state

This is a preview. The configuration can change before the evaluation recipe is locked.

Frequently Asked Questions about Substation Procurement Risk Agent with OpenAI Agents SDK