Skip to main content
Back to challenges
Challenge

OpenAI Agents SDK Threat Telemetry & Cyber Loss Payout Triage System

Following large-scale cyber attacks on financial institutions like Apollo Global, immediate threat telemetry validation is required for cyber claim payouts. Build an autonomous agent system using OpenAI Agents SDK to correlate firewall logs, attack timestamps, and policy conditions for claim validation.

Machine LearningHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Deploy an OpenAI Agents SDK autonomous agent to triage financial cyber breach claims against raw system logs.

Delivery guide

How work is evaluated

Evaluation

Evaluates accuracy of breach verification and automated policy coverage determinations.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

payout_limit_check

Ensures recommended payout does not exceed policy limit

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

triage_accuracy

Classification accuracy of claim validity • target: 0.95 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Build OpenAI Agents SDK autonomous agents with function calling tool integration

  • Parse and summarize network intrusion logs (JSON/PCAP summaries) automatically

  • Cross-reference intrusion vectors against policy cyber extortion and loss of business income endorsements

  • Measure validation precision and agent execution step efficiency

Resources and assets

Reference links and supporting material

Dataset notes

Synthetic financial institution cyber incident log dataset paired with commercial cyber policies.

Challenge setup

Evaluates accuracy of breach verification and automated policy coverage determinations.

Challenge input

JSON containing claim submission, breach log summary, and policy clauses.

OpenAI Agents SDK

Autonomous agent execution with function calling and handoffs

Evaluated output

JSON with claim_valid boolean, breach_type string, and recommended_payout_usd.

Evaluation checks
  • Ensures recommended payout does not exceed policy limit
  • Classification accuracy of claim validity • target: 0.95 • range: 0-1
Configured evaluation evidence
  • Benchmark: CyberClaimTriageBench
  • Triage Accuracy target: 0.95
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

Action Space
  • OpenAI · Required
  • Aim · Optional
  • Triton Inference Server · Optional

Evaluation contract

  • payout_limit_check · Weight 1
  • triage_accuracy · Weight 1

Metrics

  • triage_accuracy · Target 0.95

Benchmarks

  • CyberClaimTriageBench
  • security

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: JSON containing claim submission, breach log summary, and policy clauses.

    Output: JSON with claim_valid boolean, breach_type string, and recommended_payout_usd.

    Execution harness: Python

Evaluation tools

  • OpenAI Agents SDK · Agent Engine

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about OpenAI Agents SDK Threat Telemetry & Cyber Loss Payout Triage System