Skip to main content
Loading navigation
Back to challenges

OpenAI Agents SDK Threat Telemetry & Cyber Loss Payout Triage System

Following large-scale cyber attacks on financial institutions like Apollo Global, immediate threat telemetry validation is required for cyber claim payouts. Build an autonomous agent system using OpenAI Agents SDK to correlate firewall logs, attack timestamps, and policy conditions for claim validation.

Machine LearningHosted by Vera
Sign in free to start

What you are building

Deploy an OpenAI Agents SDK autonomous agent to triage financial cyber breach claims against raw system logs.

How work is evaluated

Evaluation

Evaluates accuracy of breach verification and automated policy coverage determinations.

Datasets

Public datasets, plus the private datasets you have access to.

How submissions are scored

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

payout_limit_check

Ensures recommended payout does not exceed policy limit

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

triage_accuracy

Classification accuracy of claim validity • target: 0.95 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

  • Build OpenAI Agents SDK autonomous agents with function calling tool integration

  • Parse and summarize network intrusion logs (JSON/PCAP summaries) automatically

  • Cross-reference intrusion vectors against policy cyber extortion and loss of business income endorsements

  • Measure validation precision and agent execution step efficiency

Resources

Dataset notes

Synthetic financial institution cyber incident log dataset paired with commercial cyber policies.

Challenge setup

Evaluates accuracy of breach verification and automated policy coverage determinations.

Challenge input

JSON containing claim submission, breach log summary, and policy clauses.

OpenAI Agents SDK

Autonomous agent execution with function calling and handoffs

Evaluated output

JSON with claim_valid boolean, breach_type string, and recommended_payout_usd.

Evaluation checks
  • Ensures recommended payout does not exceed policy limit
  • Classification accuracy of claim validity • target: 0.95 • range: 0-1
Configured evaluation evidence
  • Benchmark: CyberClaimTriageBench
  • Triage Accuracy target: 0.95
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Shows the first task, up to three tools, and up to three evidence items.

View technical recipe

Configured tools

Action Space
  • OpenAI · Required
  • Aim · Optional
  • Triton Inference Server · Optional

Evaluation contract

  • payout_limit_check · Weight 1
  • triage_accuracy · Weight 1

Metrics

  • triage_accuracy · Target 0.95

Benchmarks

  • CyberClaimTriageBench
  • security

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: JSON containing claim submission, breach log summary, and policy clauses.

    Output: JSON with claim_valid boolean, breach_type string, and recommended_payout_usd.

    Execution harness: Python

Evaluation tools

  • OpenAI Agents SDK · Agent Engine

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about OpenAI Agents SDK Threat Telemetry & Cyber Loss Payout Triage System