Skip to main content
Back to challenges
Challenge

Transaction Compliance Agent with OpenAI Agents SDK and Model Context Protocol (MCP)

In light of New York's lawsuit against Kalshi regarding the classification of 'illegal gambling' versus 'hedging', financial institutions must now differentiate between complex financial instruments and prohibited activities with extreme precision. This challenge requires you to build an autonomous Compliance Agent using the OpenAI Agents SDK and the Model Context Protocol (MCP). Your agent will act as a real-time transaction classifier. It must use MCP to connect to live regulatory knowledge bases (simulated) and legal statutes to evaluate if a transaction (e.g., a prediction market trade, a tokenized deposit via Project Agorá, or a corporate USD debit card swipe in Egypt) complies with local jurisdiction laws. The agent must handle multi-turn conversations with a mock 'Operations Manager' to ask for missing transaction metadata before making a final compliance determination.

Cybersecurity & PrivacyHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Develop an autonomous compliance classifier using OpenAI Agents SDK and MCP to distinguish between legal hedging and illegal gambling transactions.

Delivery guide

How work is evaluated

Evaluation

Evaluation of the agent's accuracy in classifying 'Edge Cases' where intent is ambiguous.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

MCP Connection Test

Ensures the agent attempted to query the MCP server before deciding.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

Classification F1-Score

Balance of precision and recall in detecting illegal operations. • target: 0.88 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Implement autonomous function calling using the OpenAI Agents SDK

  • Establish secure data context connections using Model Context Protocol (MCP)

  • Build a classification engine for high-risk financial instruments

  • Handle multi-turn agentic dialogs for metadata enrichment

Resources and assets

Reference links and supporting material

Dataset notes

Synthetic dataset of mixed financial activities including agricultural hedging, political betting, and tokenized commodity swaps.

Challenge setup

Evaluation of the agent's accuracy in classifying 'Edge Cases' where intent is ambiguous.

Challenge input

CSV file with columns: tx_id, description, amount, user_location, asset_type.

OpenAI Agents SDK

Official framework for autonomous GPT agents.

Model Context Protocol

Allows agent to ingest complex regulatory statutes without bloating the prompt.

o4-mini

Cost-effective, fast reasoning for real-time classification.

Evaluated output

JSON with tx_id and classification_result.

Evaluation checks
  • Ensures the agent attempted to query the MCP server before deciding.
  • Balance of precision and recall in detecting illegal operations. • target: 0.88 • range: 0-1
Configured evaluation evidence
  • Benchmark: RegTech-Intent-2026
  • Classification F1 Score target: 0.88
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

Action Space
  • OpenAI · Required
  • Azure OpenAI Service · Optional
  • OpenAI o3 · Optional

Evaluation contract

  • MCP Connection Test · Weight 1
  • Classification F1-Score · Weight 1

Metrics

  • Classification F1-Score · Target 0.88

Benchmarks

  • RegTech-Intent-2026
  • security

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: CSV file with columns: tx_id, description, amount, user_location, asset_type.

    Output: JSON with tx_id and classification_result.

    Execution harness: Python

Evaluation tools

  • OpenAI Agents SDK · Orchestration
  • Model Context Protocol · Data Connectivity
  • o4-mini · Model

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Transaction Compliance Agent with OpenAI Agents SDK and Model Context Protocol (MCP)