Public evaluation

LegalComplianceQuery

The evaluation module will rigorously assess the OpenAI Agents SDK agent's ability to provide accurate, well-justified, and compliant tax and legal advice based on a simulated MCP. Emphasis will be on correctness, completeness, and citation of regulations.

Evaluation type

task based

Challenge

Global Tax & Legal Compliance Advisor Agent

Difficulty

Advanced

Rigor

Unspecified

Open linked challenge

Evaluation overview

How the linked challenge is judged: tasks, benchmarks, and criteria count.

Tasks

Benchmarks

Criteria

Task templates

Inputs and expected outputs.

Task 1

LegalComplianceQuery

Evaluate the agent's ability to answer a specific legal/tax compliance question for a given country.

Input format

{"query": "string", "country": "string", "context": "string"}

Output format

{"advice": "string", "is_compliant": "boolean", "citations": ["string"]}

Task 2

PolicyJustificationCheck

Evaluate if the agent can correctly identify a compliance issue and justify it with regulations.

Input format

{"scenario": "string", "country": "string"}

Output format

{"compliance_issue_detected": "boolean", "issue_description": "string", "violating_regulations": ["string"]}