Skip to main content
Back to challenges
Challenge

Global Tax & Legal Compliance Advisor Agent

This challenge focuses on developing a sophisticated legal and tax compliance advisor using the OpenAI Agents SDK. The agent will interpret complex regulatory texts, answer specific compliance queries for various jurisdictions, and justify its advice by citing relevant statutes. A core component will be the integration with a simulated MCP knowledge base, powered by Pinecone, to provide the agent with a vast, searchable repository of legal and tax documents. The challenge emphasizes advanced tool use, multi-LLM verification (using GPT-4o for primary analysis and Claude Opus 4.1 for cross-validation), and rigorous evaluation of accuracy and transparency.

Special Purpose AgentsHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Develop an autonomous agent using OpenAI Agents SDK and GPT-4o to navigate and advise on global tax and legal compliance, leveraging a simulated MCP knowledge base.

Delivery guide

How work is evaluated

Evaluation

The evaluation module will rigorously assess the OpenAI Agents SDK agent's ability to provide accurate, well-justified, and compliant tax and legal advice based on a simulated MCP. Emphasis will be on correctness, completeness, and citation of regulations.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
4 scoring checks
Binary
4 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

CorrectComplianceDecision

Agent's 'is_compliant' decision matches the expected outcome for known scenarios.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

CitationCount

Advice includes at least 2 relevant citations for complex queries.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 3

AdviceCompletenessScore

Expert-rated score for the completeness of the advice (1-5). • target: 4 • range: 1-5

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 4

ReasoningClarity

Expert-rated score for how clearly the agent justifies its advice (1-5). • target: 4 • range: 1-5

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Master the OpenAI Agents SDK's function calling and tool definition mechanisms to create robust interactions with external systems.

  • Design a simulated MCP knowledge base using Pinecone vector database to store and retrieve legal and tax documents, accessible via agent tools.

  • Develop custom Python tools for the agent to query, extract, and summarize relevant information from the Pinecone-backed MCP.

  • Implement a multi-LLM strategy where GPT-4o provides primary legal analysis and Claude Opus 4.1 acts as a secondary, independent verifier for critical compliance points.

  • Craft effective prompts for GPT-4o to ensure accurate interpretation of specific legal clauses and generation of precise compliance advice, citing relevant regulations.

  • Build an evaluation harness with Testaify to systematically test the agent's responses against a corpus of legal scenarios, measuring accuracy, completeness, and adherence to legal principles.

  • Implement mechanisms for the agent to explicitly state its reasoning and cite specific regulations to justify its compliance advice.

Resources and assets

Reference links and supporting material

Dataset notes

Sample data for 2 tasks

Challenge setup

The evaluation module will rigorously assess the OpenAI Agents SDK agent's ability to provide accurate, well-justified, and compliant tax and legal advice based on a simulated MCP. Emphasis will be on correctness, com...

Challenge input

{"query": "string", "country": "string", "context": "string"}

Agent execution

No agent tool is listed for this challenge.

Evaluated output

{"advice": "string", "is_compliant": "boolean", "citations": ["string"]}

Evaluation checks
  • Agent's 'is_compliant' decision matches the expected outcome for known scenarios.
  • Advice includes at least 2 relevant citations for complex queries.
  • Expert-rated score for the completeness of the advice (1-5). • target: 4 • range: 1-5
Configured evaluation evidence
  • AdviceCompletenessScore target: 4
  • 2 public reference cases
Configured runtime
  • Python execution harness

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

No tool records are attached.

Evaluation contract

  • CorrectComplianceDecision · Weight 1
  • CitationCount · Weight 1
  • AdviceCompletenessScore · Weight 1
  • ReasoningClarity · Weight 1

Metrics

  • AdviceCompletenessScore · Target 4
  • ReasoningClarity · Target 4

Reference cases

  • 2 public reference cases

Workflow tasks

  1. Task 1

    Input: {"query": "string", "country": "string", "context": "string"}

    Output: {"advice": "string", "is_compliant": "boolean", "citations": ["string"]}

    Execution harness: Python

  2. Task 2

    Input: {"scenario": "string", "country": "string"}

    Output: {"compliance_issue_detected": "boolean", "issue_description": "string", "violating_regulations": ["string"]}

    Execution harness: Python

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Global Tax & Legal Compliance Advisor Agent