Back to evaluations
Public evaluation

SupplyChainComplianceAudit

Evaluation will assess the multi-agent system's accuracy in identifying compliance breaches, the efficiency of LLM usage, and the clarity and completeness of generated compliance reports. Collaboration between agents and effective use of DSPy and adaptive budgets will be key.

Evaluation type
task based
Challenge
Supply Chain Compliance Agent with Adaptive Budgets
Difficulty
Advanced
Rigor
Unspecified
Open linked challenge

Evaluation overview

How the linked challenge is judged: tasks, benchmarks, and criteria count.

Tasks
1
Benchmarks
0
Criteria
0

Task templates

Inputs and expected outputs.

Task 1

SupplyChainComplianceAudit

Given a simulated dataset of supply chain transactions, vendor data, and incident reports, the multi-agent system must identify all potential compliance breaches related to a specific set of embargoed goods/entities and generate a structured audit report.

Input format

{'transaction_logs': [{'transaction_id': '...', 'item': '...', 'quantity': ..., 'vendor_id': '...', 'destination_country': '...', 'amount': ...}], 'vendor_data': [{'vendor_id': '...', 'name': '...', 'country': '...', 'owner_entity': '...'}], 'sanctions_list': [{'entity_name': '...', 'country': '...'}], 'news_articles': [{'title': '...', 'content': '...'}]}

Output format

{'audit_id': '...', 'status': 'completed', 'breaches_detected': [{'transaction_id': '...', 'item': '...', 'reason': '...', 'evidence_summary': '...'}], 'risk_score': 0.8, 'recommendations': ['...', ...], 'report_summary': '...', 'llm_tokens_total': 15000}