Automate-IT-Ticket-Resolution
The evaluation will assess the system's ability to accurately and efficiently automate IT and accounting tasks, adhering to communication protocols and demonstrating robust tool integration.
Evaluation overview
How the linked challenge is judged: tasks, benchmarks, and criteria count.
Task templates
Inputs and expected outputs.
Automate-IT-Ticket-Resolution
An agent team receives a simulated IT support ticket (e.g., 'User cannot access VPN'). The agents must diagnose the issue, consult a knowledge base (RAG), interact with a simulated user directory API (MCP), and propose a resolution. The solution should include interaction logs, proposed steps, and confirmation from the 'user'.
{ "ticket_id": "UUID", "issue_description": "String", "user_email": "String" }
{ "ticket_id": "UUID", "resolution_plan": "String", "actions_taken": ["String"], "mcp_tool_calls": ["Object"], "a2a_interactions": ["Object"], "final_status": "String" }
Automate-Invoice-Processing
An agent team receives a simulated invoice for processing. Agents must extract key information (vendor, amount, date), validate it against a ledger API (MCP), and generate an approval recommendation. The system should demonstrate error handling for invalid invoices.
{ "invoice_data": "String (simulated invoice text)" }
{ "invoice_id": "UUID", "vendor": "String", "amount": "Float", "status": "String", "approval_recommendation": "String", "mcp_tool_calls": ["Object"] }