Multi-Agent GUI Automation with VLM-Enhanced Anomaly Detection
Inspired by research into GUI agent robustness and bridging VLMs with embodied intelligence, this challenge focuses on developing a robust multi-agent system. Participants will create a team of agents using Smolagents (or similar framework like AutoGen) that can navigate and interact with a dynamic web application. The core innovation lies in integrating a Vision-Language Model (VLM) for real-time visual perception of the GUI state and leveraging a powerful LLM (e.g., Llama 3.3 70B via Amazon Bedrock or a local deployment) for complex reasoning and anomaly detection. This challenge emphasizes building resilient AI agents that can operate in unpredictable real-world environments, a crucial step towards more reliable autonomous systems. It requires expertise in agentic design, multimodal AI integration, and robust error handling.
What you are building
Build a multi-agent system leveraging a VLM and Llama 3.3 to perform GUI automation with robust anomaly detection and explanations.
How work is evaluated
The evaluation will assess the multi-agent system's ability to successfully execute complex GUI tasks, detect various types of anomalies, and provide clear, actionable explanations. Robustness under dynamic conditions will be key.
Datasets
Public datasets, plus the private datasets you have access to.
Resources
Sample data for 2 tasks
Challenge setup
The evaluation will assess the multi-agent system's ability to successfully execute complex GUI tasks, detect various types of anomalies, and provide clear, actionable explanations. Robustness under dynamic conditions...
Challenge input
{'url': 'str', 'items_to_add': [{'name': 'str', 'quantity': 'int'}], 'expected_checkout_url': 'str'}
Agent execution
No agent tool is listed for this challenge.
Evaluated output
{'task_status': 'success|failure', 'final_page_url': 'str', 'anomalies_detected': [{'type': 'str', 'description': 'str', 'severity': 'low|medium|hi...
- At least 80% of tasks must be completed successfully without critical failures.
- At least 75% of injected anomalies must be correctly identified and classified.
- Generated explanations for anomalies must be clear, concise, and provide sufficient detail for a human oper...
- AverageTaskCompletionTime target: 60
- Python execution harness
Shows the first task, up to three tools, and up to three evidence items.
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- At least 80% of tasks must be completed successfully without critical failures.
- At least 75% of injected anomalies must be correctly identified and classified.
- Generated explanations for anomalies must be clear, concise, and provide sufficient detail for a human operator.
Metrics
- AverageTaskCompletionTime · Target 60
- RecoverySuccessRate · Target 0.7
- FalsePositiveRate · Target 0.05
Workflow tasks
- Task 1
Input: {'url': 'str', 'items_to_add': [{'name': 'str', 'quantity': 'int'}], 'expected_checkout_url': 'str'}
Output: {'task_status': 'success|failure', 'final_page_url': 'str', 'anomalies_detected': [{'type': 'str', 'description': 'str', 'severity': 'low|medium|high', 'explanation': 'str'}], 'recovery_attempted': 'bool', 'recovery_success': 'bool'}
Execution harness: Python
- Task 2
Input: {'url': 'str', 'form_data': {'field1': 'value1', 'field2': 'value2'}, 'expected_confirmation_text': 'str'}
Output: {'task_status': 'success|failure', 'confirmation_text_found': 'bool', 'anomalies_detected': [{'type': 'str', 'description': 'str', 'severity': 'low|medium|high', 'explanation': 'str'}]}
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.