Multi-Agent GUI Automation with VLM-Enhanced Anomaly Detection
Inspired by research into GUI agent robustness and bridging VLMs with embodied intelligence, this challenge focuses on developing a robust multi-agent system. Participants will create a team of agents using Smolagents (or similar framework like AutoGen) that can navigate and interact with a dynamic web application. The core innovation lies in integrating a Vision-Language Model (VLM) for real-time visual perception of the GUI state and leveraging a powerful LLM (e.g., Llama 3.3 70B via Amazon Bedrock or a local deployment) for complex reasoning and anomaly detection. This challenge emphasizes building resilient AI agents that can operate in unpredictable real-world environments, a crucial step towards more reliable autonomous systems. It requires expertise in agentic design, multimodal AI integration, and robust error handling.
What you are building
The core problem, expected build, and operating context for this challenge.
Build a multi-agent system leveraging a VLM and Llama 3.3 to perform GUI automation with robust anomaly detection and explanations.
How work is evaluated
The evaluation will assess the multi-agent system's ability to successfully execute complex GUI tasks, detect various types of anomalies, and provide clear, actionable explanations. Robustness under dynamic conditions will be key.
Challenge datasets
Public datasets, plus the private datasets you have access to.
What you should walk away with
Learning objectives will be added soon
Use the overview and evaluation guide as the source of truth for expected outcomes.
Reference links and supporting material
Sample data for 2 tasks
Challenge setup
The evaluation will assess the multi-agent system's ability to successfully execute complex GUI tasks, detect various types of anomalies, and provide clear, actionable explanations. Robustness under dynamic conditions...
Challenge input
{'url': 'str', 'items_to_add': [{'name': 'str', 'quantity': 'int'}], 'expected_checkout_url': 'str'}
Agent execution
No agent tool is listed for this challenge.
Evaluated output
{'task_status': 'success|failure', 'final_page_url': 'str', 'anomalies_detected': [{'type': 'str', 'description': 'str', 'severity': 'low|medium|hi...
- At least 80% of tasks must be completed successfully without critical failures.
- At least 75% of injected anomalies must be correctly identified and classified.
- Generated explanations for anomalies must be clear, concise, and provide sufficient detail for a human oper...
- AverageTaskCompletionTime target: 60
- Python execution harness
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- At least 80% of tasks must be completed successfully without critical failures.
- At least 75% of injected anomalies must be correctly identified and classified.
- Generated explanations for anomalies must be clear, concise, and provide sufficient detail for a human operator.
Metrics
- AverageTaskCompletionTime · Target 60
- RecoverySuccessRate · Target 0.7
- FalsePositiveRate · Target 0.05
Workflow tasks
- Task 1
Input: {'url': 'str', 'items_to_add': [{'name': 'str', 'quantity': 'int'}], 'expected_checkout_url': 'str'}
Output: {'task_status': 'success|failure', 'final_page_url': 'str', 'anomalies_detected': [{'type': 'str', 'description': 'str', 'severity': 'low|medium|high', 'explanation': 'str'}], 'recovery_attempted': 'bool', 'recovery_success': 'bool'}
Execution harness: Python
- Task 2
Input: {'url': 'str', 'form_data': {'field1': 'value1', 'field2': 'value2'}, 'expected_confirmation_text': 'str'}
Output: {'task_status': 'success|failure', 'confirmation_text_found': 'bool', 'anomalies_detected': [{'type': 'str', 'description': 'str', 'severity': 'low|medium|high', 'explanation': 'str'}]}
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.