Human-Robot Team Collaboration
Addressing the challenges of Tesla's Optimus humanoid robots, this challenge focuses on enhancing human-robot collaboration in complex manufacturing or assembly tasks. You will build a multi-agent system using AutoGen, leveraging GPT-5 for sophisticated task planning, problem-solving, and natural language understanding. The system will feature MCP-enabled tool integration for robots to interact with simulated manufacturing execution systems (MES) and access operational knowledge via RAG with vector search. The emphasis is on creating a seamless human-in-the-loop experience, allowing human operators to provide natural language feedback that the agents use for continuous learning and adaptive task allocation. HappyPath AI Engineering Tooling will be instrumental in visualizing, debugging, and optimizing these complex human-robot workflows, ensuring efficient and error-resilient operations.
What you are building
The core problem, expected build, and operating context for this challenge.
Build an AutoGen multi-agent system with GPT-5 to facilitate human-robot collaboration for complex tasks, integrating MCP tools, RAG with vector search, extended thinking, and HappyPath tooling.
How work is evaluated
The evaluation will focus on the multi-agent system's effectiveness in orchestrating a simulated human-robot assembly task, emphasizing task completion rate, efficiency of error handling with human intervention, and the utility of RAG.
Challenge datasets
Public datasets, plus the private datasets you have access to.
What you should walk away with
Develop MCP-enabled tool integration for simulated robot control and system feedback.
Integrate GPT-5 for advanced planning, reasoning, and natural language interaction with extended thinking.
Utilize HappyPath AI Engineering Tooling for workflow design, testing, and optimization.
Design and implement an AutoGen multi-agent system for human-robot teamwork.
Implement RAG with vector search for robot operational knowledge access.
Reference links and supporting material
Sample data for 1 tasks
Challenge setup
The evaluation will focus on the multi-agent system's effectiveness in orchestrating a simulated human-robot assembly task, emphasizing task completion rate, efficiency of error handling with human intervention, and t...
Challenge input
{"assembly_plan": [{"step_id": "int", "task": "string", "assigned_to": "human|robot"}, ...], "initial_robot_state": {"position": "string", "gripper...
Agent execution
No agent tool is listed for this challenge.
Evaluated output
{"task_execution_log": [{"step_id": "int", "agent_action": "string", "status": "string", "error": "string|null", "human_interaction": "string|null"...
- All defined assembly steps must be marked as 'completed' in the final task execution log.
- When an error is simulated, the system must demonstrate a valid recovery sequence involving human feedback,...
- Average Task Completion Time (Seconds) target: 120
- Python execution harness
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- All defined assembly steps must be marked as 'completed' in the final task execution log.
- When an error is simulated, the system must demonstrate a valid recovery sequence involving human feedback, GPT-5 re-planning, and successful re-execution.
Metrics
- Average Task Completion Time (seconds) · Target 120
- RAG Effectiveness Score · Target 0.8
- Human Interaction Efficiency · Target 1
Workflow tasks
- Task 1
Input: {"assembly_plan": [{"step_id": "int", "task": "string", "assigned_to": "human|robot"}, ...], "initial_robot_state": {"position": "string", "gripper_status": "string"}, "error_scenarios": [{"step_id": "int", "type": "string", "description": "string"}, ...]}
Output: {"task_execution_log": [{"step_id": "int", "agent_action": "string", "status": "string", "error": "string|null", "human_interaction": "string|null", "rag_queries": [{"query": "string", "result_summary": "string"}]}, ...], "final_product_status": "string", "overall_efficiency_score": "float"}
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.