Agentic Orchestration for Wheeled Mobile Manipulators
Inspired by the rise of Agentic AI, this challenge task developers with building a high-level coordination layer for a fleet of wheeled mobile manipulators. Traditional robotics relies on rigid state machines; however, this project shifts toward an agentic paradigm where the robot interprets natural language commands and breaks them down into executable motion primitives. You will leverage LlamaIndex to query robot capability documentation (URDFs, API specs) and Apache Airflow to orchestrate long-running, multi-step tasks that require both mobility and manipulation. The core of the challenge involves creating a 'Cognitive Controller' that uses LlamaIndex to retrieve the correct joint constraints and spatial data for a specific environment. This data is then fed into an Agentic AI loop that generates DAGs (Directed Acyclic Graphs) for Apache Airflow, ensuring that the robot's navigation and arm movement are synchronized and robust to environmental changes. This represents a significant shift toward 'General Purpose' industrial automation where robots can adapt to new warehouse layouts without manual reprogramming.
What you are building
The core problem, expected build, and operating context for this challenge.
Implement an agentic planning system using LlamaIndex to convert natural language instructions into synchronized ROS2-compatible workflow tasks managed by Apache Airflow.
How work is evaluated
The system will be evaluated on its ability to translate a complex natural language task into a valid, executable, and logically sound Airflow DAG that controls a simulated manipulator.
Challenge datasets
Public datasets, plus the private datasets you have access to.
What you should walk away with
Master LlamaIndex Query Engines to parse complex URDF (Unified Robot Description Format) files and extract kinematics constraints.
Implement Agentic AI loops that utilize ReAct (Reasoning and Acting) patterns to validate robot trajectory plans before execution.
Design a custom Apache Airflow Operator specifically for ROS2 (Robot Operating System) service calls and action goals.
Build a Vector Store using Qdrant or Pinecone to store environmental spatial maps that the agent can query for navigation routing.
Orchestrate complex sequences where a mobile base must arrive at a coordinate before an arm manipulator sequence is triggered.
Optimize LLM prompt engineering for 'Zero-Shot' robot command generation using JSON-schema constrained outputs.
Deploy a monitoring dashboard that visualizes the LLM's 'chain of thought' alongside real-time telemetry from the mobile manipulator.
Reference links and supporting material
Sample data for 1 tasks
Challenge setup
The system will be evaluated on its ability to translate a complex natural language task into a valid, executable, and logically sound Airflow DAG that controls a simulated manipulator.
Challenge input
Natural language string
Agent execution
No agent tool is listed for this challenge.
Evaluated output
JSON Airflow DAG specification
- Check if the agent correctly identified that the robot arm can reach the shelf based on URDF data.
- Task Success Rate target: 90
- Python execution harness
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- Check if the agent correctly identified that the robot arm can reach the shelf based on URDF data.
Metrics
- Task Success Rate ยท Target 90
Workflow tasks
- Task 1
Input: Natural language string
Output: JSON Airflow DAG specification
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.