Resilient Perception Fusion
There is a critical need for robotics companies to build sensor-agnostic perception systems. This challenge focuses on building a 'Resilient Perception Layer' that uses Mistral Large 2 for high-level reasoning and Model Context Protocol (MCP) servers to abstract various sensor inputs. Instead of relying on a single expensive Lidar unit, you will build a system that can dynamically switch between stereo vision, solid-state lidar, and ultrasonic sensors based on environmental conditions (e.g., fog, low light, or indoor glare). You will utilize Google Vertex AI to host your reasoning models and deploy MCP servers that act as 'drivers' for different sensor data streams. The goal is to implement a 'Perception Agent' that monitors sensor confidence scores and uses Mistral Large 2 to decide which sensor fusion strategy to employ in real-time, ensuring safe operation of an autonomous mobile base even when primary sensors fail.
What you are building
The core problem, expected build, and operating context for this challenge.
Build a modular sensor-fusion perception system using MCP servers to abstract hardware, with Mistral Large 2 on Vertex AI acting as the decision-making 'Arbiter' for sensor reliability.
How work is evaluated
The system is evaluated based on its ability to maintain a safe path for a robot during a simulated sensor failure (e.g., Lidar disconnecting or becoming noisy).
Challenge datasets
Public datasets, plus the private datasets you have access to.
What you should walk away with
Build MCP (Model Context Protocol) servers in Node.js or Python to wrap diverse sensor APIs into a unified interface.
Integrate Google Vertex AI SDK to call Mistral Large 2 for analyzing sensor health telemetry.
Implement a Kalman Filter or Particle Filter in Python to fuse data from multiple low-cost sensors when high-end Lidar is unavailable.
Design a 'Sensor Arbiter' agent that uses Mistral's reasoning capabilities to diagnose sensor 'blindness' (e.g., Identifying that a camera is blinded by sunlight).
Optimize the inference pipeline using Vertex AI Prediction endpoints to maintain a system latency below 200ms.
Develop a visualization tool using Streamlit to show real-time sensor weights and the model's reasoning for choosing a specific sensor set.
Orchestrate a 'Graceful Degradation' protocol where the robot reduces speed automatically if sensor confidence drops below a threshold.
Reference links and supporting material
Sample data for 1 tasks
Challenge setup
The system is evaluated based on its ability to maintain a safe path for a robot during a simulated sensor failure (e.g., Lidar disconnecting or becoming noisy).
Challenge input
JSON stream of sensor health and environmental noise levels
Agent execution
No agent tool is listed for this challenge.
Evaluated output
Sensor selection command and reasoning string
- Decision-making must occur in under 500ms to be considered valid for a moving robot.
- Navigation Safety Score target: 0.98
- Docker execution harness
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- Decision-making must occur in under 500ms to be considered valid for a moving robot.
Metrics
- Navigation Safety Score ยท Target 0.98
Workflow tasks
- Task 1
Input: JSON stream of sensor health and environmental noise levels
Output: Sensor selection command and reasoning string
Execution harness: Docker
Recipe freeze marker
No recipe freeze time is recorded.