Edge Multimodal AI for AR Glasses: Real-time Assistant
This challenge involves developing an on-device, multimodal AI assistant tailored for AR glasses. The system needs to process real-time voice and visual inputs, combined with simulated EMG handwriting (as a gesture proxy), to provide context-aware, low-latency assistance. This assistant will leverage the multimodal capabilities of Gemini 3 Pro for advanced reasoning and LangGraph for robust state management, with a strong focus on edge inference optimization using TFLite.
What you are building
The core problem, expected build, and operating context for this challenge.
Build an edge-optimized multimodal AI assistant for AR glasses using Gemini 3 Pro, Fixie for voice, LangGraph for state management, and TFLite for on-device inference.
How work is evaluated
The evaluation will focus on the assistant's ability to process multimodal inputs, respond contextually and accurately with low latency, and maintain state across multi-turn interactions. On-device inference performance will be a critical metric.
Challenge datasets
Public datasets, plus the private datasets you have access to.
What you should walk away with
Optimize generative AI models for low-latency, on-device (edge) inference.
Implement a stateful agent workflow for context-aware assistance.
Develop a multimodal AI system integrating voice, vision, and gesture inputs.
Design a conversational interface suitable for AR wearables.
Reference links and supporting material
Sample data for 2 tasks
Challenge setup
The evaluation will focus on the assistant's ability to process multimodal inputs, respond contextually and accurately with low latency, and maintain state across multi-turn interactions. On-device inference performan...
Challenge input
{'voice_input': 'What is this building?', 'image_data': 'base64_encoded_image_of_a_landmark', 'simulated_gesture': 'pointing_up'}
Agent execution
No agent tool is listed for this challenge.
Evaluated output
{'response_text': 'That appears to be the Empire State Building, erected in 1931.', 'response_audio_url': 'https://...', 'action': 'display_info'}
- Verifies that the LangGraph workflow successfully processes inputs and produces an output without errors.
- Ensures the second turn's response correctly references the context from the first turn.
- MultimodalResponseAccuracy target: 0.85
- Python execution harness
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- Verifies that the LangGraph workflow successfully processes inputs and produces an output without errors.
- Ensures the second turn's response correctly references the context from the first turn.
Metrics
- MultimodalResponseAccuracy · Target 0.85
- OnDeviceInferenceLatency · Target 250
- ConversationalCoherence · Target 0.8
Workflow tasks
- Task 1
Input: {'voice_input': 'What is this building?', 'image_data': 'base64_encoded_image_of_a_landmark', 'simulated_gesture': 'pointing_up'}
Output: {'response_text': 'That appears to be the Empire State Building, erected in 1931.', 'response_audio_url': 'https://...', 'action': 'display_info'}
Execution harness: Python
- Task 2
Input: [{'turn': 1, 'voice': 'Tell me about the weather.'}, {'turn': 2, 'voice': 'And what about tomorrow?'}]
Output: [{'turn': 1, 'response': 'Current weather is sunny, 25°C.'}, {'turn': 2, 'response': 'Tomorrow expects partial clouds, 23°C.'}]
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.