Edge Multimodal AI for AR Glasses: Real-time Assistant
This challenge involves developing an on-device, multimodal AI assistant tailored for AR glasses. The system needs to process real-time voice and visual inputs, combined with simulated EMG handwriting (as a gesture proxy), to provide context-aware, low-latency assistance. This assistant will leverage the multimodal capabilities of Gemini 3 Pro for advanced reasoning and LangGraph for robust state management, with a strong focus on edge inference optimization using TFLite.
What you are building
The core problem, expected build, and operating context for this challenge.
This challenge involves developing an on-device, multimodal AI assistant tailored for AR glasses. The system needs to process real-time voice and visual inputs, combined with simulated EMG handwriting (as a gesture proxy), to provide context-aware, low-latency assistance. This assistant will leverage the multimodal capabilities of Gemini 3 Pro for advanced reasoning and LangGraph for robust state management, with a strong focus on edge inference optimization using TFLite.
Shared data for this challenge
Review public datasets and any private uploads tied to your build.
What you should walk away with
Master the integration of `Gemini 3 Pro` for sophisticated multimodal understanding and generation, handling combined inputs from voice, vision, and contextual data for real-time problem-solving.
Implement a robust, stateful conversational workflow using `LangGraph` to manage user interactions, context switching, and multi-turn dialogues for the AR assistant.
Utilize `Fixie` for building a highly responsive, natural language conversational interface, specifically tailored for voice input and output on an AR device, focusing on low latency and natural turn-taking.
Optimize and deploy generative AI components for on-device inference using `TFLite`, including model quantization and compilation for efficient execution on resource-constrained edge hardware.
Design and implement a unified input pipeline that fuses real-time audio streams (voice), camera feeds (vision), and simulated gesture inputs (e.g., from an EMG sensor proxy) into a coherent multimodal context for the AI assistant.
How this agent runs
The evaluation will focus on the assistant's ability to process multimodal inputs, respond contextually and accurately with low latency, and maintain state across multi-turn interactions. On-device inference performan...
Challenge input
{'voice_input': 'What is this building?', 'image_data': 'base64_encoded_image_of_a_landmark', 'simulated_gesture': 'pointing_up'}
Agent execution
The configured agent processes the input under the challenge policy.
Evaluated output
{'response_text': 'That appears to be the Empire State Building, erected in 1931.', 'response_audio_url': 'https://...', 'action': 'display_info'}
- Verifies that the LangGraph workflow successfully processes inputs and produces an output without errors.
- Ensures the second turn's response correctly references the context from the first turn.
- MultimodalResponseAccuracy target: 0.85
- Python execution harness
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- The evaluation module defines the checks.
Recipe state
This is a preview. The configuration can change before the evaluation recipe is locked.
Run this agent on your dataset and AI stack
Bring your dataset, model providers, and success criteria. We will scope the right managed run for your team.
Scope a managed run[ok] Wrote CHALLENGE.md
[ok] Wrote .versalist.json
[ok] Wrote eval/examples.json
Requires VERSALIST_API_KEY. Works with any MCP-aware editor.
DocsFind another challenge
Jump to a random challenge when you want a fresh benchmark or a different problem space.