Skip to main content
Loading navigation
Back to challenges

Edge Multimodal AI for AR Glasses: Real-time Assistant

This challenge involves developing an on-device, multimodal AI assistant tailored for AR glasses. The system needs to process real-time voice and visual inputs, combined with simulated EMG handwriting (as a gesture proxy), to provide context-aware, low-latency assistance. This assistant will leverage the multimodal capabilities of Gemini 3 Pro for advanced reasoning and LangGraph for robust state management, with a strong focus on edge inference optimization using TFLite.

Special Purpose AgentsHosted by Vera
Sign in free to start

What you are building

Build an edge-optimized multimodal AI assistant for AR glasses using Gemini 3 Pro, Fixie for voice, LangGraph for state management, and TFLite for on-device inference.

How work is evaluated

Evaluation

The evaluation will focus on the assistant's ability to process multimodal inputs, respond contextually and accurately with low latency, and maintain state across multi-turn interactions. On-device inference performance will be a critical metric.

Datasets

Public datasets, plus the private datasets you have access to.

Learning goals

  • Optimize generative AI models for low-latency, on-device (edge) inference.

  • Implement a stateful agent workflow for context-aware assistance.

  • Develop a multimodal AI system integrating voice, vision, and gesture inputs.

  • Design a conversational interface suitable for AR wearables.

Resources

Dataset notes

Sample data for 2 tasks

Challenge setup

The evaluation will focus on the assistant's ability to process multimodal inputs, respond contextually and accurately with low latency, and maintain state across multi-turn interactions. On-device inference performan...

Challenge input

{'voice_input': 'What is this building?', 'image_data': 'base64_encoded_image_of_a_landmark', 'simulated_gesture': 'pointing_up'}

Agent execution

No agent tool is listed for this challenge.

Evaluated output

{'response_text': 'That appears to be the Empire State Building, erected in 1931.', 'response_audio_url': 'https://...', 'action': 'display_info'}

Evaluation checks
  • Verifies that the LangGraph workflow successfully processes inputs and produces an output without errors.
  • Ensures the second turn's response correctly references the context from the first turn.
Configured evaluation evidence
  • MultimodalResponseAccuracy target: 0.85
Configured runtime
  • Python execution harness

Shows the first task, up to three tools, and up to three evidence items.

View technical recipe

Configured tools

No tool records are attached.

Evaluation contract

  • Verifies that the LangGraph workflow successfully processes inputs and produces an output without errors.
  • Ensures the second turn's response correctly references the context from the first turn.

Metrics

  • MultimodalResponseAccuracy · Target 0.85
  • OnDeviceInferenceLatency · Target 250
  • ConversationalCoherence · Target 0.8

Workflow tasks

  1. Task 1

    Input: {'voice_input': 'What is this building?', 'image_data': 'base64_encoded_image_of_a_landmark', 'simulated_gesture': 'pointing_up'}

    Output: {'response_text': 'That appears to be the Empire State Building, erected in 1931.', 'response_audio_url': 'https://...', 'action': 'display_info'}

    Execution harness: Python

  2. Task 2

    Input: [{'turn': 1, 'voice': 'Tell me about the weather.'}, {'turn': 2, 'voice': 'And what about tomorrow?'}]

    Output: [{'turn': 1, 'response': 'Current weather is sunny, 25°C.'}, {'turn': 2, 'response': 'Tomorrow expects partial clouds, 23°C.'}]

    Execution harness: Python

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Edge Multimodal AI for AR Glasses: Real-time Assistant