MultimodalContextualResponse
The evaluation will focus on the assistant's ability to process multimodal inputs, respond contextually and accurately with low latency, and maintain state across multi-turn interactions. On-device inference performance will be a critical metric.
Evaluation overview
How the linked challenge is judged: tasks, benchmarks, and criteria count.
Task templates
Inputs and expected outputs.
MultimodalContextualResponse
Process a combined voice command and visual input, then provide a relevant response.
{'voice_input': 'What is this building?', 'image_data': 'base64_encoded_image_of_a_landmark', 'simulated_gesture': 'pointing_up'}
{'response_text': 'That appears to be the Empire State Building, erected in 1931.', 'response_audio_url': 'https://...', 'action': 'display_info'}
StatefulMultiTurnInteraction
Engage in a multi-turn conversation, where subsequent responses depend on previous context.
[{'turn': 1, 'voice': 'Tell me about the weather.'}, {'turn': 2, 'voice': 'And what about tomorrow?'}]
[{'turn': 1, 'response': 'Current weather is sunny, 25°C.'}, {'turn': 2, 'response': 'Tomorrow expects partial clouds, 23°C.'}]