Contextual Task Execution
Evaluate the voice assistant's ability to accurately understand complex voice commands, leverage historical context for personalization, and execute tasks reliably in real-time.
Evaluation overview
How the linked challenge is judged: tasks, benchmarks, and criteria count.
Task templates
Inputs and expected outputs.
Contextual Task Execution
Tests the assistant's ability to perform tasks requiring long-term memory and contextual understanding across multiple turns.
JSON object with 'user_voice_commands' (array of strings) and 'initial_memory_state' (object representing stored preferences).
JSON object with 'actions_taken' (array of strings) and 'personalization_score' (float).
Real-time Correction Handling
Assesses the assistant's ability to handle corrections and disambiguations effectively in a real-time conversational flow.
JSON object with 'user_voice_commands' (array of strings) and 'device_state_log' (array of initial device actions).
JSON object with 'final_state' (string) and 'conversational_fluency_score' (float).