Gemini-powered Voice Navigator Agent
Develop a hands-free, multimodal conversational agent using Google's Agent Development Kit (ADK) that integrates with Google Maps for real-time navigational assistance. The agent should leverage Gemini's multimodal capabilities to understand voice commands, provide spoken directions, and offer context-aware information based on the user's location and activity (e.g., walking, cycling). This challenge focuses on building robust, real-time voice interfaces that seamlessly integrate generative AI with location-based services, prioritizing safety and natural interaction.
What you are building
Build a multimodal voice assistant using Google ADK and Gemini for real-time, context-aware navigation and safety assistance in Google Maps.
How work is evaluated
The evaluation will assess the agent's ability to provide accurate, context-aware, and timely navigational and safety assistance via a hands-free voice interface. Focus will be on conversational flow, multimodal understanding, and tool integration correctness.
Datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
CorrectToolInvocation
Verifies that the agent correctly invokes relevant Google Maps APIs for navigation and context.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
ContextualRelevance
Checks if the agent's response is relevant to the user's current location and activity.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
ResponseLatencyMs
Average time taken for the agent to generate a response. • target: 800 • range: 0-2000
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
ConversationalFluencyScore
A subjective score (1-5) on how natural and helpful the conversation feels. • target: 4 • range: 1-5
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Learning goals
Master Google ADK for orchestrating agent workflows, managing state, and integrating tools with Gemini.
Implement real-time voice input and output using Google Cloud Speech-to-Text and Text-to-Speech APIs.
Utilize Gemini 1.5 Pro's multimodal capabilities to process visual cues (simulated) and generate contextually rich responses.
Integrate with Google Maps Platform APIs to fetch real-time location, route, and point-of-interest data.
Design safety-critical conversational flows for cyclists and pedestrians, including hazard warnings and emergency assistance.
Deploy and manage the ADK agent on Google Cloud Vertex AI, ensuring scalability and low-latency inference.
Resources
Sample data for 2 tasks
Challenge setup
The evaluation will assess the agent's ability to provide accurate, context-aware, and timely navigational and safety assistance via a hands-free voice interface. Focus will be on conversational flow, multimodal under...
Challenge input
{ "user_utterance": "string", "current_location": { "latitude": "float", "longitude": "float" }, "current_activity": "string" }
Agent execution
No agent tool is listed for this challenge.
Evaluated output
{ "agent_response": "string", "api_calls_made": "array" }
- Verifies that the agent correctly invokes relevant Google Maps APIs for navigation and context.
- Checks if the agent's response is relevant to the user's current location and activity.
- Average time taken for the agent to generate a response. • target: 800 • range: 0-2000
- ResponseLatencyMs target: 800
- 2 public reference cases
- Python execution harness
Shows the first task, up to three tools, and up to three evidence items.
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- CorrectToolInvocation · Weight 1
- ContextualRelevance · Weight 1
- ResponseLatencyMs · Weight 1
- ConversationalFluencyScore · Weight 1
Metrics
- ResponseLatencyMs · Target 800
- ConversationalFluencyScore · Target 4
Reference cases
- 2 public reference cases
Workflow tasks
- Task 1
Input: { "user_utterance": "string", "current_location": { "latitude": "float", "longitude": "float" }, "current_activity": "string" }
Output: { "agent_response": "string", "api_calls_made": "array" }
Execution harness: Python
- Task 2
Input: { "user_utterance": "string", "current_location": { "latitude": "float", "longitude": "float" }, "detected_hazard": "string", "current_activity": "string" }
Output: { "agent_response": "string", "safety_action_recommended": "string" }
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.