Skip to main content
Back to challenges
Challenge

Gemini-powered Voice Navigator Agent

Develop a hands-free, multimodal conversational agent using Google's Agent Development Kit (ADK) that integrates with Google Maps for real-time navigational assistance. The agent should leverage Gemini's multimodal capabilities to understand voice commands, provide spoken directions, and offer context-aware information based on the user's location and activity (e.g., walking, cycling). This challenge focuses on building robust, real-time voice interfaces that seamlessly integrate generative AI with location-based services, prioritizing safety and natural interaction.

Assistants & InterfacesHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Build a multimodal voice assistant using Google ADK and Gemini for real-time, context-aware navigation and safety assistance in Google Maps.

Delivery guide

How work is evaluated

Evaluation

The evaluation will assess the agent's ability to provide accurate, context-aware, and timely navigational and safety assistance via a hands-free voice interface. Focus will be on conversational flow, multimodal understanding, and tool integration correctness.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
4 scoring checks
Binary
4 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

CorrectToolInvocation

Verifies that the agent correctly invokes relevant Google Maps APIs for navigation and context.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

ContextualRelevance

Checks if the agent's response is relevant to the user's current location and activity.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 3

ResponseLatencyMs

Average time taken for the agent to generate a response. • target: 800 • range: 0-2000

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 4

ConversationalFluencyScore

A subjective score (1-5) on how natural and helpful the conversation feels. • target: 4 • range: 1-5

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Master Google ADK for orchestrating agent workflows, managing state, and integrating tools with Gemini.

  • Implement real-time voice input and output using Google Cloud Speech-to-Text and Text-to-Speech APIs.

  • Utilize Gemini 1.5 Pro's multimodal capabilities to process visual cues (simulated) and generate contextually rich responses.

  • Integrate with Google Maps Platform APIs to fetch real-time location, route, and point-of-interest data.

  • Design safety-critical conversational flows for cyclists and pedestrians, including hazard warnings and emergency assistance.

  • Deploy and manage the ADK agent on Google Cloud Vertex AI, ensuring scalability and low-latency inference.

Resources and assets

Reference links and supporting material

Dataset notes

Sample data for 2 tasks

Challenge setup

The evaluation will assess the agent's ability to provide accurate, context-aware, and timely navigational and safety assistance via a hands-free voice interface. Focus will be on conversational flow, multimodal under...

Challenge input

{ "user_utterance": "string", "current_location": { "latitude": "float", "longitude": "float" }, "current_activity": "string" }

Agent execution

No agent tool is listed for this challenge.

Evaluated output

{ "agent_response": "string", "api_calls_made": "array" }

Evaluation checks
  • Verifies that the agent correctly invokes relevant Google Maps APIs for navigation and context.
  • Checks if the agent's response is relevant to the user's current location and activity.
  • Average time taken for the agent to generate a response. • target: 800 • range: 0-2000
Configured evaluation evidence
  • ResponseLatencyMs target: 800
  • 2 public reference cases
Configured runtime
  • Python execution harness

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

No tool records are attached.

Evaluation contract

  • CorrectToolInvocation · Weight 1
  • ContextualRelevance · Weight 1
  • ResponseLatencyMs · Weight 1
  • ConversationalFluencyScore · Weight 1

Metrics

  • ResponseLatencyMs · Target 800
  • ConversationalFluencyScore · Target 4

Reference cases

  • 2 public reference cases

Workflow tasks

  1. Task 1

    Input: { "user_utterance": "string", "current_location": { "latitude": "float", "longitude": "float" }, "current_activity": "string" }

    Output: { "agent_response": "string", "api_calls_made": "array" }

    Execution harness: Python

  2. Task 2

    Input: { "user_utterance": "string", "current_location": { "latitude": "float", "longitude": "float" }, "detected_hazard": "string", "current_activity": "string" }

    Output: { "agent_response": "string", "safety_action_recommended": "string" }

    Execution harness: Python

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Gemini-powered Voice Navigator Agent