Skip to main content
Back to challenges
Challenge

LlamaIndex & Deepgram Assistant for Autonomous Maritime Patrol Telemetry

Abu Dhabi Mobility has launched field trials for autonomous patrol boats to secure maritime channels and streamline harbor operations. Managing autonomous vessel fleets requires real-time processing of high-frequency telemetry data, vessel sensor streams, operational manuals, and voice commands transmitted from maritime control centers. In this challenge, you will implement an intelligent voice-enabled operational assistant using LlamaIndex for advanced Retrieval-Augmented Generation (RAG) and Deepgram for speech recognition. The assistant will ingest transcribed voice instructions from port commanders, retrieve relevant autonomous system protocols from technical documentation, and query real-time boat telemetry data. By building hybrid indexes over structured telemetry databases and unstructured operational manuals with LlamaIndex, your agent will provide instant situational awareness, diagnose autonomous navigation anomalies, and recommend immediate course adjustments for maritime safety teams.

Business OperationsHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Build a voice-enabled operational assistant combining Deepgram speech recognition and LlamaIndex advanced RAG to query autonomous boat telemetry and maritime protocols.

Delivery guide

How work is evaluated

Evaluation

Evaluates speech transcription integration, telemetry retrieval accuracy, and response synthesis quality.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
3 scoring checks
Binary
3 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

correct_vessel_identified

Verifies the query response targets the correct vessel ID.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

safety_protocol_triggered

Ensures speed reduction safety action is present in recommendation.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 3

retrieval_relevance

Measures relevance of retrieved manual sections and telemetry values. • target: 0.95 • range: 0.8-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Build an advanced RAG pipeline using LlamaIndex over structured sensor telemetry and unstructured maritime manuals.

  • Integrate Deepgram Speech-to-Text API for processing audio commands from maritime control room operators.

  • Implement sub-question query engines and router engines in LlamaIndex to route voice queries dynamically.

  • Evaluate retrieval quality and latency for real-time maritime decision support systems.

Resources and assets

Reference links and supporting material

Dataset notes

Dataset consisting of 30 mock audio voice commands from maritime traffic controllers, paired with telemetry sensor JSON feeds and PDF operating manuals for autonomous patrol vessels in Abu Dhabi.

Challenge setup

Evaluates speech transcription integration, telemetry retrieval accuracy, and response synthesis quality.

Challenge input

JSON object containing query_text or audio_path, current_vessel_telemetry, and context_docs

LlamaIndex

Enables multi-document, multi-modal, and hybrid index querying across tabular telemetry and text documentation.

Deepgram

Provides fast, highly accurate Speech-to-Text transcription with customizable domain vocabulary.

Evaluated output

JSON object containing transcribed_query, vessel_id, status_summary, and recommended_action

Evaluation checks
  • Verifies the query response targets the correct vessel ID.
  • Ensures speed reduction safety action is present in recommendation.
  • Measures relevance of retrieved manual sections and telemetry values. • target: 0.95 • range: 0.8-1
Configured evaluation evidence
  • Benchmark: Autonomous Maritime Assistant Evaluation Suite
  • Retrieval Relevance target: 0.95
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

Action Space
  • Deepgram · Required
  • Llama Index · Optional
Observation
  • Langfuse · Optional
Reward / Eval
  • Langfuse · Optional

Evaluation contract

  • correct_vessel_identified · Weight 1
  • safety_protocol_triggered · Weight 1
  • retrieval_relevance · Weight 1

Metrics

  • retrieval_relevance · Target 0.95

Benchmarks

  • Autonomous Maritime Assistant Evaluation Suite
  • robotics

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: JSON object containing query_text or audio_path, current_vessel_telemetry, and context_docs

    Output: JSON object containing transcribed_query, vessel_id, status_summary, and recommended_action

    Execution harness: Python

Evaluation tools

  • LlamaIndex · Data framework & RAG pipeline
  • Deepgram · Speech recognition

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about LlamaIndex & Deepgram Assistant for Autonomous Maritime Patrol Telemetry