Skip to main content
Back to challenges
Challenge

Pediatric Speech Acoustic and Semantic Trajectory Analyzer with Google ADK

Inspired by NIH findings connecting early childhood language patterns with later internalizing disorders, this challenge focuses on developing a multi-modal agent pipeline using Google ADK and Vertex AI. Participants will construct an agent that processes both acoustic speech parameters (pitch variability, pause durations) and transcript semantic features to output early risk trajectories.

Data ScienceHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Construct a multi-modal pediatric speech analysis pipeline using Google ADK to process acoustic and transcript data.

Delivery guide

How work is evaluated

Evaluation

Evaluates multi-modal risk score computation combining audio metrics and transcript text.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

check_contributions_sum

Ensure acoustic and text risk contributions combine to total risk score within 0.05

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

combined_risk_mae

Mean Absolute Error of combined multi-modal risk score against clinical expert rating • target: 0.06 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Build multi-modal agent processing tools using Google ADK

  • Extract acoustic features (pitch jitter, speech rate, pause ratios) using librosa

  • Combine acoustic signals with Gemini text embeddings for multi-modal risk scoring

  • Validate prospective anxiety prediction against empirical longitudinal study metrics

Resources and assets

Reference links and supporting material

Dataset notes

Synthetic pediatric audio feature extractions and transcript excerpts paired with longitudinal anxiety risk outcomes.

Challenge setup

Evaluates multi-modal risk score computation combining audio metrics and transcript text.

Challenge input

JSON containing mean_pitch_hz, mean_pause_duration_s, transcript_text

Google ADK

Mandatory Google ADK framework

Evaluated output

JSON containing combined_risk_score (float 0-1), acoustic_risk_contrib (float), text_risk_contrib (float)

Evaluation checks
  • Ensure acoustic and text risk contributions combine to total risk score within 0.05
  • Mean Absolute Error of combined multi-modal risk score against clinical expert rating • target: 0.06 • rang...
Configured evaluation evidence
  • Benchmark: MultiModalSpeech-Bench
  • Combined Risk Mae target: 0.06
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

Action Space
  • Google · Required
  • Google ADK · Optional
  • Google Gemini · Optional

Evaluation contract

  • check_contributions_sum · Weight 1
  • combined_risk_mae · Weight 1

Metrics

  • combined_risk_mae · Target 0.06

Benchmarks

  • MultiModalSpeech-Bench
  • biotech

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: JSON containing mean_pitch_hz, mean_pause_duration_s, transcript_text

    Output: JSON containing combined_risk_score (float 0-1), acoustic_risk_contrib (float), text_risk_contrib (float)

    Execution harness: Python

Evaluation tools

  • Google ADK · Agent Framework

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Pediatric Speech Acoustic and Semantic Trajectory Analyzer with Google ADK