Skip to main content
Back to challenges
Challenge

Real-time Voice Assistant for Market Intelligence

Develop a sophisticated real-time voice assistant capable of performing competitive market intelligence and product analysis. This challenge focuses on building a responsive, multi-modal agent system that leverages the AI SDK for seamless streaming interactions and sophisticated tool orchestration. The core reasoning will be powered by Google's Gemini 3 Flash, providing rapid and accurate insights based on voice input. The system will integrate Microsoft AutoGen to spin up specialized, scriptable agents that handle deep-dive research tasks, collaborating with the primary voice agent. Security and data integrity are paramount, so LatticeFlow AI will be utilized to implement robust model safety policies and secure data pipelines. For comprehensive monitoring and evaluation, LangFuse will be integrated to trace agent interactions and performance metrics, ensuring the system operates efficiently and reliably. Retell AI will provide the real-time voice-to-text and text-to-speech capabilities, enabling natural language interactions for the end-user. This challenge emphasizes cutting-edge multi-agent orchestration, real-time voice processing, and advanced AI engineering practices to deliver a high-performance, secure, and observable AI application.

Special Purpose AgentsHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Build a real-time, voice-activated multi-agent system using Vercel's AI SDK with Gemini 3 Flash, AutoGen agents, Retell AI, LatticeFlow AI, and LangFuse for market intelligence.

Delivery guide

How work is evaluated

Evaluation

Evaluation will focus on the real-time responsiveness, analytical accuracy, security compliance, and observability of the voice-activated market intelligence agent. Automated tests will simulate user queries and verify generated insights against expected outcomes, while manual review assesses naturalness of voice interaction.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
7 scoring checks
Binary
7 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

Real-time Voice Latency

Ensure average response time from audio input to audio output is under 3 seconds.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

Market Analysis Accuracy

Verify key findings from market analysis queries are factually correct and relevant.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 3

AutoGen Task Completion

Confirm AutoGen agents successfully complete their assigned sub-tasks.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 4

Security Policy Adherence

Check if LatticeFlow AI correctly identifies and mitigates policy violations.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 5

Semantic Similarity Score (Analysis)

Cosine similarity between generated analysis text and expert-reviewed benchmark text. • target: 0.85 • range: 0.7-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 6

Tool Invocation Rate

Percentage of relevant tools invoked per complex query. • target: 0.95 • range: 0.8-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 7

LangFuse Trace Completeness

Percentage of agent steps and tool calls captured in LangFuse traces. • target: 1 • range: 0.95-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Master Vercel's AI SDK for building streaming AI applications in TypeScript, including advanced tool calling and state management.

  • Implement real-time voice input and output using Retell AI, connecting it seamlessly with the AI SDK and underlying language models.

  • Orchestrate dynamic agent teams with Microsoft AutoGen, defining specialized roles for information gathering, synthesis, and summarization.

  • Design secure agent-to-agent communication protocols and data handling practices, integrating LatticeFlow AI for compliance and risk management.

  • Build extended reasoning pipelines leveraging Gemini 3 Flash for rapid and context-aware analysis of complex market data.

  • Integrate LangFuse into the AI SDK and AutoGen workflows for end-to-end tracing, monitoring, and performance evaluation of agent interactions.

  • Develop custom tools for the AI SDK agents to access external market data APIs, web scrapers, and internal knowledge bases.

Resources and assets

Reference links and supporting material

Dataset notes

Sample data for 3 tasks

Challenge setup

Evaluation will focus on the real-time responsiveness, analytical accuracy, security compliance, and observability of the voice-activated market intelligence agent. Automated tests will simulate user queries and verif...

Challenge input

{ "audio_input": "base64_encoded_audio", "query": "Analyze the potential market impact of Sony's new WF-1000XM6 earbuds, considering competitor pro...

Agent execution

No agent tool is listed for this challenge.

Evaluated output

{ "response_audio": "base64_encoded_audio", "analysis_text": "string", "key_findings": ["string"] }

Evaluation checks
  • Ensure average response time from audio input to audio output is under 3 seconds.
  • Verify key findings from market analysis queries are factually correct and relevant.
  • Confirm AutoGen agents successfully complete their assigned sub-tasks.
Configured evaluation evidence
  • Semantic Similarity Score (Analysis) target: 0.85
  • 3 public reference cases
Configured runtime
  • Python execution harness

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

No tool records are attached.

Evaluation contract

  • Real-time Voice Latency · Weight 1
  • Market Analysis Accuracy · Weight 1
  • AutoGen Task Completion · Weight 1
  • Security Policy Adherence · Weight 1
  • Semantic Similarity Score (Analysis) · Weight 1
  • Tool Invocation Rate · Weight 1
  • LangFuse Trace Completeness · Weight 1

Metrics

  • Semantic Similarity Score (Analysis) · Target 0.85
  • Tool Invocation Rate · Target 0.95
  • LangFuse Trace Completeness · Target 1

Reference cases

  • 3 public reference cases

Workflow tasks

  1. Task 1

    Input: { "audio_input": "base64_encoded_audio", "query": "Analyze the potential market impact of Sony's new WF-1000XM6 earbuds, considering competitor products and consumer trends." }

    Output: { "response_audio": "base64_encoded_audio", "analysis_text": "string", "key_findings": ["string"] }

    Execution harness: Python

  2. Task 2

    Input: { "trigger_query": "Provide a summary of recent analyst reports on the audio electronics market trends." }

    Output: { "autogen_log_snapshot": "string", "synthesized_report_excerpt": "string" }

    Execution harness: Python

  3. Task 3

    Input: { "query": "string" }

    Output: { "policy_violation_detected": "boolean", "sanitized_output": "string" }

    Execution harness: Python

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Real-time Voice Assistant for Market Intelligence