Skip to main content
Back to challenges
Challenge

Build Autonomous Media Auditing Agents with Vercel AI SDK and OpenHands

Develop an automated media auditing suite that evaluates streaming content performance and viewer attrition metrics using the Vercel AI SDK. This system integrates the GPT-5.4-mini model to synthesize Netflix-style engagement data and simulate predictive analysis of show abandonment patterns. The agentic flow utilizes OpenHands for autonomous script-based interactions and Google Jules for assisting in the generation of structured audit logs. The entire pipeline is instrumented with Arize Phoenix to track model behavior and ensure high-fidelity reporting.

Workflow AutomationHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Utilize Vercel AI SDK to orchestrate autonomous agents that analyze viewer attrition data and generate actionable content performance reports.

Delivery guide

How work is evaluated

Evaluation

Automated evaluation of agent reasoning quality and tool output accuracy.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

SchemaCheck

Ensure output follows prescribed JSON structure

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

Accuracy

Correct identification of drop-off thresholds • target: 0.9 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Master the Vercel AI SDK core abstractions for streaming and multi-provider agent orchestration

  • Implement autonomous task execution with OpenHands to parse streaming engagement logs

  • Integrate Google Jules as a code-assistance layer for automating data pipeline generation

  • Configure Arize Phoenix for real-time observability and performance metrics of agent workflows

  • Orchestrate voice-enabled alerts using Retell AI to notify production teams of critical viewer drop-off trends

Resources and assets

Reference links and supporting material

Dataset notes

Sample data for 1 tasks

Challenge setup

Automated evaluation of agent reasoning quality and tool output accuracy.

Challenge input

JSON

OpenHands

Open-source software development agent.

Vercel AI SDK

TypeScript toolkit for AI apps.

Arize Phoenix

Open-source LLM observability and evals.

Evaluated output

JSON

Evaluation checks
  • Ensure output follows prescribed JSON structure
  • Correct identification of drop-off thresholds • target: 0.9 • range: 0-1
Configured evaluation evidence
  • Accuracy target: 0.9
  • 1 public reference case
Configured runtime
  • Python execution harness

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

Environment
  • OpenHands · Required
Action Space
  • OpenHands · Required
  • Vercel AI SDK · Optional
Observation
  • Arize Phoenix · Optional
Reward / Eval
  • Arize Phoenix · Optional
Orchestration
  • OpenHands · Required
  • Vercel AI SDK · Optional

Evaluation contract

  • SchemaCheck · Weight 1
  • Accuracy · Weight 1

Metrics

  • Accuracy · Target 0.9

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: JSON

    Output: JSON

    Execution harness: Python

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Build Autonomous Media Auditing Agents with Vercel AI SDK and OpenHands