Back to evaluations
Public evaluation

VideoNeedleInHaystack

The evaluation will assess the system's ability to accurately answer complex 'needle-in-a-haystack' queries against provided 30-minute video files, measuring recall, precision, and reasoning depth.

Evaluation type
task based
Challenge
Multimodal Video Intelligence with Qwen3-VL, GPT-5 & LlamaIndex
Difficulty
Advanced
Rigor
Unspecified
Open linked challenge

Evaluation overview

How the linked challenge is judged: tasks, benchmarks, and criteria count.

Tasks
1
Benchmarks
0
Criteria
0

Task templates

Inputs and expected outputs.

Task 1

VideoNeedleInHaystack

Evaluates the system's multimodal RAG and agent orchestration by asking detailed questions about specific events or objects buried within a long video.

Input format

{"video_url": "<url_to_30_min_video>", "queries": ["<query1>", "<query2>"]}

Output format

{"query_results": [{"query": "<query>", "answer": "<text>", "evidence_timestamps": "<list_of_timestamps>"}]}