Multimodal Video Intelligence with Qwen3-VL, GPT-5 & LlamaIndex
Inspired by advancements in long-context multimodal understanding, this challenge tasks you with building a cutting-edge video intelligence system. You will integrate the Qwen3-VL model for robust video and image analysis with GPT-5 for higher-level reasoning and synthesis. The system will leverage LlamaIndex for advanced RAG over multimodal data, allowing it to accurately answer complex 'needle-in-a-haystack' queries spanning long video durations. The core of the system will involve processing entire 30-minute video segments, extracting key visual and auditory information, generating multimodal embeddings, and indexing them using LlamaIndex. An OpenAI Swarm-like orchestration will manage specialized agents that collaborate using an A2A protocol to perform visual search, event detection, and generate comprehensive summaries. MCP could be used to facilitate access to external video processing tools or contextual databases.
What you are building
The core problem, expected build, and operating context for this challenge.
Develop a multimodal video intelligence system using Qwen3-VL, GPT-5, and LlamaIndex for advanced RAG over long video content, orchestrated by an OpenAI Swarm-like agent collective.
How work is evaluated
The evaluation will assess the system's ability to accurately answer complex 'needle-in-a-haystack' queries against provided 30-minute video files, measuring recall, precision, and reasoning depth.
Shared data for this challenge
Review public datasets and any private uploads tied to your build.
What you should walk away with
Utilize GPT-5 for complex query answering and synthesis over extracted video information
Implement multimodal RAG for long-form video content using LlamaIndex
Achieve high accuracy on 'needle-in-a-haystack' queries across 30-minute videos
Integrate Qwen3-VL for advanced visual and auditory understanding
Orchestrate specialized agents with an OpenAI Swarm-like architecture and A2A protocol
Reference links and supporting material
Sample data for 1 tasks
How this agent runs
The evaluation will assess the system's ability to accurately answer complex 'needle-in-a-haystack' queries against provided 30-minute video files, measuring recall, precision, and reasoning depth.
Challenge input
{"video_url": "<url_to_30_min_video>", "queries": ["<query1>", "<query2>"]}
Agent execution
The configured agent processes the input under the challenge policy.
Evaluated output
{"query_results": [{"query": "<query>", "answer": "<text>", "evidence_timestamps": "<list_of_timestamps>"}]}
- All provided queries receive an answer.
- Answers demonstrate linkage to both visual (Qwen3-VL) and textual/auditory (GPT-5/transcription) evidence.
- AnswerAccuracy target: 0.95
- Python execution harness
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- The evaluation module defines the checks.
Recipe state
This is a preview. The configuration can change before the evaluation recipe is locked.