Multimodal Video Intelligence with Qwen3-VL, GPT-5 & LlamaIndex
Inspired by advancements in long-context multimodal understanding, this challenge tasks you with building a cutting-edge video intelligence system. You will integrate the Qwen3-VL model for robust video and image analysis with GPT-5 for higher-level reasoning and synthesis. The system will leverage LlamaIndex for advanced RAG over multimodal data, allowing it to accurately answer complex 'needle-in-a-haystack' queries spanning long video durations. The core of the system will involve processing entire 30-minute video segments, extracting key visual and auditory information, generating multimodal embeddings, and indexing them using LlamaIndex. An OpenAI Swarm-like orchestration will manage specialized agents that collaborate using an A2A protocol to perform visual search, event detection, and generate comprehensive summaries. MCP could be used to facilitate access to external video processing tools or contextual databases.
What you are building
The core problem, expected build, and operating context for this challenge.
Inspired by advancements in long-context multimodal understanding, this challenge tasks you with building a cutting-edge video intelligence system. You will integrate the Qwen3-VL model for robust video and image analysis with GPT-5 for higher-level reasoning and synthesis. The system will leverage LlamaIndex for advanced RAG over multimodal data, allowing it to accurately answer complex 'needle-in-a-haystack' queries spanning long video durations. The core of the system will involve processing entire 30-minute video segments, extracting key visual and auditory information, generating multimodal embeddings, and indexing them using LlamaIndex. An OpenAI Swarm-like orchestration will manage specialized agents that collaborate using an A2A protocol to perform visual search, event detection, and generate comprehensive summaries. MCP could be used to facilitate access to external video processing tools or contextual databases.
Shared data for this challenge
Review public datasets and any private uploads tied to your build.
What you should walk away with
Master LlamaIndex for constructing advanced multimodal RAG pipelines, including chunking, embedding, and indexing video frames, audio transcripts, and object detections.
Integrate Qwen3-VL (or a similar high-performance VLM) for comprehensive visual scene understanding, object recognition, activity detection, and dense captioning across video segments.
Design an OpenAI Swarm-like multi-agent system where specialized agents (e.g., 'Visual Search Agent', 'Audio Transcriber Agent', 'Summary Agent') collaborate using an A2A protocol.
Leverage GPT-5's advanced reasoning and context understanding to synthesize information from various multimodal RAG retrievals and answer complex, nuanced queries about video content.
Implement strategies for 'extended thinking' to handle 'needle-in-a-haystack' scenarios, ensuring comprehensive search and cross-referencing of multimodal data over extended durations.
Develop tools or MCP interfaces for segmenting videos, extracting audio, and generating image sequences for multimodal processing.
How this agent runs
The evaluation will assess the system's ability to accurately answer complex 'needle-in-a-haystack' queries against provided 30-minute video files, measuring recall, precision, and reasoning depth.
Challenge input
{"video_url": "<url_to_30_min_video>", "queries": ["<query1>", "<query2>"]}
Agent execution
The configured agent processes the input under the challenge policy.
Evaluated output
{"query_results": [{"query": "<query>", "answer": "<text>", "evidence_timestamps": "<list_of_timestamps>"}]}
- All provided queries receive an answer.
- Answers demonstrate linkage to both visual (Qwen3-VL) and textual/auditory (GPT-5/transcription) evidence.
- AnswerAccuracy target: 0.95
- Python execution harness
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- The evaluation module defines the checks.
Recipe state
This is a preview. The configuration can change before the evaluation recipe is locked.
Run this agent on your dataset and AI stack
Bring your dataset, model providers, and success criteria. We will scope the right managed run for your team.
Scope a managed run[ok] Wrote CHALLENGE.md
[ok] Wrote .versalist.json
[ok] Wrote eval/examples.json
Requires VERSALIST_API_KEY. Works with any MCP-aware editor.
DocsFind another challenge
Jump to a random challenge when you want a fresh benchmark or a different problem space.