Agentic Video Scene Skipper
This challenge involves building an advanced agentic system that can interpret complex natural language requests to navigate video content. You will leverage Gemini 3 Pro's multimodal understanding and Langroid's robust agent capabilities to process user queries, perform semantic search over video metadata, and execute simulated playback commands. The system must accurately identify specific scenes based on descriptions, character names, or quotes, demonstrating sophisticated hybrid reasoning and MCP tool integration for real-time control of a simulated media player. This project focuses on combining cutting-edge LLMs with specialized agent frameworks and advanced RAG techniques. You will design a graph-based workflow for parsing queries, retrieving relevant video segments, and interacting with external tools, simulating a highly responsive and intelligent content navigation system. Success will require meticulous prompt engineering, efficient data indexing, and robust error handling to deliver a seamless user experience.
What you are building
The core problem, expected build, and operating context for this challenge.
Build a Langroid agent with Gemini 3 Pro for intelligent video scene navigation using LlamaIndex-powered RAG and MCP tool integration.
How work is evaluated
The evaluation will assess the agent's ability to accurately identify and 'skip' to video scenes based on diverse natural language queries. It will measure the precision of scene identification, the correctness of simulated API calls, and the robustness of the RAG pipeline.
Challenge datasets
Public datasets, plus the private datasets you have access to.
What you should walk away with
Utilize Gemini 2.5 Pro for multimodal understanding of video queries.
Design MCP-enabled tool integration for a simulated video player API.
Build extended thinking patterns for complex scene identification.
Master Langroid for developing stateful, multi-step conversational agents.
Implement advanced RAG pipelines with LlamaIndex for video content.
Reference links and supporting material
Sample data for 3 tasks
Challenge setup
The evaluation will assess the agent's ability to accurately identify and 'skip' to video scenes based on diverse natural language queries. It will measure the precision of scene identification, the correctness of sim...
Challenge input
{'video_id': 'string', 'title': 'string', 'description': 'string', 'scenes': [{'start_time': 'float', 'end_time': 'float', 'description': 'string',...
Agent execution
No agent tool is listed for this challenge.
Evaluated output
{'status': 'success', 'indexed_scenes_count': 'int', 'vector_db_ready': 'boolean'}
- Verify that all simulated API calls to the video player were successful.
- Ensure at least 80% of identified scene times fall within the expected time ranges (+/- 5 seconds buffer).
- AverageSceneIdentificationLatency target: 1500
- Python execution harness
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- Verify that all simulated API calls to the video player were successful.
- Ensure at least 80% of identified scene times fall within the expected time ranges (+/- 5 seconds buffer).
Metrics
- AverageSceneIdentificationLatency · Target 1500
- RAGRecallScore · Target 0.9
Workflow tasks
- Task 1
Input: {'video_id': 'string', 'title': 'string', 'description': 'string', 'scenes': [{'start_time': 'float', 'end_time': 'float', 'description': 'string', 'keywords': ['string']}], 'transcript': 'string'}
Output: {'status': 'success', 'indexed_scenes_count': 'int', 'vector_db_ready': 'boolean'}
Execution harness: Python
- Task 2
Input: {'video_content_schema': 'json', 'simulated_player_api_spec': 'json'}
Output: {'agent_initialized': 'boolean', 'tool_bindings_verified': 'boolean'}
Execution harness: Python
- Task 3
Input: {'queries': ['string'], 'expected_scene_times': [{'query': 'string', 'expected_time_range': [float, float]}]}
Output: {'results': [{'query': 'string', 'identified_scene_time': 'float', 'api_call_success': 'boolean', 'reasoning_trace': 'string'}]}
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.