Multimodal 3D Object Verification
Leveraging the advancements in multimodal AI and 3D vision models like Meta's SAM 3D, this challenge tasks you with building a multimodal agent system using Semantic Kernel. Your agent will act as a '3D Model Quality Assurance' specialist. It will receive a natural language request along with a simulated '3D scene description' (derived from SAM 3D output) and verify if objects within the scene meet specified criteria. The Gemini 2.5 Pro model will be at the core, orchestrating visual analysis tools (simulated APIs for SAM 3D) and performing extended reasoning to identify discrepancies or compliance issues.
What you are building
The core problem, expected build, and operating context for this challenge.
Develop a multimodal agent using Semantic Kernel and Gemini 2.5 Pro for 3D object verification, integrating with simulated SAM 3D tools.
How work is evaluated
Evaluation will focus on the agent's ability to accurately identify issues in simulated 3D scenes based on complex criteria, demonstrating effective multimodal reasoning and tool integration.
Challenge datasets
Public datasets, plus the private datasets you have access to.
What you should walk away with
Implement a multimodal agent capable of interpreting natural language and visual data.
Develop a system for verifying compliance of 3D objects against specified criteria.
Design extended thinking pipelines for complex visual reasoning and problem-solving.
Integrate simulated 3D vision tools (e.g., SAM 3D API) using Semantic Kernel's plugin architecture.
Utilize Gemini 2.5 Pro's multimodal capabilities for scene understanding and discrepancy detection.
Reference links and supporting material
Sample data for 2 tasks
Challenge setup
Evaluation will focus on the agent's ability to accurately identify issues in simulated 3D scenes based on complex criteria, demonstrating effective multimodal reasoning and tool integration.
Challenge input
{'scene_description': 'json', 'design_spec': 'string'}
Agent execution
No agent tool is listed for this challenge.
Evaluated output
{'compliance_report': [{'object_id': 'string', 'status': 'compliant|non-compliant', 'discrepancy': 'string'}], 'overall_status': 'compliant|non-com...
- Ensure the compliance report and findings adhere to the specified JSON structure.
- Verify that the simulated SAM 3D tool was logically invoked and its output processed.
- Discrepancy Detection Accuracy target: 0.9
- Python execution harness
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- Ensure the compliance report and findings adhere to the specified JSON structure.
- Verify that the simulated SAM 3D tool was logically invoked and its output processed.
Metrics
- Discrepancy_Detection_Accuracy · Target 0.9
- False_Positive_Rate · Target 0.1
- Reasoning_Coherence · Target 0.8
Workflow tasks
- Task 1
Input: {'scene_description': 'json', 'design_spec': 'string'}
Output: {'compliance_report': [{'object_id': 'string', 'status': 'compliant|non-compliant', 'discrepancy': 'string'}], 'overall_status': 'compliant|non-compliant', 'summary': 'string'}
Execution harness: Python
- Task 2
Input: {'scene_description': 'json', 'analysis_prompt': 'string'}
Output: {'detailed_findings': 'string', 'recommended_actions': ['string'], 'confidence_score': 'float'}
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.