Public evaluation

Object_Compliance_Check

Evaluation will focus on the agent's ability to accurately identify issues in simulated 3D scenes based on complex criteria, demonstrating effective multimodal reasoning and tool integration.

Evaluation type

task based

Challenge

Multimodal 3D Object Verification

Difficulty

Advanced

Rigor

Unspecified

Open linked challenge

Evaluation overview

How the linked challenge is judged: tasks, benchmarks, and criteria count.

Tasks

Benchmarks

Criteria

Task templates

Inputs and expected outputs.

Task 1

Object_Compliance_Check

Agent verifies if objects in a simulated 3D scene comply with a given design specification.

Input format

{'scene_description': 'json', 'design_spec': 'string'}

Output format

{'compliance_report': [{'object_id': 'string', 'status': 'compliant|non-compliant', 'discrepancy': 'string'}], 'overall_status': 'compliant|non-compliant', 'summary': 'string'}

Task 2

Complex_Scene_Analysis

Agent analyzes a more complex simulated 3D scene to detect subtle errors or omissions based on a detailed prompt.

Input format

{'scene_description': 'json', 'analysis_prompt': 'string'}

Output format

{'detailed_findings': 'string', 'recommended_actions': ['string'], 'confidence_score': 'float'}