Back to evaluations
Public evaluation

HardwareOptimizationRecommendation

The system will be evaluated on its ability to generate optimized deployment configurations and code, demonstrate effective agent collaboration, and utilize MCP-enabled tools to achieve specific performance and cost targets for AI model inference on target hardware.

Evaluation type
task based
Challenge
Gemini 2.5 Pro & AutoGen for AI Hardware Optimization Agents
Difficulty
Advanced
Rigor
Unspecified
Open linked challenge

Evaluation overview

How the linked challenge is judged: tasks, benchmarks, and criteria count.

Tasks
2
Benchmarks
0
Criteria
0

Task templates

Inputs and expected outputs.

Task 1

HardwareOptimizationRecommendation

Evaluates the agents' ability to recommend an optimized hardware configuration and deployment strategy for a given AI model.

Input format

{'ai_model_spec': {'name': 'string', 'parameters_count': 'int', 'inference_latency_target_ms': 'int'}, 'available_hardware_profiles': [{'id': 'string', 'type': 'string', 'compute_gflops': 'int', 'memory_gb': 'int', 'cost_per_hour_usd': 'float'}]}

Output format

{'recommended_hardware_id': 'string', 'deployment_strategy': 'string', 'estimated_latency_ms': 'int', 'estimated_cost_per_hour_usd': 'float', 'optimization_notes': ['string']}

Task 2

CodeGenerationForOptimization

Evaluates the 'Code Optimizer' agent's ability to generate relevant code snippets for performance tuning based on identified bottlenecks.

Input format

{'bottleneck_description': 'string', 'programming_language': 'string', 'target_hardware_spec': 'dict'}

Output format

{'generated_code_snippet': 'string', 'explanation': 'string', 'performance_gain_estimate_percent': 'float'}