PyTorch Model Bottleneck Analysis and Optimization
The evaluation will assess the agent's capability to analyze, optimize, and generate PyTorch code effectively, focusing on performance gains, code quality, and the accuracy of explanations and tool interactions.
Evaluation overview
How the linked challenge is judged: tasks, benchmarks, and criteria count.
Task templates
Inputs and expected outputs.
PyTorch Model Bottleneck Analysis and Optimization
Given a sub-optimal PyTorch training loop and model definition, the agent must identify performance bottlenecks (e.g., inefficient data loading, sub-optimal tensor operations) and suggest concrete code-level optimizations. The agent should be able to interact with a mock profiler tool and explain its findings and proposed changes.
{'pytorch_code': 'string', 'optimization_goal': 'lower inference latency'}
{'bottlenecks_identified': ['string'], 'proposed_optimizations': {'code_changes': 'string', 'explanation': 'string'}, 'tool_interactions_log': 'string'}
Generate PyTorch Module for Specific Task with RAG Context
Generate a PyTorch module for a novel task (e.g., a custom attention mechanism) given a high-level description and specific performance constraints. The agent must use RAG to incorporate best practices from relevant research papers or existing code examples and generate well-commented, performant code. It should then use a mock linter tool to verify code quality.
{'task_description': 'string', 'performance_constraints': {'memory_limit_mb': 'int', 'target_flops': 'float'}, 'rag_context_keywords': ['attention mechanism', 'efficient pytorch']}
{'generated_pytorch_module': 'string', 'design_justification': 'string', 'linter_report_status': 'string'}