Secure Code Generation for a Defined Task
The generated code will be evaluated for correctness, adherence to specifications, security vulnerabilities (via static analysis), and passing unit tests. The efficiency of the DSPy pipeline and MCP tool integration will also be assessed.
Evaluation overview
How the linked challenge is judged: tasks, benchmarks, and criteria count.
Task templates
Inputs and expected outputs.
Secure Code Generation for a Defined Task
Generate Python code for a specific, security-sensitive task (e.g., secure data serialization, cryptographic utility) based on provided specifications.
{'task_description': 'str', 'security_constraints': 'list_of_strings', 'language': 'str'}
{'generated_code': 'str', 'static_analysis_report': 'dict', 'unit_test_results': 'dict', 'agent_reasoning_log': 'list'}
Code Refinement Based on Test Feedback
Given a code snippet and failing unit test results, use the agent to refine the code until all tests pass.
{'initial_code': 'str', 'failing_tests': 'dict', 'task_context': 'str'}
{'refined_code': 'str', 'final_test_results': 'dict', 'refinement_iterations': 'int'}