Back to evaluations
Public evaluation

Automated Feature Development

The evaluation will assess the completeness, correctness, and security of the generated code, as well as the agents' ability to effectively collaborate and securely interact with simulated enterprise systems via Model Context Protocol (MCP).

Evaluation type
task based
Challenge
Automate Secure Enterprise Code
Difficulty
Advanced
Rigor
Unspecified
Open linked challenge

Evaluation overview

How the linked challenge is judged: tasks, benchmarks, and criteria count.

Tasks
2
Benchmarks
0
Criteria
0

Task templates

Inputs and expected outputs.

Task 1

Automated Feature Development

Given a software feature request, the agent team generates a Python function, writes unit tests, and integrates it into a mock repository.

Input format

{ 'feature_request': 'Develop a Python function to securely hash user passwords using a specified algorithm and salt.' }

Output format

{ 'generated_code': 'def hash_password(...)', 'unit_tests': 'import unittest...', 'repository_update_log': ['git add...', 'git commit...'] }

Task 2

MCP Secure Tool Usage Audit

Audit the agent's interactions with a simulated enterprise API for security and adherence to Model Context Protocol (MCP) guidelines.

Input format

{ 'agent_logs': ['MCP_CALL: git.push ...', 'API_CALL: secrets.get_key ...'], 'mcp_rules': { 'secrets.get_key': {'policy': 'encrypted_audit'} } }

Output format

{ 'audit_report': [{'event': 'secrets.get_key', 'status': 'compliant', 'justification': '...'}] }