Skip to main content
Back to challenges
Challenge

Accelerated Code Dev & Review Agent

Inspired by Claude's growing footprint in GitHub commits, this challenge focuses on building an advanced agentic development environment. You will use Mastra AI to orchestrate a team of agents that automate parts of the software development lifecycle, from generating code snippets based on user stories to automated testing and code review. The system should integrate with a simulated codebase, providing intelligent suggestions and even committing code. Emphasis is placed on code quality, security, and developer productivity.

Special Purpose AgentsHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Build an agentic development assistant using Mastra AI, Claude Sonnet 4, and Llama 3 8B Instruct for automated code generation, review, and continuous integration.

Delivery guide

How work is evaluated

Evaluation

The system will be evaluated on its ability to correctly implement a simple feature, generate appropriate tests, and provide insightful code reviews, all while adhering to specified coding standards. Evaluation will involve checking the generated code against requirements and analyzing agent logs for effective collaboration.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
5 scoring checks
Binary
5 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

CodeSyntacticallyCorrect

Generated code is syntactically valid Python.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

TestsPass

Generated unit tests pass against the generated function.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 3

PEP8Compliance

Refactored code adheres to PEP8 guidelines.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 4

CodeQualityScore

Automated score based on linting, complexity, and docstrings. • target: 85 • range: 0-100

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 5

FeatureCompleteness

Percentage of described features correctly implemented. • target: 95 • range: 0-100

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Master Mastra AI for defining agent roles, tools, and workflows, including its built-in memory and RAG capabilities for contextual code generation.

  • Integrate Claude Sonnet 4 for high-quality code generation and complex logical reasoning tasks, particularly for design patterns and architectural decisions.

  • Deploy Llama 3 8B Instruct via Hugging Face Inference Endpoints for highly optimized, specific code completion, syntax checking, and boilerplate generation.

  • Build custom tools within Mastra AI to interact with a mock Git repository and a simulated IDE (emulating Cursor's features) for reading, writing, and modifying code files.

  • Utilize Cohere's embedding models for semantic search over the codebase, enabling agents to quickly find relevant code examples, functions, or documentation for context.

  • Design an agent team where individual agents (e.g., 'Feature Developer Agent', 'Test Engineer Agent', 'Code Review Agent') collaborate using Mastra AI's messaging primitives.

  • Implement automated code quality checks and vulnerability scanning using a simulated or simplified code analysis tool.

Resources and assets

Reference links and supporting material

Dataset notes

Sample data for 2 tasks

Challenge setup

The system will be evaluated on its ability to correctly implement a simple feature, generate appropriate tests, and provide insightful code reviews, all while adhering to specified coding standards. Evaluation will i...

Challenge input

{'feature_description': 'string', 'language': 'string', 'test_framework': 'string'}

Agent execution

No agent tool is listed for this challenge.

Evaluated output

{'function_code': 'string', 'test_code': 'string', 'review_summary': 'string'}

Evaluation checks
  • Generated code is syntactically valid Python.
  • Generated unit tests pass against the generated function.
  • Refactored code adheres to PEP8 guidelines.
Configured evaluation evidence
  • CodeQualityScore target: 85
  • 2 public reference cases
Configured runtime
  • Python execution harness

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

No tool records are attached.

Evaluation contract

  • CodeSyntacticallyCorrect · Weight 1
  • TestsPass · Weight 1
  • PEP8Compliance · Weight 1
  • CodeQualityScore · Weight 1
  • FeatureCompleteness · Weight 1

Metrics

  • CodeQualityScore · Target 85
  • FeatureCompleteness · Target 95

Reference cases

  • 2 public reference cases

Workflow tasks

  1. Task 1

    Input: {'feature_description': 'string', 'language': 'string', 'test_framework': 'string'}

    Output: {'function_code': 'string', 'test_code': 'string', 'review_summary': 'string'}

    Execution harness: Python

  2. Task 2

    Input: {'code_snippet': 'string', 'refactoring_goal': 'string'}

    Output: {'refactored_code': 'string', 'review_comments': 'list', 'justification': 'string'}

    Execution harness: Python

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Accelerated Code Dev & Review Agent