Skip to main content
Back to challenges
Challenge

Agentic Code Generation & Refinement

This challenge tasks you with building a robust AI agent using the OpenAI Agents SDK. Your agent will specialize in generating, debugging, and refining code snippets based on natural language prompts. It will simulate interaction with an IDE environment, leveraging external tools for code linting, static analysis, and version control operations. A key aspect is implementing MCP principles for structured tool integration, allowing the agent to dynamically select and utilize code-related services with clear input/output schemas. This project emphasizes advanced agentic design, tool orchestration, and the practical application of AI in developer workflows to enhance productivity and code quality.

Special Purpose AgentsHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Develop an OpenAI Agents SDK-based system to generate, debug, and refactor code using MCP for structured tool integration.

Delivery guide

How work is evaluated

Evaluation

The evaluation will assess the agent's ability to generate functional, high-quality code, its error-handling capabilities, and the correctness of its tool interactions using Trulens-Eval for tracing and metrics.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
4 scoring checks
Binary
4 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

CodeFunctionalityTest

The 'final_code' must pass a set of predefined unit tests for the given prompt.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

ToolUsageCorrectness

Trulens-Eval trace must show correct and relevant tool calls (e.g., linter, static analyzer) in refinement steps.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 3

CodeQualityScore

Score from SonarQube (or simulated linter) for the final code, higher is better. • target: 85 • range: 0-100

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 4

RefinementIterations

Number of iterations taken to achieve correct code, lower is better. • target: 2 • range: 1-5

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Master the OpenAI Agents SDK for defining agent behavior, tools, and multi-turn interactions.

  • Implement MCP-enabled tool integration with a simulated IDE interface for dynamic function calling to services like SonarQube or a custom linter.

  • Build a pipeline for code generation using advanced models like GPT-4o, focusing on contextual understanding and error handling.

  • Design and integrate a code debugging and refinement loop, allowing the agent to identify and fix issues iteratively.

  • Utilize Trulens-Eval for comprehensive observability and evaluation of agent reasoning paths and generated code quality.

  • Integrate with a mock GitHub Actions environment for simulating automated testing and deployment of generated code.

  • Understand and apply best practices for prompt engineering in code generation tasks to improve output accuracy and reduce hallucinations.

Resources and assets

Reference links and supporting material

Dataset notes

Sample data for 1 tasks

Challenge setup

The evaluation will assess the agent's ability to generate functional, high-quality code, its error-handling capabilities, and the correctness of its tool interactions using Trulens-Eval for tracing and metrics.

Challenge input

{'prompt': 'string', 'initial_code': 'string'}

Agent execution

No agent tool is listed for this challenge.

Evaluated output

{'final_code': 'string', 'iterations': 'integer', 'trace_id': 'string'}

Evaluation checks
  • The 'final_code' must pass a set of predefined unit tests for the given prompt.
  • Trulens-Eval trace must show correct and relevant tool calls (e.g., linter, static analyzer) in refinement...
  • Score from SonarQube (or simulated linter) for the final code, higher is better. • target: 85 • range: 0-100
Configured evaluation evidence
  • CodeQualityScore target: 85
  • 1 public reference case
Configured runtime
  • Python execution harness

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

No tool records are attached.

Evaluation contract

  • CodeFunctionalityTest · Weight 1
  • ToolUsageCorrectness · Weight 1
  • CodeQualityScore · Weight 1
  • RefinementIterations · Weight 1

Metrics

  • CodeQualityScore · Target 85
  • RefinementIterations · Target 2

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: {'prompt': 'string', 'initial_code': 'string'}

    Output: {'final_code': 'string', 'iterations': 'integer', 'trace_id': 'string'}

    Execution harness: Python

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Agentic Code Generation & Refinement