Build AI Dev Agent Teams
Amidst the booming AI coding market, this challenge focuses on developing sophisticated AI agent teams to automate web application development and deployment tasks. Participants will leverage the LangChain framework to orchestrate a multi-agent system, simulating a collaborative development team capable of generating code, performing quality checks, and interacting with stakeholders via voice. The system will demonstrate how advanced generative AI can streamline the software development lifecycle, from initial concept to deployment readiness.
What you are building
The core problem, expected build, and operating context for this challenge.
Orchestrate a LangChain multi-agent system with Claude 4 Opus for automated web development, integrating code generation, evaluation, and voice interaction.
How work is evaluated
The solution will be evaluated on the functionality of the multi-agent system, its ability to generate correct and robust code, and the effectiveness of its real-time communication interface. Performance and traceability via Coval metrics will also be assessed.
Challenge datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
Feature Files Present
All required files (HTML, CSS, JS, report) for the generated feature must exist and be accessible.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Coval Trace Completeness
Coval logs must show a complete execution trace, including agent transitions and tool calls.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Code Correctness Score
Automated linting and basic functionality tests on generated code (0-100). • target: 85 • range: 0-100
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Security Vulnerability Count
Number of critical security vulnerabilities identified by StarCoder 2 (lower is better). • target: 0 • range: 0-5
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Master LangChain's AgentExecutor and LangGraph for designing stateful, graph-based agent workflows that manage task decomposition and inter-agent communication.
Integrate Claude 4 Opus into LangChain agents to perform sophisticated code generation, architectural design, and problem-solving tasks for web applications.
Implement real-time, bidirectional voice communication for agent interfaces using LiveKit, enabling natural language interaction for task assignment and status updates.
Leverage StarCoder 2 within a LangChain tool-use agent for focused code quality analysis, vulnerability scanning, and automated refactoring suggestions.
Design an evaluation and observability pipeline using Coval to trace agent decisions, monitor performance metrics, and ensure adherence to development standards.
Build a 'Project Manager' agent that uses LangChain's planning capabilities to orchestrate 'Developer' and 'QA' agents for end-to-end web app feature delivery.
Develop custom tools for LangChain agents to interact with external systems, simulating version control and deployment platforms.
Reference links and supporting material
Sample data for 2 tasks
Challenge setup
The solution will be evaluated on the functionality of the multi-agent system, its ability to generate correct and robust code, and the effectiveness of its real-time communication interface. Performance and traceabil...
Challenge input
JSON object with 'prompt' string for feature description and 'output_path' string.
LangChain
Framework for building LLM applications
Claude 4 Opus
Policy Serving in the agent workflow.
Evaluated output
Directory containing generated HTML, CSS, JavaScript files, and a 'report.md' detailing the process.
- All required files (HTML, CSS, JS, report) for the generated feature must exist and be accessible.
- Coval logs must show a complete execution trace, including agent transitions and tool calls.
- Automated linting and basic functionality tests on generated code (0-100). • target: 85 • range: 0-100
- Code Correctness Score target: 85
- 2 public reference cases
- Python execution harness
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
- LangChain · Required
- Langchain · Optional
- Claude 4 Opus · Optional
- LangChain · Required
- Langchain · Optional
Evaluation contract
- Feature Files Present · Weight 1
- Coval Trace Completeness · Weight 1
- Code Correctness Score · Weight 1
- Security Vulnerability Count · Weight 1
Metrics
- Code Correctness Score · Target 85
- Security Vulnerability Count · Target 0
Reference cases
- 2 public reference cases
Workflow tasks
- Task 1
Input: JSON object with 'prompt' string for feature description and 'output_path' string.
Output: Directory containing generated HTML, CSS, JavaScript files, and a 'report.md' detailing the process.
Execution harness: Python
- Task 2
Input: Path to a generated code file (e.g., 'script.js').
Output: JSON object containing identified issues, severity, and suggested fixes.
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.