Automating Open Source Maintainer Support with AutoGen Multi-Agent Teams
Address the challenge of low-quality AI-generated pull requests by creating a multi-agent system using AutoGen. This system will act as a triage layer that evaluates contributions, checks for style guide compliance, and provides automated feedback to developers. By implementing a gatekeeper agent, an analyst agent, and a communicator agent, you can effectively filter noise while maintaining community standards. The project focuses on automated quality assurance and human-in-the-loop workflows to protect project integrity.
What you are building
The core problem, expected build, and operating context for this challenge.
Build an AutoGen-powered multi-agent team to triage and filter AI-generated contributions to open-source software projects.
How work is evaluated
Evaluation of triage accuracy and guardrail adherence
Challenge datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
Style Compliance
Verify code block triggers guardrail block
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Precision
Accuracy of triage classification • target: 0.9 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Master AutoGen group chat manager for orchestrating specialized contributor triage agents
Implement custom NVIDIA NeMo Guardrails to define rejection policies for low-effort code submissions
Design evaluation harnesses using Braintrust to score the effectiveness of the triage agents against human-verified ground truths
Deploy OpenHands agent sessions to automatically suggest refactors or style corrections for incoming contributions
Orchestrate Retell AI voice notifications to alert maintainers when high-priority, human-vetted issues require immediate review
Build a persistent message history handler to maintain state across multi-agent turns
Reference links and supporting material
Sample data for 1 tasks
Challenge setup
Evaluation of triage accuracy and guardrail adherence
Challenge input
JSON pull request metadata
OpenHands
Open-source software development agent.
AutoGen
Agent Systems · Multi-Agent Systems
Braintrust
Evaluation and tracing platform for AI apps.
Evaluated output
Classification score 0-1
- Verify code block triggers guardrail block
- Accuracy of triage classification • target: 0.9 • range: 0-1
- Precision target: 0.9
- 1 public reference case
- Python execution harness
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
- OpenHands · Optional
- AutoGen · Required
- OpenHands · Optional
- Braintrust · Optional
- Braintrust · Optional
- OpenHands · Optional
Evaluation contract
- Style Compliance · Weight 1
- Precision · Weight 1
Metrics
- Precision · Target 0.9
Reference cases
- 1 public reference case
Workflow tasks
- Task 1
Input: JSON pull request metadata
Output: Classification score 0-1
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.