Multi-Agent System for Internal Security Anomaly Detection
This challenge focuses on building a sophisticated multi-agent system using AutoGen to detect potential data leaks or anomalous behavior. Participants will design and implement a collaborative team of AI agents capable of monitoring internal communication logs, system access records, and cross-referencing this data with external news feeds or public information. The system will identify patterns and anomalies that might indicate security incidents or insider threats. The core of the challenge involves orchestrating diverse agents, each with specific roles like 'Log Monitor', 'News Analyst', 'Incident Investigator', and 'Reporting Agent'. These agents will communicate and collaborate autonomously, using o4-mini for reasoning and specific tools to interact with simulated data sources. The goal is to build an intelligent, proactive security monitoring system that can identify subtle indicators of risk and present a consolidated, actionable report.
What you are building
The core problem, expected build, and operating context for this challenge.
Build an AutoGen multi-agent system using o4-mini to detect internal security anomalies by cross-referencing simulated logs and external news.
How work is evaluated
The evaluation will assess the multi-agent system's ability to accurately detect and report simulated security anomalies, its robustness in handling diverse data, and the clarity of its output.
Shared data for this challenge
Review public datasets and any private uploads tied to your build.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
Correct Anomaly Identification
The 'anomaly_detected' flag must be true for positive cases and false for negative cases.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Report Clarity
The 'report_summary' must clearly describe the anomaly and contributing factors.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Confidence Score
The reported confidence in the anomaly detection. • target: 0.8 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Master AutoGen for building complex, conversational multi-agent systems with shared context and human-in-the-loop capabilities.
Implement role-based agent collaboration patterns in AutoGen, defining specialized agents like Log Monitor, News Analyst, and Security Investigator.
Integrate o4-mini models into AutoGen agents for advanced reasoning, natural language processing, and pattern recognition tasks.
Design and implement custom tools for AutoGen agents to interact with simulated internal access logs, email archives, and external news APIs.
Utilize FLAML within AutoGen workflows for automated hyperparameter tuning and efficient resource management for agent-based tasks.
Develop reporting mechanisms using All Hands AI for summarizing security incidents and communicating findings to human operators.
Apply CodeRabbit principles for ensuring code quality and best practices in the AutoGen agent codebase, emphasizing maintainability and security.
Explore Neurolink patterns for designing resilient and adaptive agent systems capable of handling dynamic security threat landscapes.
Reference links and supporting material
Sample data for 1 tasks
How this agent runs
The evaluation will assess the multi-agent system's ability to accurately detect and report simulated security anomalies, its robustness in handling diverse data, and the clarity of its output.
Challenge input
{'access_logs': 'list of strings', 'email_samples': 'list of strings', 'news_articles': 'list of strings'}
Agent execution
The configured agent processes the input under the challenge policy.
Evaluated output
{'anomaly_detected': 'boolean', 'report_summary': 'string', 'confidence_score': 'float'}
- The 'anomaly_detected' flag must be true for positive cases and false for negative cases.
- The 'report_summary' must clearly describe the anomaly and contributing factors.
- The reported confidence in the anomaly detection. • target: 0.8 • range: 0-1
- Confidence Score target: 0.8
- 1 public reference case
- Python execution harness
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- Correct Anomaly Identification · Weight 1
- Report Clarity · Weight 1
- Confidence Score · Weight 1
Recipe state
This is a preview. The configuration can change before the evaluation recipe is locked.