Skip to main content
Back to challenges
Challenge

Multi-Agent System for Internal Security Anomaly Detection

This challenge focuses on building a sophisticated multi-agent system using AutoGen to detect potential data leaks or anomalous behavior. Participants will design and implement a collaborative team of AI agents capable of monitoring internal communication logs, system access records, and cross-referencing this data with external news feeds or public information. The system will identify patterns and anomalies that might indicate security incidents or insider threats. The core of the challenge involves orchestrating diverse agents, each with specific roles like 'Log Monitor', 'News Analyst', 'Incident Investigator', and 'Reporting Agent'. These agents will communicate and collaborate autonomously, using o4-mini for reasoning and specific tools to interact with simulated data sources. The goal is to build an intelligent, proactive security monitoring system that can identify subtle indicators of risk and present a consolidated, actionable report.

Special Purpose AgentsHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Build an AutoGen multi-agent system using o4-mini to detect internal security anomalies by cross-referencing simulated logs and external news.

Delivery guide

How work is evaluated

Evaluation

The evaluation will assess the multi-agent system's ability to accurately detect and report simulated security anomalies, its robustness in handling diverse data, and the clarity of its output.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
3 scoring checks
Binary
3 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

Correct Anomaly Identification

The 'anomaly_detected' flag must be true for positive cases and false for negative cases.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

Report Clarity

The 'report_summary' must clearly describe the anomaly and contributing factors.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 3

Confidence Score

The reported confidence in the anomaly detection. • target: 0.8 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Master AutoGen for building complex, conversational multi-agent systems with shared context and human-in-the-loop capabilities.

  • Implement role-based agent collaboration patterns in AutoGen, defining specialized agents like Log Monitor, News Analyst, and Security Investigator.

  • Integrate o4-mini models into AutoGen agents for advanced reasoning, natural language processing, and pattern recognition tasks.

  • Design and implement custom tools for AutoGen agents to interact with simulated internal access logs, email archives, and external news APIs.

  • Utilize FLAML within AutoGen workflows for automated hyperparameter tuning and efficient resource management for agent-based tasks.

  • Develop reporting mechanisms using All Hands AI for summarizing security incidents and communicating findings to human operators.

  • Apply CodeRabbit principles for ensuring code quality and best practices in the AutoGen agent codebase, emphasizing maintainability and security.

  • Explore Neurolink patterns for designing resilient and adaptive agent systems capable of handling dynamic security threat landscapes.

Resources and assets

Reference links and supporting material

Dataset notes

Sample data for 1 tasks

Challenge setup

The evaluation will assess the multi-agent system's ability to accurately detect and report simulated security anomalies, its robustness in handling diverse data, and the clarity of its output.

Challenge input

{'access_logs': 'list of strings', 'email_samples': 'list of strings', 'news_articles': 'list of strings'}

Agent execution

No agent tool is listed for this challenge.

Evaluated output

{'anomaly_detected': 'boolean', 'report_summary': 'string', 'confidence_score': 'float'}

Evaluation checks
  • The 'anomaly_detected' flag must be true for positive cases and false for negative cases.
  • The 'report_summary' must clearly describe the anomaly and contributing factors.
  • The reported confidence in the anomaly detection. • target: 0.8 • range: 0-1
Configured evaluation evidence
  • Confidence Score target: 0.8
  • 1 public reference case
Configured runtime
  • Python execution harness

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

No tool records are attached.

Evaluation contract

  • Correct Anomaly Identification · Weight 1
  • Report Clarity · Weight 1
  • Confidence Score · Weight 1

Metrics

  • Confidence Score · Target 0.8

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: {'access_logs': 'list of strings', 'email_samples': 'list of strings', 'news_articles': 'list of strings'}

    Output: {'anomaly_detected': 'boolean', 'report_summary': 'string', 'confidence_score': 'float'}

    Execution harness: Python

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Multi-Agent System for Internal Security Anomaly Detection