Skip to main content
Back to challenges
Challenge

Multi-Modal Edge AI for Defense with AutoGen and OpenAI o3/GPT-5

Challenge involves developing a sophisticated edge-to-cloud multi-modal agent system for real-time situational awareness. Participants will build a prototype where a simulated 'Edge Perception Agent' on a device processes visual and audio data, performing instant reasoning. This agent then securely communicates critical insights and requests for deeper analysis to a 'Cloud Command Agent' capable of advanced strategic reasoning and decision-making. The system must leverage AutoGen for orchestrating the heterogeneous agents and MCP for secure, efficient, and tool-integrated communication between the edge and cloud components. Focus on hybrid instant/deep reasoning, where quick local processing handles immediate threats, and a powerful cloud-based LLM is engaged for complex problem-solving based on aggregated or highly specific data.

Special Purpose AgentsHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Build an AutoGen multi-agent system for edge-to-cloud multi-modal perception and reasoning using OpenAI o3 (simulated) and GPT-5 with MCP integration.

Delivery guide

How work is evaluated

Evaluation

The evaluation will assess the system's ability to process multi-modal input at the edge, securely transmit relevant information to the cloud, and enable the Cloud Command Agent to make intelligent, context-aware decisions in simulated scenarios.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Learning goals

What you should walk away with

  • Implement a multi-modal data processing pipeline for simulated edge devices.

  • Integrate MCP for secure and efficient tool-enabled communication between edge and cloud agents.

  • Demonstrate real-time situational awareness and collaborative problem-solving across the edge-cloud continuum.

  • Design and orchestrate heterogeneous agents (edge and cloud) using AutoGen.

  • Develop a hybrid instant/deep reasoning architecture leveraging OpenAI o3 (simulated) and GPT-5.

Resources and assets

Reference links and supporting material

Dataset notes

Sample data for 2 tasks

Challenge setup

The evaluation will assess the system's ability to process multi-modal input at the edge, securely transmit relevant information to the cloud, and enable the Cloud Command Agent to make intelligent, context-aware deci...

Challenge input

JSON object: {'image_path': 'path/to/image.jpg', 'audio_path': 'path/to/audio.wav'}

Agent execution

No agent tool is listed for this challenge.

Evaluated output

JSON object: {'detected_objects': ['obj1', 'obj2'], 'audio_events': ['event1'], 'confidence': 0.X}

Evaluation checks
  • Verify successful and secure communication between edge and cloud agents via MCP.
  • Confirm that complex scenarios automatically trigger GPT-5's deep reasoning via the Cloud Command Agent.
  • Ensure edge processing time for simple detections meets a predefined threshold (e.g., <500ms).
Configured evaluation evidence
  • ObjectDetectionF1Score target: 0.85
Configured runtime
  • Docker execution harness

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

No tool records are attached.

Evaluation contract

  • Verify successful and secure communication between edge and cloud agents via MCP.
  • Confirm that complex scenarios automatically trigger GPT-5's deep reasoning via the Cloud Command Agent.
  • Ensure edge processing time for simple detections meets a predefined threshold (e.g., <500ms).

Metrics

  • ObjectDetectionF1Score · Target 0.85
  • ActionPlanRelevance · Target 90
  • SecurityAuditScore · Target 95

Workflow tasks

  1. Task 1

    Input: JSON object: {'image_path': 'path/to/image.jpg', 'audio_path': 'path/to/audio.wav'}

    Output: JSON object: {'detected_objects': ['obj1', 'obj2'], 'audio_events': ['event1'], 'confidence': 0.X}

    Execution harness: Docker

  2. Task 2

    Input: JSON object: {'edge_report': {'detected_objects': [...], 'audio_events': [...]}, 'mission_context': 'string', 'strategic_goals': ['goal1', 'goal2']}

    Output: Markdown string of a detailed action plan, including justification and risk assessment.

    Execution harness: Python

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Multi-Modal Edge AI for Defense with AutoGen and OpenAI o3/GPT-5