Multi-Modal Edge AI for Defense with AutoGen and OpenAI o3/GPT-5
Challenge involves developing a sophisticated edge-to-cloud multi-modal agent system for real-time situational awareness. Participants will build a prototype where a simulated 'Edge Perception Agent' on a device processes visual and audio data, performing instant reasoning. This agent then securely communicates critical insights and requests for deeper analysis to a 'Cloud Command Agent' capable of advanced strategic reasoning and decision-making. The system must leverage AutoGen for orchestrating the heterogeneous agents and MCP for secure, efficient, and tool-integrated communication between the edge and cloud components. Focus on hybrid instant/deep reasoning, where quick local processing handles immediate threats, and a powerful cloud-based LLM is engaged for complex problem-solving based on aggregated or highly specific data.
What you are building
Build an AutoGen multi-agent system for edge-to-cloud multi-modal perception and reasoning using OpenAI o3 (simulated) and GPT-5 with MCP integration.
How work is evaluated
The evaluation will assess the system's ability to process multi-modal input at the edge, securely transmit relevant information to the cloud, and enable the Cloud Command Agent to make intelligent, context-aware decisions in simulated scenarios.
Datasets
Public datasets, plus the private datasets you have access to.
Learning goals
Implement a multi-modal data processing pipeline for simulated edge devices.
Integrate MCP for secure and efficient tool-enabled communication between edge and cloud agents.
Demonstrate real-time situational awareness and collaborative problem-solving across the edge-cloud continuum.
Design and orchestrate heterogeneous agents (edge and cloud) using AutoGen.
Develop a hybrid instant/deep reasoning architecture leveraging OpenAI o3 (simulated) and GPT-5.
Resources
Sample data for 2 tasks
Challenge setup
The evaluation will assess the system's ability to process multi-modal input at the edge, securely transmit relevant information to the cloud, and enable the Cloud Command Agent to make intelligent, context-aware deci...
Challenge input
JSON object: {'image_path': 'path/to/image.jpg', 'audio_path': 'path/to/audio.wav'}
Agent execution
No agent tool is listed for this challenge.
Evaluated output
JSON object: {'detected_objects': ['obj1', 'obj2'], 'audio_events': ['event1'], 'confidence': 0.X}
- Verify successful and secure communication between edge and cloud agents via MCP.
- Confirm that complex scenarios automatically trigger GPT-5's deep reasoning via the Cloud Command Agent.
- Ensure edge processing time for simple detections meets a predefined threshold (e.g., <500ms).
- ObjectDetectionF1Score target: 0.85
- Docker execution harness
Shows the first task, up to three tools, and up to three evidence items.
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- Verify successful and secure communication between edge and cloud agents via MCP.
- Confirm that complex scenarios automatically trigger GPT-5's deep reasoning via the Cloud Command Agent.
- Ensure edge processing time for simple detections meets a predefined threshold (e.g., <500ms).
Metrics
- ObjectDetectionF1Score · Target 0.85
- ActionPlanRelevance · Target 90
- SecurityAuditScore · Target 95
Workflow tasks
- Task 1
Input: JSON object: {'image_path': 'path/to/image.jpg', 'audio_path': 'path/to/audio.wav'}
Output: JSON object: {'detected_objects': ['obj1', 'obj2'], 'audio_events': ['event1'], 'confidence': 0.X}
Execution harness: Docker
- Task 2
Input: JSON object: {'edge_report': {'detected_objects': [...], 'audio_events': [...]}, 'mission_context': 'string', 'strategic_goals': ['goal1', 'goal2']}
Output: Markdown string of a detailed action plan, including justification and risk assessment.
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.