Public evaluation

IoTAnomalyResponseSimulation

The evaluation will assess the neuro-symbolic agent's ability to accurately detect IoT anomalies and execute appropriate, safe, and context-aware responses in a simulated environment.

Evaluation type

task based

Challenge

Build a Neuro-Symbolic Agent for IoT Anomaly Response

Difficulty

Advanced

Rigor

Unspecified

Open linked challenge

Evaluation overview

How the linked challenge is judged: tasks, benchmarks, and criteria count.

Tasks

Benchmarks

Criteria

Task templates

Inputs and expected outputs.

Task 1

IoTAnomalyResponseSimulation

Simulate an IoT environment with an embedded anomaly and evaluate the agent's response.

Input format

{ "sensor_data_stream": [{"timestamp": "datetime", "sensor_id": "string", "value": "float"}], "anomaly_type": "string" }

Output format

{ "anomaly_detected": "boolean", "detected_anomaly_type": "string", "explanation": "string", "action_sequence": [{"device_id": "string", "command": "string", "parameters": "object"}], "adherence_to_safety_rules": "boolean" }

Linked challenge

Build a Neuro-Symbolic Agent for IoT Anomaly Response

Addressing the challenge of reasoning under perceptual uncertainty in real-world applications like IoT device operation, this challenge focuses on developing a neuro-symbolic agent. The agent will integrate the natural language understanding and high-level planning capabilities of an agent with a symbolic reasoning system. Its primary function will be to detect anomalies in simulated IoT sensor data streams and generate adaptive, context-aware responses to maintain system stability, akin to optimizing power distribution system restoration. Participants will need to design how continuous, often noisy, sensor data is translated into discrete symbolic facts, which then inform a symbolic planner. this will be crucial for enforcing schema-driven validation and ensuring the logical consistency and safety of the LLM's generated plans and actions. The goal is a robust system that can handle complex scenarios where a purely neural or symbolic approach might fall short, bridging the gap between continuous perception and discrete symbolic planning to enable intelligent, adaptive control.

Open challenge