Public evaluation

AnomalyDetectionAccuracy

The evaluation will assess the accuracy of anomaly detection, the effectiveness of the interactive agent, and the quality and relevance of the AI-generated explanations. Participants will provide their Python code and a demonstration of the interactive system.

Evaluation type

task based

Challenge

AI-Powered SSA: Anomaly Detection with Gemma and Explainable AI

Difficulty

Advanced

Rigor

Unspecified

Open linked challenge

Evaluation overview

How the linked challenge is judged: tasks, benchmarks, and criteria count.

Tasks

Benchmarks

Criteria

Task templates

Inputs and expected outputs.

Task 1

AnomalyDetectionAccuracy

Evaluates the precision, recall, and F1-score of the anomaly detection model on a hidden test dataset of synthetic orbital telemetry with known anomalies.

Input format

JSON array of time-series orbital telemetry data (e.g., {'timestamp': ..., 'sma': ..., 'inc': ..., 'ecc': ...}) with a 'true_anomaly_label' field.

Output format

JSON array of predictions for each data point (0 for normal, 1 for anomaly) and an associated confidence score.

Task 2

ExplanationQuality

Evaluates the relevance, coherence, and actionable insights of the explanations generated by the Anthropic API via the Langroid agent for specific detected anomalies.

Input format

JSON object containing a detected anomaly's features and metadata (e.g., {'anomaly_id': 'AX123', 'features': {'sma_change': 50, 'inc_change': 0.1}, 'detection_time': '2025-11-28T10:00:00Z'}).

Output format

JSON object with a 'explanation_text' field (string) and 'action_recommendations' (array of strings).

Linked challenge

AI-Powered SSA: Anomaly Detection with Gemma and Explainable AI

The increasing congestion of orbital space necessitates advanced Space Situational Awareness (SSA) capabilities to identify anomalous object behavior, predict potential collisions, obtain insights into mission profiles, and assess potential threats. This challenge focuses on developing an intelligent system that uses synthetic or real-world space object telemetry to detect and categorize anomalous events. Participants will design, implement, and evaluate a machine learning pipeline capable of identifying unusual orbital maneuvers, conjunction events, or non-standard satellite operations. The system should go beyond simple detection by leveraging large language models to provide actionable, explainable insights into the nature and potential implications of detected anomalies, crucial for both civil and defense space tracking efforts. The solution will integrate a robust anomaly detection model, potentially built with or informed by Gemma 2 principles, processing diverse sensor data (e.g., TLEs, radar, optical observations). Furthermore, it will utilize the Langroid framework to build an interactive agent that queries the anomaly detection system and uses the Anthropic API to generate rich, context-aware explanations for identified anomalies. This will enable human operators to quickly understand complex situations and make informed decisions, directly supporting enhanced Space Situational Awareness and threat assessment capabilities.

Open challenge