Public evaluation

System Health Monitor

Evaluation of agent responsiveness and diagnostic accuracy.

Evaluation type

task based

Challenge

Real-Time AI Operations Controller with Claude Agents SDK

Difficulty

Advanced

Rigor

Unspecified

Open linked challenge

Evaluation overview

How the linked challenge is judged: tasks, benchmarks, and criteria count.

Tasks

Benchmarks

Criteria

Task templates

Inputs and expected outputs.

Task 1

System Health Monitor

Evaluates agent response time to synthetic anomaly triggers.

Input format

System logs JSON

Output format

JSON decision structure

Linked challenge

Real-Time AI Operations Controller with Claude Agents SDK

Build an intelligent operations monitoring system designed for real-time decision-making in large-scale tech infrastructure. Using the Claude Agents SDK, you will implement an agent capable of extended reasoning to monitor system health and resolve anomalies. This system will integrate LiveKit for real-time voice-based operational alerts and use Portkey to maintain high-precision evaluation pipelines and observability for agent decisions. You will incorporate Neurolink to manage the persistent state and context of the operational agent, while using Captum to analyze and debug model decision-making patterns during operation. This setup focuses on the high-stakes requirement of corporate infrastructure monitoring, mirroring concerns faced by platforms managing complex domain-blocking or enterprise policy adherence.

Open challenge