Back to evaluations
Public evaluation

ScenarioGenerationAndRiskAssessment

The system will be evaluated on its ability to effectively simulate AI safety red-teaming, generate plausible threat scenarios, identify vulnerabilities, and propose concrete mitigation strategies, focusing on the sophistication of agent interactions and adaptive reasoning.

Evaluation type
task based
Challenge
Red-Team AI Safety with GPT-5 and LangGraph Adaptive Thinking Agents
Difficulty
Advanced
Rigor
Unspecified
Open linked challenge

Evaluation overview

How the linked challenge is judged: tasks, benchmarks, and criteria count.

Tasks
2
Benchmarks
0
Criteria
0

Task templates

Inputs and expected outputs.

Task 1

ScenarioGenerationAndRiskAssessment

Evaluates the Red Team's ability to generate a novel AI deployment scenario and identify potential catastrophic risks.

Input format

{'ai_system_description': 'string', 'contextual_parameters': ['string']}

Output format

{'scenario_title': 'string', 'threat_vector': 'string', 'catastrophic_risk_score': 'float', 'identified_vulnerabilities': ['string'], 'reasoning_path': ['string']}

Task 2

MitigationStrategyDevelopment

Evaluates the Blue Team's ability to propose effective and comprehensive mitigation strategies for an identified threat.

Input format

{'threat_scenario': {'scenario_title': 'string', 'threat_vector': 'string', 'identified_vulnerabilities': ['string']}, 'budget_constraints': {'time_days': 'int', 'resources_usd': 'int'}}

Output format

{'mitigation_plan_summary': 'string', 'detailed_actions': [{'action': 'string', 'priority': 'string', 'estimated_cost_usd': 'int', 'effectiveness_score': 'float'}], 'adaptive_thinking_usage': 'dict'}