IncidentAnalysisAndProcedureGeneration
The evaluation will assess the Claude agent's ability to accurately analyze robotaxi incidents, correctly interpret safety policies, and generate effective, compliant, and safe operational procedures. It will emphasize the agent's reasoning, 'computer use' for document interaction, and the quality of its output.
Evaluation overview
How the linked challenge is judged: tasks, benchmarks, and criteria count.
Task templates
Inputs and expected outputs.
IncidentAnalysisAndProcedureGeneration
Evaluates the agent's response to a simulated robotaxi incident, requiring analysis of incident details and generation of a relevant, updated safety procedure.
{ "incident_report": "string", "relevant_policies": "array" }
{ "analysis_summary": "string", "root_causes_identified": "array", "generated_procedure": "string", "policy_references": "array" }
RegulatoryComplianceCheck
Tests the agent's ability to interpret new or updated regulations and identify potential compliance gaps in existing procedures.
{ "new_regulation_text": "string", "existing_procedure_text": "string" }
{ "compliance_status": "string", "identified_gaps": "array", "recommended_updates": "string" }