Skip to main content
Back to challenges
Challenge

OSHA Plant Safety Audit Multi-Agent System with AutoGen

Manufacturing facilities face heavy penalties and worker safety hazards when asset management practices diverge from OSHA compliance standards. Construct an automated safety compliance system using Microsoft AutoGen to cross-examine machinery asset logs against regulatory safety standards and output actionable audit reports.

EngineeringHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Build an interactive multi-agent conversation framework using AutoGen to execute OSHA regulatory compliance audits on factory maintenance records.

Delivery guide

How work is evaluated

Evaluation

Evaluates multi-agent AutoGen debate output for accurate detection of OSHA LOTO protocol violations.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

citation_accuracy

Verifies correct OSHA 1910 standard is cited for LOTO failure.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

hazard_detection_recall

Recall metric of non-compliant maintenance logs identified • target: 0.98 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Set up AutoGen ConversableAgents representing Auditor, Plant Engineer, and Safety Regulator

  • Parse equipment maintenance histories, Lockout/Tagout (LOTO) logs, and PPE compliance records

  • Identify OSHA standards non-compliance (e.g., 29 CFR 1910.147 Control of Hazardous Energy)

  • Synthesize unified plant safety remediation plans with priority hazard indices

Resources and assets

Reference links and supporting material

Dataset notes

Factory maintenance logs containing normal servicing actions and safety protocol omissions.

Challenge setup

Evaluates multi-agent AutoGen debate output for accurate detection of OSHA LOTO protocol violations.

Challenge input

JSON object with asset_id, action_logged, loto_applied, ppe_verified

AutoGen

Microsoft framework for building multi-agent conversational systems

Evaluated output

JSON object with compliant (boolean), citation (string), risk_level (string)

Evaluation checks
  • Verifies correct OSHA 1910 standard is cited for LOTO failure.
  • Recall metric of non-compliant maintenance logs identified • target: 0.98 • range: 0-1
Configured evaluation evidence
  • Benchmark: OSHA-SafetyAudit-2025
  • Hazard Detection Recall target: 0.98
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

Action Space
  • AutoGen · Required
  • Microsoft AutoGen · Optional

Evaluation contract

  • citation_accuracy · Weight 1
  • hazard_detection_recall · Weight 1

Metrics

  • hazard_detection_recall · Target 0.98

Benchmarks

  • OSHA-SafetyAudit-2025
  • security

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: JSON object with asset_id, action_logged, loto_applied, ppe_verified

    Output: JSON object with compliant (boolean), citation (string), risk_level (string)

    Execution harness: Python

Evaluation tools

  • AutoGen · agent_framework

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about OSHA Plant Safety Audit Multi-Agent System with AutoGen