Human-AI Collaborative Data Engineering Pipeline
Data engineering teams require active human oversight when AI agents transform public datasets. Create an automated human-in-the-loop pipeline that cleans public datasets, flags data quality anomalies, and routes uncertain transformations to human engineers.
What you are building
The core problem, expected build, and operating context for this challenge.
Develop an interactive human-in-the-loop data pipeline for automated data cleaning and anomaly triage.
How work is evaluated
Evaluates pipeline ability to flag ambiguous records for human review vs auto-processing clear records.
Shared data for this challenge
Review public datasets and any private uploads tied to your build.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
correct_triage
Verifies that invalid age triggers human review
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
triage_precision
Precision of routing corrupt data to human review • target: 0.95 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Build dynamic data quality profiling scripts
Implement human reviewer decision handoff interfaces
Automate target ETL loads post-human verification
Reference links and supporting material
Synthetic public service request records containing intentional noise and edge cases.
How this agent runs
Evaluates pipeline ability to flag ambiguous records for human review vs auto-processing clear records.
Challenge input
JSON dataset record with missing or corrupted fields
Great Expectations
Automated data validation expectations
Evaluated output
JSON containing action (AUTO_CLEAN/HUMAN_REVIEW) and confidence_score
- Verifies that invalid age triggers human review
- Precision of routing corrupt data to human review • target: 0.95 • range: 0-1
- Benchmark: DataQuality-Bench
- Triage Precision target: 0.95
- 1 public reference case
- Python execution harness
- Python sandbox
View technical recipe
Configured tools
- Evidently AI · Required
- Shakudo · Optional
- Haystack · Optional
Evaluation contract
- correct_triage · Weight 1
- triage_precision · Weight 1
Recipe state
This is a preview. The configuration can change before the evaluation recipe is locked.