Build a Fraud Model Change Review and Shadow Operations Console
Compare a candidate risk model with an existing model on chronological data. Give reviewers calibration, workload, subgroup uncertainty, and rollback evidence before simulated promotion.
What you are building
Would a candidate model still deserve promotion if its ranking held up but its probabilities became unreliable? Build a shadow comparison using chronological outcomes. Report calibration, investigator workload, subgroup uncertainty, and rollback criteria. The starter probabilities of 0.1 and 0.9 against outcomes zero and one have mean squared error of 0.01. Do not promote a live model.
Rules
The build includes an operational review workflow, not only a leaderboard or a claim of fair production deployment. No fabricated records, source citations, professional endorsements, or expert-adjudication claims. No future-data leakage into dated decisions, model fitting, or historical valuation inputs. No unauthorized payments, account restrictions, regulatory filings, or production changes. No automated appraisal certification, tax-compliance determination, or unsupported fraud accusation. No concealed uncertainty or silently invented inputs required for a numerical conclusion.
How work is evaluated
100-point weighted review with safety gates. Visible starter cases do not establish full application correctness.
A versioned shadow-scoring workflow. A model-change record linking data, metrics, threshold decisions, and reviewer signoff. Runnable local application, setup commands, and a dependency lockfile. Versioned input data, data dictionary, acquisition instructions, and rights notes. Calculation traces, source references, assumption records, and exported review artifacts. A deterministic baseline and tests for normal, incomplete, contradictory, and failure-recovery cases. A recorded complete workflow with a reproducible manifest and explicit limitations.
Datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
Learning goals
Freeze chronological training, calibration, and test partitions.
Compare shadow scores under fixed review-capacity policies.
Inspect calibration, subgroup uncertainty, and distribution shifts.
Export a model-change decision packet with rollback conditions.
Resources
2 attached datasets, including two visible synthetic starter cases. The fixture ZIP supplies synthetic exercise inputs. It does not supply independently verified real-world records.
Challenge setup
Challenge input
Input details are not configured.
Agent execution
No agent tool is listed for this challenge.
Evaluated output
No output checks are configured.
- Rubric dimension 1
- Rubric dimension 2
- Rubric dimension 3
- 2 public reference cases
- No runtime settings are configured.
Shows the first task, up to three tools, and up to three evidence items.
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- Rubric dimension 1 · Weight 30
- Rubric dimension 2 · Weight 25
- Rubric dimension 3 · Weight 20
- Rubric dimension 4 · Weight 15
- Rubric dimension 5 · Weight 10
Reference cases
- 2 public reference cases
Workflow tasks
No workflow tasks are configured.
Recipe freeze marker
No recipe freeze time is recorded.