Build a Work-Sample Assessment Review Assistant
Build an agent that organizes work samples, maps evidence to rubric criteria, and drafts review notes. Let human reviewers inspect the work, adjust criterion judgments, and publish their own feedback.
What you are building
The core problem, expected build, and operating context for this challenge.
A reviewer assessing submitted technical work against a published rubric.
Build an agent that organizes work samples, maps evidence to rubric criteria, and drafts review notes. Let human reviewers inspect the work, adjust criterion judgments, and publish their own feedback.
Synthetic work submissions
Published job-relevant rubric
Reviewer comments
Remove irrelevant identity fields from the review view.
Extract evidence for each rubric criterion.
Draft criterion-level judgments and clarification questions.
Collect human edits and export an evidence-linked feedback report.
A reviewer workspace with the submission, criterion evidence, editable judgments, and a feedback preview.
Draft five points for R1 and zero for R2 with citations to the work, then let a reviewer edit the feedback.
A working local application with the specified review interface.
Source code, dependency lockfile, and setup commands.
A recorded end-to-end demo using the supplied starter data.
Example outputs, action logs, and automated tests.
These fictional records are a small development fixture, not a production dataset. Implement the mock services described in the brief.
```json { "submission": { "id": "S1", "synthetic_name": "Person A", "text": "The solution returns the correct total. It includes no tests." }, "rubric": [ { "id": "R1", "criterion": "Correct result", "maximum_points": 5 }, { "id": "R2", "criterion": "Regression tests", "maximum_points": 5 } ], "evidence": { "correct_result": true, "tests_present": false } } ```
Use synthetic work. Keep judgments reviewable and do not automate real employment decisions. Choose an available model that supports your implementation. Record its identifier and version.
Review the complete application, not only the starter-case answers.
The draft total is five and each judgment has evidence.
Identity fields are absent from the scoring view.
No hiring decision or applicant ranking is generated. Add one normal case, one incomplete-input case, and one failure-recovery case.
Participation guide
Use synthetic work. Keep judgments reviewable and do not automate real employment decisions.
How work is evaluated
Review workflow completion, interface usability, evidence quality, and safe failure handling. No agent execution has been verified.
A working local application with the specified review interface. Source code, dependency lockfile, and setup commands. A recorded end-to-end demo using the supplied starter data. Example outputs, action logs, and automated tests.
Challenge datasets
Public datasets, plus the private datasets you have access to.
What you should walk away with
Remove irrelevant identity fields from the review view.
Extract evidence for each rubric criterion.
Draft criterion-level judgments and clarification questions.
Collect human edits and export an evidence-linked feedback report.
Reference links and supporting material
Supplied fictional starter records: Synthetic work submissions, Published job-relevant rubric, Reviewer comments. The overview includes the complete JSON bundle. Implement local mock services and extend the sample coverage.
Challenge setup
Configuration summary for this challenge.
Challenge input
Input details are not configured.
Pydantic AI
Typed Python agent framework.
Evaluated output
No output checks are configured.
- No evaluation checks are configured.
- 1 public reference case
- 1 configured tool
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
- Pydantic AI ยท Optional
Evaluation contract
- No evaluation checks are configured.
Reference cases
- 1 public reference case
Workflow tasks
No workflow tasks are configured.
Recipe freeze marker
No recipe freeze time is recorded.