Psychopathology Synthetic Patient Modeling with Pydantic AI and Baseten
Based on research in The Lancet Digital Health examining LLMs as experimental systems in human psychopathology, this challenge targets the modeling of synthetic clinical patient dynamics. Developers will construct a type-safe simulation pipeline using Pydantic AI hosted on Baseten high-throughput inference runtimes. The agent will simulate psychiatric interviews, log symptom trajectories across longitudinal cognitive dimensions, and enforce rigid psychometric validation boundaries.
What you are building
The core problem, expected build, and operating context for this challenge.
Build a type-safe psychopathology patient simulator using Pydantic AI and Baseten inference runtimes.
How work is evaluated
Evaluates synthetic patient response adherence to psychometric profiles and response validation schemas.
Challenge datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
pydantic_schema_validation
Output validates successfully against Pydantic schema
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
psychometric_consistency
Correlation between intended severity profile and generated interview PHQ-9 equivalent score • target: 0.88 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Implement structured psychopathology response schemas using Pydantic AI
Deploy custom LLM inference endpoints on Baseten model runtime engine
Simulate standardized psychiatric diagnostic interviews across diverse diagnostic archetypes
Validate psychiatric diagnostic agreement with empirical clinical benchmarks
Reference links and supporting material
Synthetic diagnostic interview transcripts annotated with DSM-5 criteria, symptom intensity scores, and longitudinal depression indices.
Challenge setup
Evaluates synthetic patient response adherence to psychometric profiles and response validation schemas.
Challenge input
JSON containing patient profile parameters (DSM-5 axis criteria)
Pydantic AI
Mandatory type-safe agent framework
Baseten
Assigned model deployment platform
Evaluated output
JSON containing structured response dialogue and calculated PHQ-9 score
- Output validates successfully against Pydantic schema
- Correlation between intended severity profile and generated interview PHQ-9 equivalent score • target: 0.88...
- Benchmark: PsyEval-Bench
- Psychometric Consistency target: 0.88
- 1 public reference case
- Python execution harness
- Python sandbox (unavailable on Versalist)
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
- Baseten · Required
- Pydantic AI · Optional
- Exa · Optional
- Pydantic AI · Optional
Evaluation contract
- pydantic_schema_validation · Weight 1
- psychometric_consistency · Weight 1
Metrics
- psychometric_consistency · Target 0.88
Benchmarks
- PsyEval-Bench
- biotech
Reference cases
- 1 public reference case
Workflow tasks
- Task 1
Input: JSON containing patient profile parameters (DSM-5 axis criteria)
Output: JSON containing structured response dialogue and calculated PHQ-9 score
Execution harness: Python
Evaluation tools
- Pydantic AI · Agent Framework
- Baseten · Inference Infrastructure
Recipe freeze marker
No recipe freeze time is recorded.