Intelligent CAD Metrology Validation Agent with Pydantic AI and Sourcegraph Amp
Standard CAD packages cannot perform cognitive geometric dimensioning and tolerancing (GD&T) analysis against manufacturing capabilities. You will construct a type-safe engineering intelligence agent using Pydantic AI and Sourcegraph Amp to evaluate STEP/IGES files and validate GD&T tolerances against shop floor CNC precision capabilities. The system must achieve zero schema validation errors and 95% pass accuracy on tolerance checks.
What you are building
The core problem, expected build, and operating context for this challenge.
Develop a type-safe Pydantic AI agent augmented with Sourcegraph Amp for automated CAD design validation and GD&T tolerance compliance.
How work is evaluated
Evaluates Pydantic AI structural output correctness against ground-truth metrology spec limits.
Challenge datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
pydantic_schema_validation
Validates that response strictly satisfies Pydantic Model contract.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
manufacturability_classification_accuracy
Accuracy of pass/fail classification on mechanical parts • target: 0.95 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Build strongly-typed Pydantic AI agent models for complex GD&T geometric schemas
Utilize Sourcegraph Amp for rapid context analysis across large C++ / Python CAD parser codebases
Extract dimensional features from STEP/IGES metadata using Python OpenCASCADE (pythonocc)
Enforce deterministic validation rules comparing tool path tolerances with machine capability indices (Cpk)
Reference links and supporting material
Synthetic JSON representations of 3D mechanical CAD parts with embedded GD&T annotations and CNC capability profiles.
Challenge setup
Evaluates Pydantic AI structural output correctness against ground-truth metrology spec limits.
Challenge input
JSON detailing part geometry bounding boxes, target tolerances, and CNC machine precision limits
Pydantic AI
Required framework assignment.
Sourcegraph Amp
Required tool assignment.
Evaluated output
Pydantic structured output object with pass_status, critical_flaws, and suggested_tolerance_adjustments
- Validates that response strictly satisfies Pydantic Model contract.
- Accuracy of pass/fail classification on mechanical parts • target: 0.95 • range: 0-1
- Benchmark: CAD Design Intelligence Evaluation Standard
- Manufacturability Classification Accuracy target: 0.95
- 1 public reference case
- Python execution harness
- Python sandbox (unavailable on Versalist)
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
- Pydantic AI · Required
- Sourcegraph Amp · Optional
- Sourcegraph Amp · Optional
- Pydantic AI · Required
Evaluation contract
- pydantic_schema_validation · Weight 1
- manufacturability_classification_accuracy · Weight 1
Metrics
- manufacturability_classification_accuracy · Target 0.95
Benchmarks
- CAD Design Intelligence Evaluation Standard
- materials
Reference cases
- 1 public reference case
Workflow tasks
- Task 1
Input: JSON detailing part geometry bounding boxes, target tolerances, and CNC machine precision limits
Output: Pydantic structured output object with pass_status, critical_flaws, and suggested_tolerance_adjustments
Execution harness: Python
Evaluation tools
- Pydantic AI · Type-Safe Agent Framework
- Sourcegraph Amp · AI Code Intelligence Tool
Recipe freeze marker
No recipe freeze time is recorded.