Office Conversion Feasibility RAG Pipeline with LlamaIndex & Fiddler AI
Structural realities hinder office-to-residential conversions due to deep floorplates and core layout constraints. Build a LlamaIndex RAG agent integrated with Fiddler AI monitoring to parse architectural specs, calculate floorplate feasibility, and extract municipal constraints with 90% precision.
What you are building
The core problem, expected build, and operating context for this challenge.
Build an architectural document parsing and structural assessment workflow using LlamaIndex sentence-window retrieval and Fiddler AI performance monitoring.
How work is evaluated
Evaluates accuracy of extracted structural parameters and feasibility classification against ground truth engineering specs.
Challenge datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
exact_parameter_match
Extracted dimensions match ground truth within 5% tolerance
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
extraction_accuracy
Percentage of correctly extracted structural fields across test suite • target: 0.9 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Extract structural parameters and MEP constraints from PDF architectural blueprints.
Implement a LlamaIndex document index using custom node parsers and window retrieval.
Integrate Fiddler AI to monitor model drift and extraction accuracy over time.
Calculate floorplate depth feasibility ratios against residential daylighting requirements.
Reference links and supporting material
Synthetic dataset of 50 office building architectural specifications, structural reports, and MEP municipal filings.
Challenge setup
Evaluates accuracy of extracted structural parameters and feasibility classification against ground truth engineering specs.
Challenge input
JSON containing document_id and blueprint_text
LlamaIndex
Framework assignment for indexing and structured query processing.
Fiddler AI
Required tool for tracking model accuracy and drift in real estate analytics.
Evaluated output
JSON with core_to_window_dist_ft, mechanical_shaft_capacity, conversion_viable boolean
- Extracted dimensions match ground truth within 5% tolerance
- Percentage of correctly extracted structural fields across test suite • target: 0.9 • range: 0-1
- Benchmark: RAG-Eval-RE
- Extraction Accuracy target: 0.9
- 1 public reference case
- Python execution harness
- Python sandbox (unavailable on Versalist)
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
- Fiddler AI · Required
- Llama Index · Optional
- Synthflow · Optional
Evaluation contract
- exact_parameter_match · Weight 1
- extraction_accuracy · Weight 1
Metrics
- extraction_accuracy · Target 0.9
Benchmarks
- RAG-Eval-RE
- general
Reference cases
- 1 public reference case
Workflow tasks
- Task 1
Input: JSON containing document_id and blueprint_text
Output: JSON with core_to_window_dist_ft, mechanical_shaft_capacity, conversion_viable boolean
Execution harness: Python
Evaluation tools
- LlamaIndex · Agent Data Orchestration Framework
- Fiddler AI · Model Observability & Monitoring
Recipe freeze marker
No recipe freeze time is recorded.