Back to challenges
Challenge

CAD Graph Analysis and RAG Search using LangChain and Azure OpenAI

Legacy CAD repositories contain static dynamic models that fail to expose structural semantic search capabilities to design teams. Build an Engineering Intelligence agent using LangChain, Azure OpenAI, and graph vector databases to query 3D STEP file metadata and assembly dependencies. Deliver sub-2 second semantic search query responses with 90% top-3 recall for component reuse.

EngineeringHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Build a CAD metadata semantic graph search engine using LangChain expression language (LCEL) connected to Azure OpenAI embeddings and vector storage.

Delivery guide

How work is evaluated

Evaluation

Evaluates mechanical assembly search recall and response latency when querying dynamic STEP metadata graph.

Datasets

Shared data for this challenge

Review public datasets and any private uploads tied to your build.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

response_time_check

Ensure total search latency is below 2000ms

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

top_3_recall

Proportion of target CAD components present in top 3 results • target: 0.9 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Construct a dynamic knowledge graph from 3D CAD STEP file structural metadata

  • Implement hybrid vector and graph search chains using LangChain and Azure OpenAI

  • Evaluate semantic query recall for CAD assembly dependency queries

Resources and assets

Reference links and supporting material

Dataset notes

Dataset of 50 STEP CAD assembly graphs and associated structural metadata representing complex automated conveyor drive units.

How this agent runs

Evaluates mechanical assembly search recall and response latency when querying dynamic STEP metadata graph.

Challenge input

Natural language query string

LangChain

Provides modular chains and retrievers for graph and dynamic vector hybrid search.

Azure OpenAI

Enterprise enterprise reliability and security compliance for manufacturing data.

Evaluated output

JSON list of dynamic matching CAD part identifiers

Checks for
  • Ensure total search latency is below 2000ms
  • Proportion of target CAD components present in top 3 results • target: 0.9 • range: 0-1
Proof of success
  • Benchmark: Engineering CAD Intelligence Retrieval (ECAD-RAG-2025)
  • Top 3 Recall target: 0.9
  • 1 public reference case
Runtime evidence
  • Python execution harness
  • Python sandbox (unavailable on Versalist)
View technical recipe

Configured tools

Action Space
  • Langchain · Required
  • LangChain · Optional
  • OpenAI · Optional
Orchestration
  • Langchain · Required
  • LangChain · Optional

Evaluation contract

  • response_time_check · Weight 1
  • top_3_recall · Weight 1

Recipe state

This is a preview. The configuration can change before the evaluation recipe is locked.

Frequently Asked Questions about CAD Graph Analysis and RAG Search using LangChain and Azure OpenAI