Skip to main content
Back to challenges
Challenge

CAD Graph Analysis and RAG Search using LangChain and Azure OpenAI

Legacy CAD repositories contain static dynamic models that fail to expose structural semantic search capabilities to design teams. Build an Engineering Intelligence agent using LangChain, Azure OpenAI, and graph vector databases to query 3D STEP file metadata and assembly dependencies. Deliver sub-2 second semantic search query responses with 90% top-3 recall for component reuse.

EngineeringHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Build a CAD metadata semantic graph search engine using LangChain expression language (LCEL) connected to Azure OpenAI embeddings and vector storage.

Delivery guide

How work is evaluated

Evaluation

Evaluates mechanical assembly search recall and response latency when querying dynamic STEP metadata graph.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

response_time_check

Ensure total search latency is below 2000ms

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

top_3_recall

Proportion of target CAD components present in top 3 results • target: 0.9 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Construct a dynamic knowledge graph from 3D CAD STEP file structural metadata

  • Implement hybrid vector and graph search chains using LangChain and Azure OpenAI

  • Evaluate semantic query recall for CAD assembly dependency queries

Resources and assets

Reference links and supporting material

Dataset notes

Dataset of 50 STEP CAD assembly graphs and associated structural metadata representing complex automated conveyor drive units.

Challenge setup

Evaluates mechanical assembly search recall and response latency when querying dynamic STEP metadata graph.

Challenge input

Natural language query string

LangChain

Provides modular chains and retrievers for graph and dynamic vector hybrid search.

Azure OpenAI

Enterprise enterprise reliability and security compliance for manufacturing data.

Evaluated output

JSON list of dynamic matching CAD part identifiers

Evaluation checks
  • Ensure total search latency is below 2000ms
  • Proportion of target CAD components present in top 3 results • target: 0.9 • range: 0-1
Configured evaluation evidence
  • Benchmark: Engineering CAD Intelligence Retrieval (ECAD-RAG-2025)
  • Top 3 Recall target: 0.9
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

Action Space
  • Langchain · Required
  • LangChain · Optional
  • OpenAI · Optional
Orchestration
  • Langchain · Required
  • LangChain · Optional

Evaluation contract

  • response_time_check · Weight 1
  • top_3_recall · Weight 1

Metrics

  • top_3_recall · Target 0.9

Benchmarks

  • Engineering CAD Intelligence Retrieval (ECAD-RAG-2025)
  • general

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: Natural language query string

    Output: JSON list of dynamic matching CAD part identifiers

    Execution harness: Python

Evaluation tools

  • LangChain · Orchestration Library
  • Azure OpenAI · LLM & Embedding Host

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about CAD Graph Analysis and RAG Search using LangChain and Azure OpenAI