Skip to main content
Back to challenges
Challenge

Build an OT Knowledge Gap Copilot with LlamaIndex and GPT-5 Pro

Control integrators suffer severe knowledge loss as senior control engineers retire. Build an OT Knowledge Copilot using LlamaIndex and GPT-5 Pro that indexes historical trouble tickets, SCADA logs, and device manuals. Achieve an answer accuracy score of 92% on diagnostic troubleshooting questions while referencing exact manual page sources.

EngineeringHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Develop an advanced RAG copilot using LlamaIndex indexing and GPT-5 Pro reasoning over multimodal operational manuals and maintenance logs.

Delivery guide

How work is evaluated

Evaluation

Evaluates operational issue resolution diagnostic accuracy and citation source verification.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

citation_presence

Ensure citations contain exact document and page references

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

diagnostic_accuracy

Accuracy of parameter fault recommendations against technical manual ground truth • target: 0.92 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Index technical manual PDFs and historical SCADA alarm logs using LlamaIndex VectorStoreIndex

  • Configure GPT-5 Pro for high-precision technical troubleshooting reasoning

  • Implement citation backlinking to guarantee verifiable source manual page attribution

Resources and assets

Reference links and supporting material

Dataset notes

Curated dataset of 30 VFD and SCADA fault manuals paired with historic CSIA integration trouble ticket resolutions.

Challenge setup

Evaluates operational issue resolution diagnostic accuracy and citation source verification.

Challenge input

JSON object with fault code and symptom description

LlamaIndex

Specialized RAG indexing and citation engine tools for technical enterprise documents.

GPT-5 Pro

State of the art reasoning capability over complex technical domain manuals.

Evaluated output

JSON object with proposed resolution steps and citation pages

Evaluation checks
  • Ensure citations contain exact document and page references
  • Accuracy of parameter fault recommendations against technical manual ground truth • target: 0.92 • range: 0-1
Configured evaluation evidence
  • Benchmark: OT Maintenance Troubleshooting QA (OT-QA-2025)
  • Diagnostic Accuracy target: 0.92
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

Action Space
  • Llama Index · Required
  • LlamaIndex · Optional
  • GPT-5 Pro · Optional

Evaluation contract

  • citation_presence · Weight 1
  • diagnostic_accuracy · Weight 1

Metrics

  • diagnostic_accuracy · Target 0.92

Benchmarks

  • OT Maintenance Troubleshooting QA (OT-QA-2025)
  • general

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: JSON object with fault code and symptom description

    Output: JSON object with proposed resolution steps and citation pages

    Execution harness: Python

Evaluation tools

  • LlamaIndex · Data Framework
  • GPT-5 Pro · Reasoning LLM

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Build an OT Knowledge Gap Copilot with LlamaIndex and GPT-5 Pro