ISA/IEC 62443 Compliance Auditor Agent with LlamaIndex RAG
Adopting ISA/IEC 62443 cybersecurity standards across plant networks requires auditing complex control system configurations against extensive compliance documentation. Build a Retrieval-Augmented Generation (RAG) agent using LlamaIndex to query ISA/IEC 62443 standard controls and assess plant firewall configurations. Achieve 90% semantic query precision across regulatory controls.
What you are building
Develop a LlamaIndex RAG pipeline indexing ISA/IEC 62443 documentation to evaluate control system security posture and generate audit gap analysis.
How work is evaluated
Evaluates RAG retrieval accuracy and compliance gap identification against ISA/IEC 62443 standards.
Datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
clause_citation_accuracy
Ensures the correct ISA/IEC 62443 section is cited in the gap evaluation.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
semantic_query_precision
Precision of retrieved knowledge nodes relative to ground truth references. • target: 0.9 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Learning goals
Index ISA/IEC 62443-3-3 and 62443-4-2 standard requirements into LlamaIndex vector stores
Build a query engine capable of retrieving security level (SL) targets and foundational requirements
Analyze plant security configuration files against retrieved standard requirements
Resources
ISA/IEC 62443 control requirement summaries paired with sample industrial device configuration files.
Challenge setup
Evaluates RAG retrieval accuracy and compliance gap identification against ISA/IEC 62443 standards.
Challenge input
JSON containing query_string, security_level_target, and system_config_text
LlamaIndex
Data framework and RAG architecture
Evaluated output
JSON with answer, cited_clauses array, and compliance_status
- Ensures the correct ISA/IEC 62443 section is cited in the gap evaluation.
- Precision of retrieved knowledge nodes relative to ground truth references. • target: 0.9 • range: 0-1
- Benchmark: Regulatory Compliance QA Benchmark
- Semantic Query Precision target: 0.9
- 1 public reference case
- Python execution harness
- Python sandbox (unavailable on Versalist)
Shows the first task, up to three tools, and up to three evidence items.
View technical recipe
Configured tools
- Llama Index · Required
- LlamaIndex · Optional
- Llama 3.2 · Optional
Evaluation contract
- clause_citation_accuracy · Weight 1
- semantic_query_precision · Weight 1
Metrics
- semantic_query_precision · Target 0.9
Benchmarks
- Regulatory Compliance QA Benchmark
- security
Reference cases
- 1 public reference case
Workflow tasks
- Task 1
Input: JSON containing query_string, security_level_target, and system_config_text
Output: JSON with answer, cited_clauses array, and compliance_status
Execution harness: Python
Evaluation tools
- LlamaIndex · framework
Recipe freeze marker
No recipe freeze time is recorded.