Control Integrator Troubleshooting Copilot using Vercel AI SDK and Galileo
Field integrators spend hours cross-referencing alarm manuals and wiring diagrams during plant commissioned outages. Build a troubleshooting assistant using Vercel AI SDK and Galileo to evaluate rag hallucination rates while diagnosing Fieldbus and Ethernet/IP bus faults in real time.
What you are building
The core problem, expected build, and operating context for this challenge.
Construct an interactive troubleshooting system with Vercel AI SDK and Galileo observability to resolve control system communication faults.
How work is evaluated
Evaluates response faithfulness and task resolution for industrial networking fault scenarios.
Challenge datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
Root Cause Accuracy
Confirms correct identification of duplex mismatch.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
galileo_faithfulness_score
Faithfulness metric score computed by Galileo hooks. • target: 0.9 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Build a streaming troubleshooting assistant using Vercel AI SDK (`ai/react` or `ai`)
Integrate Galileo AI workflow evaluation hooks to monitor response hallucinations and faithfulness
Implement multi-turn diagnostic flows covering Ethernet/IP packet loss and duplicate IP node collisions
Reference links and supporting material
Network switch telemetry, Ethernet/IP collision logs, and CSIA troubleshooting handbook cases.
Challenge setup
Evaluates response faithfulness and task resolution for industrial networking fault scenarios.
Challenge input
JSON containing fault_code, packet_drop_rate, and switch_logs
AI SDK
Vercel AI SDK provides low-latency streaming and typed tool invocation.
Galileo
Provides real-time scoring of faithfulness and context grounding.
Evaluated output
JSON with root_cause, corrective_action, and confidence
- Confirms correct identification of duplex mismatch.
- Faithfulness metric score computed by Galileo hooks. • target: 0.9 • range: 0-1
- Benchmark: Industrial Networking Diagnostic Eval
- Galileo Faithfulness Score target: 0.9
- 1 public reference case
- JavaScript execution harness
- javascript sandbox (unavailable on Versalist)
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
- Vercel AI SDK · Optional
- Galileo · Required
- Galileo · Required
- Vercel AI SDK · Optional
Evaluation contract
- Root Cause Accuracy · Weight 1
- galileo_faithfulness_score · Weight 1
Metrics
- galileo_faithfulness_score · Target 0.9
Benchmarks
- Industrial Networking Diagnostic Eval
- security
Reference cases
- 1 public reference case
Workflow tasks
- Task 1
Input: JSON containing fault_code, packet_drop_rate, and switch_logs
Output: JSON with root_cause, corrective_action, and confidence
Execution harness: JavaScript
Evaluation tools
- AI SDK · Streaming Client and Tool Handler
- Galileo · AI Workflow Automation & Observability
Recipe freeze marker
No recipe freeze time is recorded.