Back to challenges
Challenge

Control Integrator Troubleshooting Copilot using Vercel AI SDK and Galileo

Field integrators spend hours cross-referencing alarm manuals and wiring diagrams during plant commissioned outages. Build a troubleshooting assistant using Vercel AI SDK and Galileo to evaluate rag hallucination rates while diagnosing Fieldbus and Ethernet/IP bus faults in real time.

EngineeringHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Construct an interactive troubleshooting system with Vercel AI SDK and Galileo observability to resolve control system communication faults.

Delivery guide

How work is evaluated

Evaluation

Evaluates response faithfulness and task resolution for industrial networking fault scenarios.

Datasets

Shared data for this challenge

Review public datasets and any private uploads tied to your build.

Loading datasets...
Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

Root Cause Accuracy

Confirms correct identification of duplex mismatch.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

galileo_faithfulness_score

Faithfulness metric score computed by Galileo hooks. • target: 0.9 • range: 0-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Build a streaming troubleshooting assistant using Vercel AI SDK (`ai/react` or `ai`)

  • Integrate Galileo AI workflow evaluation hooks to monitor response hallucinations and faithfulness

  • Implement multi-turn diagnostic flows covering Ethernet/IP packet loss and duplicate IP node collisions

Resources and assets

Reference links and supporting material

Dataset notes

Network switch telemetry, Ethernet/IP collision logs, and CSIA troubleshooting handbook cases.

How this agent runs

Evaluates response faithfulness and task resolution for industrial networking fault scenarios.

Challenge input

JSON containing fault_code, packet_drop_rate, and switch_logs

AI SDK

Vercel AI SDK provides low-latency streaming and typed tool invocation.

Galileo

Provides real-time scoring of faithfulness and context grounding.

Evaluated output

JSON with root_cause, corrective_action, and confidence

Checks for
  • Confirms correct identification of duplex mismatch.
  • Faithfulness metric score computed by Galileo hooks. • target: 0.9 • range: 0-1
Proof of success
  • Benchmark: Industrial Networking Diagnostic Eval
  • Galileo Faithfulness Score target: 0.9
  • 1 public reference case
Runtime evidence
  • JavaScript execution harness
  • Javascript sandbox
View technical recipe

Configured tools

Action Space
  • Vercel AI SDK · Optional
Observation
  • Galileo · Required
Reward / Eval
  • Galileo · Required
Orchestration
  • Vercel AI SDK · Optional

Evaluation contract

  • Root Cause Accuracy · Weight 1
  • galileo_faithfulness_score · Weight 1

Recipe state

This is a preview. The configuration can change before the evaluation recipe is locked.

Frequently Asked Questions about Control Integrator Troubleshooting Copilot using Vercel AI SDK and Galileo