Multimodal Plant Diagnostics Agent using Google ADK & Llama 3.3 70B
Control engineers transitioning across domain disciplines need multimodal assistance capable of simultaneously evaluating thermal imaging photos, acoustic vibration spectral audio, and written operational manuals. In this challenge, developers will construct a multimodal diagnostic system using Google ADK (Agent Development Kit) backed by Gemini and Llama 3.3 70B vision models. The system accepts visual thermal camera snapshots of overheating control panels alongside acoustic recordings of motor gearboxes. Using Google ADK's structured context pipeline, the agent parses multimodal signals, correlates visual hotspots with machine drawings, and outputs a step-by-step troubleshooting guide tailored for field control engineers.
What you are building
The core problem, expected build, and operating context for this challenge.
Build a multimodal plant diagnostic assistant using Google ADK and Llama 3.3 70B to analyze thermal images and acoustic logs.
How work is evaluated
Evaluates multimodal accuracy in detecting thermal hotspots and identifying correct switchgear components from visual inputs.
Challenge datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
Hazard Level Identification
Temperatures > 100°C must be flagged as CRITICAL hazard level
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Component Detection Precision
IOU or exact match precision on affected breaker identifiers • target: 0.9 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Build a Google ADK agent capable of handling multimodal inputs (image, audio, text)
Deploy Llama 3.3 70B vision capabilities for thermal image hotspot analysis
Correlate optical bounding boxes with industrial electrical schematics
Generate interactive troubleshooting steps for control engineers
Reference links and supporting material
Dataset containing 40 annotated thermal inspection images of industrial motor control centers (MCC) and acoustic recording clips of failing bearings.
Challenge setup
Evaluates multimodal accuracy in detecting thermal hotspots and identifying correct switchgear components from visual inputs.
Challenge input
Base64 encoded PNG thermal image and textual query
Google ADK
Assigned primary agent framework
Llama 3.3 70B
Assigned multimodal model tool
Evaluated output
JSON object with detected_components, max_temp_c, safety_hazard_level
- Temperatures > 100°C must be flagged as CRITICAL hazard level
- IOU or exact match precision on affected breaker identifiers • target: 0.9 • range: 0-1
- Benchmark: Multimodal Industrial Inspection Benchmark
- Component Detection Precision target: 0.9
- 1 public reference case
- Python execution harness
- Python sandbox (unavailable on Versalist)
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
- Google · Required
- Google ADK · Optional
- Llama 3.3 70B · Optional
Evaluation contract
- Hazard Level Identification · Weight 1
- Component Detection Precision · Weight 1
Metrics
- Component Detection Precision · Target 0.9
Benchmarks
- Multimodal Industrial Inspection Benchmark
- robotics
Reference cases
- 1 public reference case
Workflow tasks
- Task 1
Input: Base64 encoded PNG thermal image and textual query
Output: JSON object with detected_components, max_temp_c, safety_hazard_level
Execution harness: Python
Evaluation tools
- Google ADK · Agent development kit
- Llama 3.3 70B · Multimodal vision reasoning
Recipe freeze marker
No recipe freeze time is recorded.