Multimodal Macro-Reasoning RAG for Colonoscopy Analysis with LlamaIndex and Together AI
Standard RAG systems often fail on complex technical documents like medical reports because they focus on 'micro-matching' (finding specific text chunks) while ignoring 'macro-reasoning' (understanding the overall procedural context). Inspired by the 'DualG-MRAG' and colonoscopy foundation model research, this challenge requires you to build a multimodal RAG pipeline using LlamaIndex. You will implement a dual-graph approach where one graph tracks the high-level medical procedure (macro) and the other handles specific visual/textual findings (micro). You will use Together AI's high-speed inference for Llama 3.2 Vision to process both medical imagery and textual routine reports.
What you are building
Build a Decoupled Macro-Reasoning and Micro-Matching RAG pipeline for medical diagnostics using LlamaIndex and Together AI.
How work is evaluated
The system must accurately identify the 'withdrawal phase' findings across 5 different colonoscopy reports, correctly linking visual polyp images to the textual size description.
Datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
Phase Alignment
Fails if findings from the 'Insertion' phase are misattributed to 'Withdrawal'.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Retrieval Precision
Percentage of retrieved chunks that are clinically relevant. • target: 0.88 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Learning goals
Implement the DualG-MRAG architecture (Macro-Reasoning vs Micro-Matching) within LlamaIndex
Process and embed multimodal data (images and text) using Together AI inference
Construct a procedural knowledge graph to represent medical workflows (e.g., colonoscopy stages)
Develop a query-routing logic that decides when to use global procedural context versus local chunk retrieval
Optimize retrieval accuracy for high-stakes medical report grounding
Resources
Dataset of 280,000 anonymized colonoscopy routine reports and a subset of visual frames from procedures.
Challenge setup
The system must accurately identify the 'withdrawal phase' findings across 5 different colonoscopy reports, correctly linking visual polyp images to the textual size description.
Challenge input
Zip file containing 10 images and 5 report PDFs.
LlamaIndex
Expertly handles the dual-graph retrieval and multimodal indexing required for medical data.
Together AI
Provides low-latency access to large vision-language models for procedural analysis.
Evaluated output
Structured Diagnostic Summary (JSON).
- Fails if findings from the 'Insertion' phase are misattributed to 'Withdrawal'.
- Percentage of retrieved chunks that are clinically relevant. • target: 0.88 • range: 0-1
- Benchmark: Med-RAG Benchmark
- Retrieval Precision target: 0.88
- 1 public reference case
- Docker execution harness
- Python sandbox (unavailable on Versalist)
Shows the first task, up to three tools, and up to three evidence items.
View technical recipe
Configured tools
- Modal · Optional
- RAI · Required
- Llama Index · Optional
- Modal · Optional
Evaluation contract
- Phase Alignment · Weight 1
- Retrieval Precision · Weight 1
Metrics
- Retrieval Precision · Target 0.88
Benchmarks
- Med-RAG Benchmark
- biotech
Reference cases
- 1 public reference case
Workflow tasks
- Task 1
Input: Zip file containing 10 images and 5 report PDFs.
Output: Structured Diagnostic Summary (JSON).
Execution harness: Docker
Evaluation tools
- LlamaIndex · Data Framework
- Together AI · Inference Engine
Recipe freeze marker
No recipe freeze time is recorded.