Multimodal Macro-Reasoning RAG for Colonoscopy Analysis with LlamaIndex and Together AI

Standard RAG systems often fail on complex technical documents like medical reports because they focus on 'micro-matching' (finding specific text chunks) while ignoring 'macro-reasoning' (understanding the overall procedural context). Inspired by the 'DualG-MRAG' and colonoscopy foundation model research, this challenge requires you to build a multimodal RAG pipeline using LlamaIndex. You will implement a dual-graph approach where one graph tracks the high-level medical procedure (macro) and the other handles specific visual/textual findings (micro). You will use Together AI's high-speed inference for Llama 3.2 Vision to process both medical imagery and textual routine reports.

• Configure LlamaIndex for Multimodal RAG: Set up a MultiModalVectorStoreIndex to handle both colonoscopy images and report text.

• Integrate Together AI Inference: Use Together AI's API to run Llama 3.2 Vision for rapid image feature extraction and grounding.

• Build the Macro-Reasoning Graph: Use LlamaIndex Property Graphs to model the standard phases of a colonoscopy (insertion, withdrawal, inspection).

• Implement Micro-Matching Retrieval: Develop a specialized retriever that finds high-resolution visual evidence for specific polyps or lesions.

• Orchestrate Dual Retrieval: Write a custom query engine that fuses findings from the macro-graph (contextual severity) and micro-matching (exact location).

• Grounding via Foundation Models: Apply techniques from the 280,000-report vision-language model research to align agent responses with clinical standards.