Build a Multi-Reference RAG Shopping Agent
This challenge focuses on developing an advanced shopping agent capable of researching and comparing products with high factual accuracy. There is a need to reduce hallucination as observed in applications like ChatGPT, this agent will leverage a sophisticated multi-reference Retrieval Augmented Generation (RAG) system. The core innovation lies in requiring information to be cross-referenced and validated by multiple independent sources before being presented, ensuring robustness and trustworthiness. You will utilize Mistral Large 2 for its powerful reasoning and language generation capabilities, orchestrated through a LangGraph-based workflow. The agent will interact with external data sources (e.g., product websites, review platforms) and a vector database to build comprehensive, evidence-backed product comparison reports. This project addresses the critical industry need for reliable AI agents in high-stakes information-seeking tasks.
What you are building
The core problem, expected build, and operating context for this challenge.
Develop a LangGraph-orchestrated shopping agent utilizing Mistral Large 2 and a FAISS-backed multi-reference RAG system for accurate product comparison and research.
How work is evaluated
The agent will be evaluated on its ability to accurately compare products from multiple sources, provide evidence-backed responses, and avoid unsubstantiated claims or hallucinations.
Challenge datasets
Public datasets, plus the private datasets you have access to.
What you should walk away with
Master LangGraph for stateful multi-agent orchestration: Learn to define states, nodes, and conditional edges to manage complex agentic workflows, specifically for a multi-step product research and comparison process.
Implement advanced RAG strategies with Mistral Large 2: Design a retrieval system that queries multiple distinct data sources (e.g., product reviews, spec sheets, e-commerce sites) and synthesizes information while identifying conflicting data using Mistral Large 2's reasoning capabilities.
Build a multi-reference validation module: Create a mechanism that requires information to be corroborated by at least N (e.g., 2-3) independent sources before being presented, mitigating hallucination and 'user delusion validation' issues.
Integrate vector databases (specifically FAISS): Utilize FAISS for efficient semantic search across diverse product documentation, ensuring relevant context is retrieved for comparison.
Develop custom tools for web scraping and API interaction: Enable the agent to dynamically gather real-time product data from e-commerce platforms and review sites.
Design an evaluation framework for comparative product analysis: Define metrics to assess the accuracy, completeness, and neutrality of product comparisons generated by the agent.
Optimize token usage and latency with Mistral Large 2: Explore strategies for efficient prompting and response generation to balance performance with cost-effectiveness.
Reference links and supporting material
Sample data for 1 tasks
Challenge setup
The agent will be evaluated on its ability to accurately compare products from multiple sources, provide evidence-backed responses, and avoid unsubstantiated claims or hallucinations.
Challenge input
{'products': ['product_A_name', 'product_B_name'], 'criteria': ['feature_1', 'feature_2', 'price']}
Agent execution
No agent tool is listed for this challenge.
Evaluated output
{'comparison_report': {'product_A': {'feature_1': 'val', 'feature_2': 'val', 'price': 'val', 'sources': ['url1', 'url2']}, 'product_B': {'feature_1...
- Verify the output adheres to the specified JSON structure and contains all required keys.
- Ensure at least two distinct source URLs are provided for each factual claim in the comparison.
- Verify that all presented facts can be traced back to the provided source URLs and are not fabricated (this...
- Accuracy Score target: 90
- Python execution harness
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- Verify the output adheres to the specified JSON structure and contains all required keys.
- Ensure at least two distinct source URLs are provided for each factual claim in the comparison.
- Verify that all presented facts can be traced back to the provided source URLs and are not fabricated (this test may involve a separate LLM judge or manual verification against provided sources).
Metrics
- Accuracy_Score · Target 90
- Completeness_Score · Target 95
- Source_Diversity_Score · Target 3
Workflow tasks
- Task 1
Input: {'products': ['product_A_name', 'product_B_name'], 'criteria': ['feature_1', 'feature_2', 'price']}
Output: {'comparison_report': {'product_A': {'feature_1': 'val', 'feature_2': 'val', 'price': 'val', 'sources': ['url1', 'url2']}, 'product_B': {'feature_1': 'val', 'feature_2': 'val', 'price': 'val', 'sources': ['url3', 'url4']}}, 'recommendation': 'product_A/B/neither', 'justification': '...'}
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.