Build a Multi-Reference RAG Shopping Agent
This challenge focuses on developing an advanced shopping agent capable of researching and comparing products with high factual accuracy. There is a need to reduce hallucination as observed in applications like ChatGPT, this agent will leverage a sophisticated multi-reference Retrieval Augmented Generation (RAG) system. The core innovation lies in requiring information to be cross-referenced and validated by multiple independent sources before being presented, ensuring robustness and trustworthiness. You will utilize Mistral Large 2 for its powerful reasoning and language generation capabilities, orchestrated through a LangGraph-based workflow. The agent will interact with external data sources (e.g., product websites, review platforms) and a vector database to build comprehensive, evidence-backed product comparison reports. This project addresses the critical industry need for reliable AI agents in high-stakes information-seeking tasks.
What you are building
Develop a LangGraph-orchestrated shopping agent utilizing Mistral Large 2 and a FAISS-backed multi-reference RAG system for accurate product comparison and research.
How work is evaluated
The agent will be evaluated on its ability to accurately compare products from multiple sources, provide evidence-backed responses, and avoid unsubstantiated claims or hallucinations.
Datasets
Public datasets, plus the private datasets you have access to.
Learning goals
Master LangGraph for stateful multi-agent orchestration: Learn to define states, nodes, and conditional edges to manage complex agentic workflows, specifically for a multi-step product research and comparison process.
Implement advanced RAG strategies with Mistral Large 2: Design a retrieval system that queries multiple distinct data sources (e.g., product reviews, spec sheets, e-commerce sites) and synthesizes information while identifying conflicting data using Mistral Large 2's reasoning capabilities.
Build a multi-reference validation module: Create a mechanism that requires information to be corroborated by at least N (e.g., 2-3) independent sources before being presented, mitigating hallucination and 'user delusion validation' issues.
Integrate vector databases (specifically FAISS): Utilize FAISS for efficient semantic search across diverse product documentation, ensuring relevant context is retrieved for comparison.
Develop custom tools for web scraping and API interaction: Enable the agent to dynamically gather real-time product data from e-commerce platforms and review sites.
Design an evaluation framework for comparative product analysis: Define metrics to assess the accuracy, completeness, and neutrality of product comparisons generated by the agent.
Optimize token usage and latency with Mistral Large 2: Explore strategies for efficient prompting and response generation to balance performance with cost-effectiveness.
Resources
Sample data for 1 tasks
Challenge setup
The agent will be evaluated on its ability to accurately compare products from multiple sources, provide evidence-backed responses, and avoid unsubstantiated claims or hallucinations.
Challenge input
{'products': ['product_A_name', 'product_B_name'], 'criteria': ['feature_1', 'feature_2', 'price']}
Agent execution
No agent tool is listed for this challenge.
Evaluated output
{'comparison_report': {'product_A': {'feature_1': 'val', 'feature_2': 'val', 'price': 'val', 'sources': ['url1', 'url2']}, 'product_B': {'feature_1...
- Verify the output adheres to the specified JSON structure and contains all required keys.
- Ensure at least two distinct source URLs are provided for each factual claim in the comparison.
- Verify that all presented facts can be traced back to the provided source URLs and are not fabricated (this...
- Accuracy Score target: 90
- Python execution harness
Shows the first task, up to three tools, and up to three evidence items.
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- Verify the output adheres to the specified JSON structure and contains all required keys.
- Ensure at least two distinct source URLs are provided for each factual claim in the comparison.
- Verify that all presented facts can be traced back to the provided source URLs and are not fabricated (this test may involve a separate LLM judge or manual verification against provided sources).
Metrics
- Accuracy_Score · Target 90
- Completeness_Score · Target 95
- Source_Diversity_Score · Target 3
Workflow tasks
- Task 1
Input: {'products': ['product_A_name', 'product_B_name'], 'criteria': ['feature_1', 'feature_2', 'price']}
Output: {'comparison_report': {'product_A': {'feature_1': 'val', 'feature_2': 'val', 'price': 'val', 'sources': ['url1', 'url2']}, 'product_B': {'feature_1': 'val', 'feature_2': 'val', 'price': 'val', 'sources': ['url3', 'url4']}}, 'recommendation': 'product_A/B/neither', 'justification': '...'}
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.