LLM-Powered Legal & Market Intelligence
Develop an advanced RAG-powered agent system using LlamaIndex to analyze complex legal filings and market intelligence related to high-profile disputes, such as the Elon Musk vs. OpenAI/Microsoft lawsuit. The system will ingest diverse data sources - legal documents, news articles, company statements, and financial reports - to provide comprehensive summaries, strategic insights, and historical context. This challenge emphasizes LlamaIndex's capabilities in multi-document retrieval, hierarchical indexing, and agentic query planning to navigate vast, unstructured datasets. The solution requires designing a robust data pipeline that connects various enterprise data sources, indexes them effectively for semantic search, and employs an agentic query engine to synthesize information. Participants will build custom tools for data extraction and transformation, ensuring the LLM (GPT-4o) can access and reason over highly specific and sometimes contradictory information to generate accurate and actionable intelligence reports.
What you are building
The core problem, expected build, and operating context for this challenge.
Build an advanced RAG system with LlamaIndex and GPT-4o to provide legal and market intelligence from diverse, complex data sources.
How work is evaluated
The evaluation will assess the accuracy, relevance, and coherence of the generated intelligence reports. It will use a combination of automated metrics and human review for qualitative aspects.
Challenge datasets
Public datasets, plus the private datasets you have access to.
What you should walk away with
Master LlamaIndex's advanced indexing strategies, including hierarchical, recursive, and sentence window retrieval for nuanced document analysis.
Implement a flexible data ingestion pipeline using LlamaIndex's data connectors for PDF, web pages, and API sources (e.g., SEC filings, news APIs).
Design and integrate a custom `QueryEngineTool` within LlamaIndex to allow GPT-4o to perform targeted searches and aggregations over indexed data.
Utilize Pinecone as a high-performance vector database backend for LlamaIndex, optimizing embedding and retrieval strategies for legal documents.
Build an agentic query planner using LlamaIndex's query engine composition to break down complex user queries into sub-queries across multiple data sources.
Employ Coval for real-time RAG tracing, evaluation, and fine-tuning, focusing on recall, precision, and contextual relevance metrics.
Deploy the LlamaIndex-powered RAG system as a scalable API endpoint using Featherless AI for efficient model serving and inference.
Develop a user interface leveraging `llamaindex` client-side components to interact with the deployed intelligence agent.
Reference links and supporting material
Sample data for 2 tasks
Challenge setup
The evaluation will assess the accuracy, relevance, and coherence of the generated intelligence reports. It will use a combination of automated metrics and human review for qualitative aspects.
Challenge input
{'query': 'Summarize the core allegations and historical context of the Musk vs. OpenAI/Microsoft lawsuit regarding nonprofit status.'}
Agent execution
No agent tool is listed for this challenge.
Evaluated output
{'summary': 'Detailed summary text.', 'sources': [{'title': 'Doc Title', 'url': 'Doc URL'}]}
- Verify that all generated content attributes to relevant indexed sources.
- Ensure the output adheres to the specified JSON format.
- RelevanceScore target: 0.9
- Python execution harness
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- Verify that all generated content attributes to relevant indexed sources.
- Ensure the output adheres to the specified JSON format.
Metrics
- RelevanceScore · Target 0.9
- FactualityScore · Target 0.9
- CoherenceScore · Target 4
Workflow tasks
- Task 1
Input: {'query': 'Summarize the core allegations and historical context of the Musk vs. OpenAI/Microsoft lawsuit regarding nonprofit status.'}
Output: {'summary': 'Detailed summary text.', 'sources': [{'title': 'Doc Title', 'url': 'Doc URL'}]}
Execution harness: Python
- Task 2
Input: {'query': 'Analyze the strategic implications of the Musk vs. OpenAI/Microsoft lawsuit on their market perception and future partnerships.'}
Output: {'analysis': 'Detailed strategic analysis text.', 'sentiment': 'positive|neutral|negative'}
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.