Document AI: Summarize & Extract from Enterprise Content
Leverage LlamaIndex to build a robust Document AI system that can ingest diverse enterprise content (e.g., meeting transcripts, research papers, internal reports) and generate concise podcast-style summaries, identify key entities, and facilitate efficient querying. This system will focus on advanced RAG techniques, knowledge graph construction, and multi-document synthesis to overcome context window limitations and deliver highly accurate, personalized insights from unstructured data. The goal is to transform static documents into dynamic, queryable knowledge assets, mirroring capabilities seen in cutting-edge platforms like Adobe Acrobat's new AI features for content summarization and interaction. Developers will gain hands-on experience with production-grade RAG pipelines, observability tooling, and scalable inference solutions.
What you are building
The core problem, expected build, and operating context for this challenge.
Build an advanced RAG and knowledge graph system using LlamaIndex with Gemini 2.5 Pro for extracting and summarizing insights from enterprise documents.
How work is evaluated
Evaluation will focus on the system's ability to accurately summarize content, extract specified entities, and answer complex queries across multiple documents.
Challenge datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
SummaryCoherence
Generated summary is coherent and flows naturally, resembling a podcast script.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
KeyHighlightAccuracy
Key highlights accurately reflect the most important information from the documents.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
EntityExtractionCompleteness
All specified entities are extracted from the document with correct types.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
KnowledgeGraphQueryAccuracy
Query answered correctly using information derived from the constructed knowledge graph.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
RAG_Context_Recall
Percentage of relevant document chunks retrieved for a given query, indicating retrieval effectiveness. • target: 0.85 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Summary_Factual_Accuracy
Factual correctness of generated summaries, assessed by comparing statements against source documents. • target: 0.9 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Knowledge_Graph_Density
Number of unique nodes and edges extracted per 1000 words of input text, reflecting graph richness. • target: 12 • range: 5-20
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Master LlamaIndex for building sophisticated RAG applications, including custom node parsers and query engines.
Implement knowledge graph extraction and storage using LlamaIndex's graph functionality and integrate with MongoDB Atlas Vector Search.
Leverage Gemini 2.5 Pro's advanced reasoning capabilities for multi-document synthesis and summarization.
Design and deploy a scalable embedding and inference service using Fireworks AI for document processing.
Integrate LangFuse for end-to-end tracing, evaluation, and monitoring of RAG pipeline performance.
Develop custom data loaders for various enterprise document formats (PDF, DOCX, TXT, audio transcripts).
Reference links and supporting material
Sample data for 2 tasks
Challenge setup
Evaluation will focus on the system's ability to accurately summarize content, extract specified entities, and answer complex queries across multiple documents.
Challenge input
{"documents": [{"title": "str", "content": "str", "type": "str"}], "summary_length_minutes": "int"}
Agent execution
No agent tool is listed for this challenge.
Evaluated output
{"summary_audio_script": "str", "key_highlights": ["str"]}
- Generated summary is coherent and flows naturally, resembling a podcast script.
- Key highlights accurately reflect the most important information from the documents.
- All specified entities are extracted from the document with correct types.
- RAG Context Recall target: 0.85
- 2 public reference cases
- Python execution harness
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- SummaryCoherence · Weight 1
- KeyHighlightAccuracy · Weight 1
- EntityExtractionCompleteness · Weight 1
- KnowledgeGraphQueryAccuracy · Weight 1
- RAG_Context_Recall · Weight 1
- Summary_Factual_Accuracy · Weight 1
- Knowledge_Graph_Density · Weight 1
Metrics
- RAG_Context_Recall · Target 0.85
- Summary_Factual_Accuracy · Target 0.9
- Knowledge_Graph_Density · Target 12
Reference cases
- 2 public reference cases
Workflow tasks
- Task 1
Input: {"documents": [{"title": "str", "content": "str", "type": "str"}], "summary_length_minutes": "int"}
Output: {"summary_audio_script": "str", "key_highlights": ["str"]}
Execution harness: Python
- Task 2
Input: {"document_content": "str", "entities_to_extract": ["str"], "query": "str"}
Output: {"extracted_entities": [{"type": "str", "value": "str"}], "query_answer": "str"}
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.