Document AI: Summarize & Extract from Enterprise Content
Leverage LlamaIndex to build a robust Document AI system that can ingest diverse enterprise content (e.g., meeting transcripts, research papers, internal reports) and generate concise podcast-style summaries, identify key entities, and facilitate efficient querying. This system will focus on advanced RAG techniques, knowledge graph construction, and multi-document synthesis to overcome context window limitations and deliver highly accurate, personalized insights from unstructured data. The goal is to transform static documents into dynamic, queryable knowledge assets, mirroring capabilities seen in cutting-edge platforms like Adobe Acrobat's new AI features for content summarization and interaction. Developers will gain hands-on experience with production-grade RAG pipelines, observability tooling, and scalable inference solutions.
What you are building
The core problem, expected build, and operating context for this challenge.
Leverage LlamaIndex to build a robust Document AI system that can ingest diverse enterprise content (e.g., meeting transcripts, research papers, internal reports) and generate concise podcast-style summaries, identify key entities, and facilitate efficient querying. This system will focus on advanced RAG techniques, knowledge graph construction, and multi-document synthesis to overcome context window limitations and deliver highly accurate, personalized insights from unstructured data. The goal is to transform static documents into dynamic, queryable knowledge assets, mirroring capabilities seen in cutting-edge platforms like Adobe Acrobat's new AI features for content summarization and interaction. Developers will gain hands-on experience with production-grade RAG pipelines, observability tooling, and scalable inference solutions.
Shared data for this challenge
Review public datasets and any private uploads tied to your build.
How submissions are scored
These dimensions define what the evaluator checks, how much each dimension matters, and which criteria separate a passable run from a strong one.
SummaryCoherence
Generated summary is coherent and flows naturally, resembling a podcast script.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
KeyHighlightAccuracy
Key highlights accurately reflect the most important information from the documents.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
EntityExtractionCompleteness
All specified entities are extracted from the document with correct types.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
KnowledgeGraphQueryAccuracy
Query answered correctly using information derived from the constructed knowledge graph.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
RAG_Context_Recall
Percentage of relevant document chunks retrieved for a given query, indicating retrieval effectiveness. • target: 0.85 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Summary_Factual_Accuracy
Factual correctness of generated summaries, assessed by comparing statements against source documents. • target: 0.9 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Knowledge_Graph_Density
Number of unique nodes and edges extracted per 1000 words of input text, reflecting graph richness. • target: 12 • range: 5-20
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Master LlamaIndex for building sophisticated RAG applications, including custom node parsers and query engines.
Implement knowledge graph extraction and storage using LlamaIndex's graph functionality and integrate with MongoDB Atlas Vector Search.
Leverage Gemini 2.5 Pro's advanced reasoning capabilities for multi-document synthesis and summarization.
Design and deploy a scalable embedding and inference service using Fireworks AI for document processing.
Integrate LangFuse for end-to-end tracing, evaluation, and monitoring of RAG pipeline performance.
Develop custom data loaders for various enterprise document formats (PDF, DOCX, TXT, audio transcripts).
[ok] Wrote CHALLENGE.md
[ok] Wrote .versalist.json
[ok] Wrote eval/examples.json
Requires VERSALIST_API_KEY. Works with any MCP-aware editor.
DocsAI Research & Mentorship
Participation status
You haven't started this challenge yet
Operating window
Key dates and the organization behind this challenge.
Find another challenge
Jump to a random challenge when you want a fresh benchmark or a different problem space.