Deploy Policy Assistant ALLIE on Modal using LlamaIndex
Carriers need performant RAG engines to deliver policy answer engines safely. You will construct a domain-specific policy QA workflow using LlamaIndex and deploy the inference microservices onto Modal's serverless GPUs. The microservice must achieve under 400ms end-to-end latency while scoring >0.85 on RAG evaluation metrics.
What you are building
Build and deploy an enterprise-grade policy assistant microservice using LlamaIndex and Modal for serverless vector retrieval.
How work is evaluated
Evaluates RAG generation precision, context relevance, and response latency on policy queries.
Datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
latency_under_400ms
Validates that latency_ms is under 400 milliseconds.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
context_precision
Precision of retrieved policy passages • target: 0.85 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Learning goals
Build modular RAG index pipelines using LlamaIndex vector store abstractions.
Deploy serverless GPU inference functions on Modal with container caching.
Evaluate retriever relevance and hallucination scores using TruLens or Ragas.
Resources
Dataset of 200 standard commercial property policy filings, endorsements, and synthetic customer inquiry pairs.
Challenge setup
Evaluates RAG generation precision, context relevance, and response latency on policy queries.
Challenge input
JSON object with query, policy_id, and context_documents
LlamaIndex
Assigned framework for data indexing and retrieval.
Modal
Assigned infrastructure tool for serverless GPU execution.
Evaluated output
JSON object with answer, retrieved_node_ids, and latency_ms
- Validates that latency_ms is under 400 milliseconds.
- Precision of retrieved policy passages • target: 0.85 • range: 0-1
- Benchmark: Enterprise RAG Speed & Accuracy Benchmark
- Context Precision target: 0.85
- 1 public reference case
- Python execution harness
- Python sandbox (unavailable on Versalist)
Shows the first task, up to three tools, and up to three evidence items.
View technical recipe
Configured tools
- Modal · Required
- Llama Index · Optional
- Exa · Optional
- Modal · Required
Evaluation contract
- latency_under_400ms · Weight 1
- context_precision · Weight 1
Metrics
- context_precision · Target 0.85
Benchmarks
- Enterprise RAG Speed & Accuracy Benchmark
- general
Reference cases
- 1 public reference case
Workflow tasks
- Task 1
Input: JSON object with query, policy_id, and context_documents
Output: JSON object with answer, retrieved_node_ids, and latency_ms
Execution harness: Python
Evaluation tools
- LlamaIndex · RAG Framework
- Modal · Serverless Runtime
Recipe freeze marker
No recipe freeze time is recorded.