Biomedical Evidence Synthesis Agent
Develop a state-of-the-art tool-calling agent designed for reproducible biomedical evidence synthesis. Using the OpenAI Agents SDK, you will orchestrate a multi-turn conversation agent that can autonomously navigate the NCBI E-utilities API. The agent must successfully map user queries (e.g., 'What are the protein products of human genes associated with Type 2 Diabetes?') into a sequence of API calls involving ESearch for record IDs, ELink for cross-database mapping (Gene to Protein/PubMed), and EFetch for data retrieval. To enhance the synthesis, you will integrate Hugging Face Transformers to perform Named Entity Recognition (NER) on retrieved abstracts, ensuring that the evidence synthesized is grounded in specific biological entities. The final system should produce a structured JSON report including provenance (PMIDs, Gene IDs) and a confidence score based on the consistency of the data found across different NCBI databases.
What you are building
The core problem, expected build, and operating context for this challenge.
Build an autonomous agent using OpenAI Agents SDK to retrieve and synthesize cross-database evidence from NCBI E-utilities with Hugging Face-powered entity validation.
How work is evaluated
The agent's performance will be evaluated on its ability to retrieve specific PMIDs and Gene IDs for a known biological query and the accuracy of its synthesized report.
Challenge datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
ID Integrity
Checks if the retrieved Gene ID correctly maps to the input name via NCBI.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
Citation Recall
Percentage of relevant PMIDs retrieved compared to the gold standard. • target: 0.8 • range: 0-1
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Implement multi-turn function calling using the OpenAI Agents SDK for sequential API interactions
Master NCBI E-utilities syntax for cross-referencing Gene, Protein, and PubMed records
Integrate Hugging Face NER models to validate biological entities within unstructured text
Design a reproducible evidence synthesis workflow that tracks data provenance and API versioning
Reference links and supporting material
Sample data for 1 tasks
Challenge setup
The agent's performance will be evaluated on its ability to retrieve specific PMIDs and Gene IDs for a known biological query and the accuracy of its synthesized report.
Challenge input
Human Gene Name (e.g., 'BRCA1')
OpenAI
OpenAI AI model provider
Hugging Face
AI model hub and inference platform
Hugging Face Transformers
AI Engineering Tooling · Developer Tools
Evaluated output
JSON object with 'gene_id', 'protein_id', and 'pubmed_refs' array
- Checks if the retrieved Gene ID correctly maps to the input name via NCBI.
- Percentage of relevant PMIDs retrieved compared to the gold standard. • target: 0.8 • range: 0-1
- Citation Recall target: 0.8
- 1 public reference case
- Python execution harness
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
- OpenAI · Required
- Hugging Face · Optional
- Hugging Face Transformers · Optional
Evaluation contract
- ID Integrity · Weight 1
- Citation Recall · Weight 1
Metrics
- Citation Recall · Target 0.8
Reference cases
- 1 public reference case
Workflow tasks
- Task 1
Input: Human Gene Name (e.g., 'BRCA1')
Output: JSON object with 'gene_id', 'protein_id', and 'pubmed_refs' array
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.