Public evaluation

GlycanIdentification

The solution will be evaluated on the accuracy of glycan identification and quantification from MS data, the efficiency of the database search mechanism, the quality of Qwen2.5-72B generated hypotheses, and the correctness of comparative analysis.

Evaluation type

task based

Challenge

AI-Assisted Glycan Structural Elucidation & Comparison

Difficulty

Advanced

Rigor

Unspecified

Open linked challenge

Evaluation overview

How the linked challenge is judged: tasks, benchmarks, and criteria count.

Tasks

Benchmarks

Criteria

Task templates

Inputs and expected outputs.

Task 1

GlycanIdentification

Evaluates the pipeline's ability to accurately identify known glycans and infer components from provided MS/MS spectra.

Input format

A directory of mzML files containing MS/MS spectra and a JSON file with ground truth glycan structures (GlycoCT format) for a subset of spectra.

Output format

A CSV file listing identified glycans per spectrum, including their GlycoCT string, confidence score, and predicted monosaccharide composition.

Task 2

ComparativeGlycomics

Evaluates the quantification and comparative analysis of glycan profiles between two hypothetical samples (e.g., 'seal' and 'human').

Input format

Two CSV files (e.g., 'seal_glycans.csv', 'human_glycans.csv') each containing identified glycans and their normalized intensity values from MS data.

Output format

A JSON file summarizing differentially expressed glycans with p-values and fold-changes, and a PNG image of a comparative heatmap/plot.

Task 3

AIHypothesisGeneration

Evaluates the Qwen2.5-72B's ability to generate plausible scientific hypotheses or structural suggestions based on partial or ambiguous MS data.

Input format

A JSON file containing partial MS/MS spectral data (m/z, intensity pairs) and relevant biological context, along with prompts for Qwen2.5-72B.

Output format

A Markdown document containing generated hypotheses or structural suggestions from Qwen2.5-72B.

Linked challenge

AI-Assisted Glycan Structural Elucidation & Comparison

The discovery of complex sugars in wild seal milk, rivalling human milk, highlights the need for advanced tools in glycomics. This challenge focuses on building a data science pipeline to analyze complex mass spectrometry (MS) data for the identification, quantification, and comparative analysis of glycans. Participants will develop a system that can process raw MS/MS data, infer potential glycan structures, and compare glycan profiles across different biological samples (e.g., seal vs. human milk). This involves signal processing, pattern recognition, and leveraging spectral databases. The solution will incorporate advanced machine learning techniques, potentially including graph neural networks or deep learning models, to interpret fragmentation patterns and predict glycan structures. Critical to this challenge is the integration of cutting-edge AI. Participants will leverage the Qwen2.5-72B model to assist in hypothesis generation for complex or novel glycan structures based on ambiguous spectral data, and utilize Groq for high-speed inference of spectral features. Furthermore, a BeeAgent-inspired system will automate the search for known glycan structures in publicly available databases, enhancing the efficiency and accuracy of annotation. This challenge has direct applications in nutrition, drug discovery, and biomarker identification.

Open challenge