Primary file: data/metadata/corpus_metadata_catalog_expanded.{parquet,csv}
— 190,601 rows × 15 columns.
| Field | Type | Description | Example |
|---|---|---|---|
sample_uid |
string | Unique sample identifier. Format: study:sample_id for cMD; raw run accession for MGnify. |
AsnicarF_2017:MV_FEI1_t1Q14 |
source |
string | Contributing database.
MGnify_full = full EBI catalogue pull;
curatedMetagenomicData = cMD v3.18;
AGP = American Gut Project;
MGnify = earlier selective MGnify pull.
|
curatedMetagenomicData |
study |
string | Study or project identifier. For cMD: author+year (e.g. AsnicarF_2017); for MGnify: EBI project accession (e.g. MGYS00001234); for AGP: Qiita_10317. |
MGYS00002925 |
modality |
string | Sequencing modality. 16S = amplicon 16S rRNA; shotgun = whole-genome shotgun. Do not merge these tiers — different profilers, incompatible feature spaces. |
16S |
body_site |
string | Harmonized anatomical body site. Values: stool, oral, skin, respiratory, urogenital, milk. Mapped from free-text EBI body-site annotations. |
stool |
region |
string | 16S hypervariable region amplified. Empty for shotgun samples. Relevant for downstream analysis (e.g. V4 for AGP, various for MGnify). |
V4 |
pipeline |
string | Bioinformatics pipeline string (verbatim from source). cMD: MetaPhlAn3 + HUMAnN (cMD 3.18 / 2021-10-14 snapshot); AGP: Greengenes 13_8 closed-ref (deblur precomputed round 1-21). |
MetaPhlAn3 + HUMAnN … |
host_age |
float | Host age in years at sampling. May be NaN when not provided. Range: 0–101 years in AGP; most cMD studies have age. |
32.0 |
host_sex |
string | Biological sex. Values: male, female, or blank. For AGP: derived from questionnaire; for cMD: from original study metadata. |
female |
subject_id |
string | De-identified subject identifier. Long hashed IDs in AGP; short human-readable codes in cMD. Use for longitudinal linking within a study. | MV_FEI1 |
country |
string | ISO 3166-1 alpha-3 country code of sample collection. Populated only for cMD shotgun tier (42 countries). Empty for MGnify and AGP in this release. | USA |
sequencing_platform |
string | Sequencing instrument. Examples: IlluminaHiSeq, IlluminaNovaSeq, IlluminaMiSeq, IonProton. Relevant as a batch effect co-variate. |
IlluminaHiSeq |
disease |
string | Disease/diagnosis label from original study (shotgun tier only). 141 distinct conditions. healthy for controls. |
IBD |
study_condition |
string | Study arm label, e.g. control, case, treatment. More granular than health_status. |
control |
health_status |
string | Curated health status. Values: healthy (14,566), diseased (8,022), unlabeled (164,958 — entire 16S tier). Use with caution for pretraining; do not filter out unlabeled samples. |
healthy |
| File | Shape | Content | Taxonomy |
|---|---|---|---|
data/16s/corpus_16s_genus_relab_expanded.parquet |
164,958 × 978 | 16S genus relative abundances (harmonized). Row index = sample_id; columns = genus tokens like g__Bacteroides. |
Greengenes 13_8 + SILVA SSU, harmonized at terminal genus |
data/16s_mgnify_full/corpus_mgnify_phylum_relab.parquet |
157,905 × 62 | Per-phylum relative abundances (MGnify full pull only). | SILVA SSU r138 |
data/16s_mgnify_full/corpus_mgnify_genus_relab.parquet |
157,905 × 809 | Per-genus relative abundances (MGnify full pull only, SILVA labels). | SILVA SSU r138 |
data/shotgun/corpus_shotgun_species_relab.parquet |
22,583 × 714 | Shotgun species relative abundances (MetaPhlAn3). Row index = sample_uid. |
MetaPhlAn3 (mpa_vJan21_CHOCOPhlAnSGB_202103) |
data/shotgun/corpus_shotgun_pathway_relab.parquet |
22,512 × 511 | HUMAnN3 pathway relative abundances (MetaCyc). Not all samples have pathway data. | HUMAnN3 + MetaCyc |
data/multilevel/shotgun/ |
— | Shotgun per-rank tables (phylum, class, order, family, genus) derived from the species table by collapsing MetaPhlAn3 lineages. | MetaPhlAn3 |
Executed via pipeline/run.sh. Numbered scripts in pipeline/.
Data Acquisition
curatedMetagenomicData v3.18 (2023 snapshot) via R package;
American Gut Project (Qiita study 10317) via Qiita public API;
EBI MGnify full human body-site catalogue via SSU taxonomy tables API
(mgnify_full_download_manifest.csv records per-study provenance).
16S Harmonization
Terminal genus tokens extracted from Greengenes 13_8 (AGP) and SILVA SSU r138 (MGnify)
lineage strings. Tokens prefixed g__; genus-level columns retained only
if present in ≥10 samples corpus-wide. Final vocabulary: 977 genus tokens.
Shotgun Profiling (cMD tier)
MetaPhlAn3 (mpa_vJan21_CHOCOPhlAnSGB_202103) species-level relative abundances; HUMAnN3 (v3.0) pathway abundances (MetaCyc). Source: curatedMetagenomicData 3.18 / 2021-10-14 snapshot. Not reprocessed from raw reads in this release.
Metadata Harmonization
Body-site labels mapped to controlled vocabulary (stool / oral / skin /
respiratory / urogenital / milk) from free-text EBI environment biome annotations.
Health status unified across disease and study_condition
fields to health_status ∈ {healthy, diseased, unlabeled}.
Quality Checks
Duplicate sample_uid check (0 duplicates);
relative abundance sum check (all rows sum to ≤1.0 ± 1×10⁻⁶);
minimum read count filter (samples with <10k reads excluded upstream by EBI/cMD).
See documents/validation_summary.json.
| Version | Date | Samples | Key Changes |
|---|---|---|---|
| v1.2 current | 2026 | 190,601 | Added 3,055 samples from 7 disease-specific cohorts: preclinical AD gut shotgun (Ferreiro 2023, n=164), AD brain tissue 16S (n=88), lung cancer TCGA/Poore2020 (n=1,549), BALF mNGS lung cancer vs. infection (n=400), Hartwig metastatic lung WGS (Battaglia 2024, n=576), T2D gut shotgun Qin2012 (n=134) and Karlsson2013 (n=144). Added brain as 7th body site. |
| v1.1 | 2026-07 | 187,546 | Added full EBI MGnify human body-site catalogue pull (156,866 samples). Per-rank tables at phylum–genus. All non-stool body sites grew by 1–2 orders of magnitude. |
| v1.0 | 2024 | 30,680 | Initial release. curatedMetagenomicData + American Gut Project + selective MGnify pull. Shotgun tier (22,588) + preliminary 16S (8,092). |