Download
All Corpusome v1.2 files are available on Zenodo under CC BY 4.0.
The full dataset is ~4 GB compressed; individual tier files are listed below.
Zenodo DOI:
https://doi.org/10.5281/zenodo.22150633 — versioned, citable, archived.
https://doi.org/10.5281/zenodo.22150633 — versioned, citable, archived.
Metadata & Index Files
| File | Format | Size (approx) | Description | Download |
|---|---|---|---|---|
| corpus_metadata_catalog_expanded | .parquet / .csv | 26 MB / 65 MB | Master sample catalogue — 190,601 × 15 columns (v1.2: +3,055 disease cohort samples). Includes body_site, modality, source, health_status, etc. | Zenodo ↗ |
| mgnify_human_bodysite_studies | .csv | 180 KB | Index of 774 MGnify studies in the human body-site catalogue — study accessions, body sites, sample counts. | Zenodo ↗ |
| MANIFEST.csv | .csv | 4 KB | File manifest with MD5 checksums and byte sizes for all release files. | ↓ CSV |
16S Amplicon Tier (164,958 samples)
| File | Format | Shape | Description | Download |
|---|---|---|---|---|
| corpus_16s_genus_relab_expanded | .parquet | 164,958 × 978 | Merged 16S genus relative abundances — harmonized terminal genus tokens (g__Bacteroides style). Primary pretraining feature matrix. | Zenodo ↗ |
| corpus_16s_metadata_expanded | .csv | 164,958 × 10 | 16S-tier metadata subset (sample_id, source, study, modality, body_site, region, pipeline, host_age, host_sex, subject_id). | Zenodo ↗ |
| corpus_mgnify_phylum_relab | .parquet | 157,905 × 62 | MGnify-only phylum-level abundances (SILVA SSU r138 labels). | Zenodo ↗ |
| corpus_mgnify_class_relab | .parquet | 157,905 × 101 | MGnify-only class-level abundances. | Zenodo ↗ |
| corpus_mgnify_order_relab | .parquet | 157,905 × 198 | MGnify-only order-level abundances. | Zenodo ↗ |
| corpus_mgnify_family_relab | .parquet | 157,905 × 341 | MGnify-only family-level abundances. | Zenodo ↗ |
| corpus_mgnify_genus_relab | .parquet | 157,905 × 809 | MGnify-only genus-level abundances (SILVA labels — not harmonized with AGP). | Zenodo ↗ |
| mgnify_full_metadata | .csv | 156,866 × 6 | MGnify full pull metadata (sample_id, study, body_site, host_age, host_sex, subject_id). | Zenodo ↗ |
Shotgun Metagenomics Tier (22,588 samples — curatedMetagenomicData)
| File | Format | Shape | Description | Download |
|---|---|---|---|---|
| corpus_shotgun_species_relab | .parquet | 22,583 × 714 | MetaPhlAn3 species relative abundances. Primary shotgun feature matrix. Row index = sample_uid. | Zenodo ↗ |
| corpus_shotgun_pathway_relab | .parquet | 22,512 × 511 | HUMAnN3 pathway relative abundances (MetaCyc). Not all samples have pathway data. | Zenodo ↗ |
| corpus_shotgun_metadata | .csv | 22,588 × 14 | Shotgun-tier metadata including disease, country, platform, study_condition. | Zenodo ↗ |
| multilevel/shotgun/ (5 files) | .parquet | ~22,500 × 20–200 | Per-rank tables (phylum, class, order, family, genus) derived from MetaPhlAn3 lineages. | Zenodo ↗ |
Quick Start (Python)
import pandas as pd
# Load 16S genus table (primary pretraining feature matrix)
X_16s = pd.read_parquet("data/16s/corpus_16s_genus_relab_expanded.parquet") \
.set_index("sample_id") # 164,958 × 977
# Load shotgun species table
X_shg = pd.read_parquet("data/shotgun/corpus_shotgun_species_relab.parquet") \
.set_index("sample_uid") # 22,583 × 713
# Load master metadata
meta = pd.read_parquet("data/metadata/corpus_metadata_catalog_expanded.parquet") \
.set_index("sample_uid") # 190,601 × 15
# Always split by modality before downstream analysis!
meta_16s = meta[meta.modality == "16S"]
meta_shg = meta[meta.modality == "shotgun"]
System Requirements
| File set | RAM needed | Disk (compressed) |
|---|---|---|
| Metadata only | ~500 MB | ~90 MB |
| 16S genus table (164k × 977) | ~8 GB | ~1.2 GB |
| Shotgun species table (22k × 713) | ~1.5 GB | ~200 MB |
| Full corpus (all tables) | ~32 GB | ~4 GB |
For low-memory environments, use pd.read_parquet(..., columns=[...])
to load only the columns you need, or process in chunks.
All parquet files use snappy compression and column-oriented storage.