-
Notifications
You must be signed in to change notification settings - Fork 1
3. Download files
leADS requires a set of object files to run the core commands along with test samples to train and predict pathways. The test samples can either be used to train or test the leADS model. Please download these files from figshare. Once you have downloaded the leADS_materials.zip file, unzip it and make sure you obtain the three folders: objectset/, model/, and dataset/, as depicted below:
Note: This tree structure for the directory was generated using the tree command in the terminal (on Linux) and in the command prompt (on Windows).
leADS_materials/
├── objectset/
│ ├── biocyc.pkl
│ ├── pathway2ec.pkl
│ ├── pathway2ec_idx.pkl
│ ├── hin.pkl
│ ├── pathway2vec_embeddings.npz
│ └── ...
├── model/
│ ├── leADS.pkl
│ └── ...
└── dataset/
├── biocyc21_X.pkl, biocyc21_Xe.pkl, biocyc21_y.pkl, biocyc21_species.pkl, leADS_samples.pkl
├── three_ecoli/
│ ├── MG1655
│ │ └── 0.pf
│ ├── EDL933
│ │ └── 0.pf
│ └── CFT073
│ └── 0.pf
├── golden_X.pkl, golden_Xe.pkl, golden_y.pkl
├── cami_X.pkl, cami_Xe.pkl, cami_y.pkl
├── symbionts_X.pkl, symbionts_Xe.pkl
├── hots_4_X.pkl, hots_4_Xe.pkl
├── delicious_X.pkl, delicious_y.pkl
├── birds_X.pkl, birds_y.pkl
└── ...
A short description of the contents of the above folders is given below.
In this folder, 8 core object files are provided that contain various pathway and enzyme information. These files are important for preprocessing, predicting, and training leADS. We will use the following five object files in this wiki:
| File | Description | Size |
|---|---|---|
| biocyc.pkl | An object containing the preprocessed MetaCyc database in the form of pathway ids, EC numbers, reaction ids, gene names, and gene ids, etc. | 91.8MB |
| pathway2ec.pkl | A matrix file representing the pathway-enzyme association. It contains 2526 pathway indices shown in the first column and 3650 enzymes (represented as EC numbers indices) in the remaining columns. | 81.0kB |
| pathway2ec_idx.pkl | A matrix of pathway2ec association indices. | 29.4kB |
| hin.pkl | A sample of heterogeneous information network. | 10.5MB |
| pathway2vec_embeddings.npz | A matrix file containing a sample of embeddings using RUST-norm. The rows (22593) correspond to the pathway, enzyme, and compound embeddings and the columns (128) represent the features. These features can be generated using pathway2vec. | 11.6MB |
Here, we show you a visual depiction of some of the object files to help deepen your understanding.
The biocyc.pkl file contains the preprocessed MetaCyc database. Genes, proteins, enzymes, reactions, pathways, and compounds are all represented as dictionaries containing the individual IDs for each of the 6 categories. This file can be obtained by following the steps highlighted in prepBioCyc
biocyc.pkl
├── list_kb_paths
│ ├── metacyc # The MetaCyc database
│ └── ... # Remaining databases (e.g. EcoCyc)
├── processed_kb
│ ├── metacyc
│ │ ├── 0 # Protein related info
│ │ ├── 1 # Compound related info
│ │ ├── 2 # Gene related info
│ │ ├── 3 # Enzyme related info
│ │ ├── 4 # Reaction related info
│ | └── 5 # Pathway related info
│ └── ...
├── protein_id # A 2-tuple (dictionary) representing protein ids and their indices
├── gene_id # A 2-tuple (dictionary) representing gene ids and their indices
├── enzyme_id # A 2-tuple (dictionary) representing enzymatic reaction ids and their indices
├── compound_id # A 2-tuple (dictionary) representing compound ids and their indices
├── reaction_id # A 2-tuple (dictionary) representing reaction ids and their indices
├── pathway_id # A 2-tuple (dictionary) representing pathway frame ids and their indices
├── ec_id # A 2-tuple (dictionary) representing EC numbers and their indices
├── gene_name_id # A 2-tuple (dictionary) representing gene names and their indices
├── go_id # A 2-tuple (dictionary) representing GO (gene ontology) ids and their indices
└── ...
The pathway2ec.pkl file contains the pathway-enzyme associations with the values in the enzyme columns depicting the number of times an enzyme contributes to the pathways shown. For example, in the table below (after including pathway and EC numbers from "biocyc.pkl"), the enzyme ketol-acid reductoisomerase (EC-1.1.1.86) contributes 1 time to the L-valine biosynthesis pathway but does not contribute to any of the other pathways shown in the table.
| Pathway | EC-1.1.1.86 | EC-1.3.1.9 | EC-2.1.1.79 | EC-2.2.1.6 | EC-2.6.1.42 | EC-2.6.1.13 | EC-3.5.3.1 | EC-4.2.1.59 | EC-6.2.1.3 | EC-6.3.2.M5 |
|---|---|---|---|---|---|---|---|---|---|---|
| L-valine biosynthesis | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 0 | 0 |
| L-arginine degradation VI (arginase 2 pathway) | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 0 |
| cyclopropane fatty acid (CFA) biosynthesis | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| palmitate biosynthesis II (bacteria and plants) | 0 | 7 | 0 | 0 | 0 | 0 | 0 | 7 | 2 | 0 |
| jasmonoyl-amino acid conjugates biosynthesis I | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 |
The pathway2vec_embeddings.npz is a matrix file corresponding to the embeddings of pathways, EC numbers, and compounds. These features are generated using pathway2vec. For example, after including pathway and EC numbers from "biocyc.pkl" in the first column and excluding compounds, the table can be seen as:
| Pathway and EC | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
|---|---|---|---|---|---|---|---|---|---|---|
| L-valine biosynthesis | 0.089106 | 0.092924 | 0.089035 | 0.101823 | 0.072792 | 0.083173 | 0.096259 | 0.064823 | 0.071481 | 0.094392 |
| methylquercetin biosynthesis | 0.112329 | 0.075717 | 0.087717 | 0.094391 | 0.081035 | 0.074514 | 0.095572 | 0.072581 | 0.068458 | 0.096449 |
| cyanide degradation | 0.073566 | 0.094817 | 0.087664 | 0.099661 | 0.089182 | 0.103727 | 0.093147 | 0.093047 | 0.083330 | 0.095017 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| EC-1.1.1.10 | 0.095318 | 0.094138 | 0.097567 | 0.087115 | 0.084483 | 0.098668 | 0.078173 | 0.091465 | 0.086675 | 0.086497 |
| EC-1.1.1.100 | 0.047987 | 0.096748 | 0.092529 | 0.092395 | 0.116745 | 0.092556 | 0.106274 | 0.107414 | 0.079025 | 0.098948 |
| EC-1.1.1.101 | 0.090137 | 0.085566 | 0.087589 | 0.089496 | 0.082936 | 0.088855 | 0.083835 | 0.091411 | 0.085721 | 0.090588 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
In this folder, 4 pre-trained models are provided to predict metabolic pathways using the datasets described in the dataset/ section. We will be using "leADS.pkl" in this wiki:
| File | Description | Size |
|---|---|---|
| leADS.pkl | A pretrained model generated using files- biocyc21_Xe.pkl and biocyc21_y.pkl with nPSP (k=50), ensemble size 10, and per% = 70%. This model was trained using the class-labels (pathways) approach. | 763.7MB |
In this folder, 23 datasets are provided to predict, train, and evaluate metabolic pathways using the pre-trained leADS model (e.g., "leADS.pkl") or to train a new model. We will use 20 datasets that are categorized into the following three types: 1)- pathway training data, 2)- pathway test data, and 3)- other multi-label data.
The following four files can be used to train leADS. Biocyc (v21) tier 2 and 3 PGDBs were processed using prepBioCyc.
| File | Description | Size |
|---|---|---|
| biocyc21_X.pkl | A matrix file of 9429 organisms, whose information is extracted from Biocyc (v21) tier 2 and 3 PGDBs. Columns (3650) for each organism, represent EC number indices filled with integer values indicating the abundance of ECs for that organism. | 27.1MB |
| biocyc21_Xe.pkl | A matrix file of 9429 organisms, whose information is extracted from Biocyc (v21) tier 2 and 3 PGDBs. Columns (3650) of each organism, represent EC number indices filled with integer values indicating the abundance of EC number indices and embeddings for that organism. | 79.8MB |
| biocyc21_y.pkl | A binary matrix indicating the presence/absence of pathway indices (2526 entries) for each of the 9429 organisms. | 67.6MB |
| biocyc21_species.pkl | A metadata in a tuple format (folder id, taxa id, species) consisting of file metadata information, extracted from Biocyc (v21) tier 2 and 3 PGDBs. | 6.47MB |
| leADS_samples.pkl | A file containing a list of integer indices of 4752 samples from biocyc21_Xe.pkl. This file was generated during training "leADS.pkl". | 146.7KB |
The following table depicts biocyc21_X.pkl (after including taxa and species information from "biocyc21_species.pkl"), where the first column represents the taxonomic identifiers (see NCBI Taxonomy) and the second column represents species name. The remaining columns represent EC numbers.
| Taxa | Species | EC-1.1.1.10 | EC-1.1.1.101 | EC-1.1.1.102 | EC-6.4.1.4 | EC-6.4.1.5 | EC-6.4.1.6 | EC-6.4.1.7 | EC-6.4.1.8 | EC-6.4.1.b | EC-6.5.1.8 | EC-6.6.1.1 | EC-6.6.1.2 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| TAX-887700 | Acetobacter aceti | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.0 | 1.0 |
| TAX-1048834 | Alicyclobacillus acidocaldarius | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 |
| TAX-521098 | Alicyclobacillus acidocaldarius | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 |
| TAX-1035194 | Aggregatibacter actinomycetemcomitans | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| TAX-1089447 | Aggregatibacter actinomycetemcomitans | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
The following table depicts biocyc21_y.pkl (after including taxa and species information from "biocyc21_species.pkl"), where the first column represents the taxonomic identifiers (see NCBI Taxonomy) and the second column represents organism information. The remaining columns represent the pathways.
| Taxa | Species | L-valine biosynthesis | L-arginine degradation VI (arginase 2 pathway) | cyclopropane fatty acid (CFA) biosynthesis | almitate biosynthesis II (bacteria and plants) | jasmonoyl-amino acid conjugates biosynthesis I | pyridoxal 5'-phosphate salvage I | adenosine deoxyribonucleotides de novo biosynthesis |
|---|---|---|---|---|---|---|---|---|
| TAX-887700 | Acetobacter aceti | 1.0 | 0.0 | 1.0 | 1.0 | 0.0 | 1.0 | 1.0 |
| TAX-1048834 | Alicyclobacillus acidocaldarius | 1.0 | 1.0 | 0.0 | 1.0 | 0.0 | 0.0 | 1.0 |
| TAX-521098 | Alicyclobacillus acidocaldarius | 1.0 | 1.0 | 0.0 | 1.0 | 0.0 | 0.0 | 1.0 |
| TAX-1035194 | Aggregatibacter actinomycetemcomitans | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 1.0 | 1.0 |
| TAX-1089447 | Aggregatibacter actinomycetemcomitans | 1.0 | 0.0 | 0.0 | 1.0 | 0.0 | 1.0 | 1.0 |
The following data can be used to perform pathway prediction and evaluation of the pre-trained leADS model. Please see the mlLGPR and triUMPF repositories on how to obtain and preprocess the data below.
| Files | Description | Size |
|---|---|---|
| three_ecoli/ | This directory contains "0.pf" files for E. coli K-12 substr. MG1655 (TAX-511145), E. coli str. CFT073 (TAX-199310), and E. coli O157:H7 str. EDL933 (TAX-155864). A tutorial on how to use this data type is provided in Tutorial on pathway prediction (Example 1). | 767KB |
| golden_X.pkl, golden_Xe.pkl, golden_y.pkl | This is the Golden dataset in a matrix format where rows correspond to AraCyc, EcoCyc, HumanCyc, LeishCyc, TrypanoCyc, and YeastCyc, respectively. Columns for "*X.pkl", "*Xe.pkl", and "*y.pkl" correspond to 3650 EC number indices, 3778 EC number indices and embeddings, and 2526 pathway indices. | 154KB |
| cami_X.pkl, cami_Xe.pkl, cami_y.pkl | These files correspond to the CAMI low complexity data with the rows representing 40 species. Columns for "*X.pkl", "*Xe.pkl", and "*y.pkl" correspond to 3650 EC number indices, 3778 EC number indices and embeddings, and 2526 pathway indices. | 396KB |
| symbionts_X.pkl and symbionts_Xe.pkl | These files correspond to the symbiont dataset with the rows representing: Moranella, Tremblaya, and a composition of both genomes. Columns for "*X.pkl" and "*Xe.pkl" correspond to 3650 EC number indices and 3778 EC number indices and embeddings. | 13.1KB |
| hots_4_X.pkl and hots_4_Xe.pkl | These files correspond to the Hawaii Ocean Time Series (HOTS) data at 10m (0, 1 rows), 75m (2, 3 rows), 110m (4 row), and 500m (5, 6 rows) ocean depth intervals. Columns for "*X.pkl" and "*Xe.pkl" correspond to 3650 EC number indices and 3778 EC number indices and embeddings. | 172KB |
The three_ecoli data corresponds to the three E. coli strains - E. coli K-12 substr. MG1655 (TAX-511145), E. coli str. CFT073 (TAX-199310), and E. coli O157:H7 str. EDL933 (TAX-155864), that have been used for several benchmarking analysis. The 0.pf file represented below is the main input file to Pathway tools that pathologic uses to make pathway predictions. It contains the annotated enzymes that result from MetaPathways v2 comparing the open reading frames (ORFs) to the MetaCyc database.
Below is an example of "0.pf" file from E. coli strain K-12 substr. MG1655.
ID ecoli-COLI-K12_0_2
NAME ecoli-COLI-K12_0_2
STARTBASE 3734
ENDBASE 5020
PRODUCT Threonine synthase # THRESYN-RXN 4.2.3.1
PRODUCT-TYPE P
EC 4.2.3.1
//
ID ecoli-COLI-K12_0_6
NAME ecoli-COLI-K12_0_6
STARTBASE 8238
ENDBASE 9191
PRODUCT Transaldolase # TRANSALDOL-RXN 2.2.1.2
PRODUCT-TYPE P
EC 2.2.1.2
//
ID ecoli-COLI-K12_0_7
NAME ecoli-COLI-K12_0_7
STARTBASE 9306
ENDBASE 9893
PRODUCT Molybdopterin adenylyltransferase # RXN-8344 2.7.7.75
PRODUCT-TYPE P
EC 2.7.7.75
//
leADS can be used to train any multi-label dataset (without including embeddings). Here, we include two well-known multi-label datasets obtained from Mulan.
| Files | Description | Size |
|---|---|---|
| delicious_X.pkl, delicious_y.pkl | These files correspond to the delicious multi-label dataset. | 4.85MB |
| birds_X.pkl, birds_y.pkl | These files correspond to the birds multi-label dataset. | 1.09MB |