All reviews of published articles are made public. This includes manuscript files, peer review comments, author rebuttals and revised materials. Note: This was optional for articles submitted before 13 February 2023.
Peer reviewers are encouraged (but not required) to provide their names to the authors when submitting their peer review. If they agree to provide their name, then their personal profile page will reflect a public acknowledgment that they performed a review (even if the article is rejected). If the article is accepted, then reviewers who provided their name will be associated with the article itself.
I have carefully reviewed your point-by-point responses to the reviewers’ comments and the corresponding changes made to the manuscript.
The reviewers raised several important points during the initial review, and you have addressed them all in a comprehensive and satisfactory manner. The addition of new experiments, including hyperparameter tuning for the attention mechanism and a comparative analysis of imputation methods, has significantly strengthened the paper’s methodology rigor. Furthermore, the inclusion of two additional cancer datasets (AML and WT) enhances the generalizability of your findings.
The expanded sections on data preprocessing, biological validation with literature support, and implementation details have substantially improved the clarity and reproducibility of your work. Your transparent handling of all concerns, including unexpected results for the LIHC dataset, is commendable and adds to the scientific integrity of the study.
[# PeerJ Staff Note - this decision was reviewed and approved by Xiangjie Kong, a PeerJ Section Editor covering this Section #]
After careful consideration, we believe the manuscript addresses an important topic and presents promising ideas, but substantial revisions are required before we can consider it for publication.
The reviewers have raised several significant concerns regarding the current version of the manuscript, including:
- Lack of clarity and justification regarding sample selection from TCGA and the criteria used to include or exclude specific data;
- Insufficient methodological detail in key sections such as data preprocessing, feature construction, and pathway integration;
- Absence of a proper comparative evaluation against relevant state-of-the-art approaches;
- Limited reporting of performance metrics and missing statistical analyses to support the results;
- Reproducibility issues due to a lack of access to code, parameters, and full data descriptions.
We are therefore inviting you to submit a major revision that thoroughly addresses all the reviewers’ concerns. Please include a point-by-point response to each reviewer comment and ensure that all changes in the manuscript are clearly indicated.
1. Methodological Clarity and Technical Detail
While the model architecture is comprehensively described, certain aspects of the mathematical formulations and technical implementation lack sufficient clarity. For example, the rationale behind setting the number of attention heads to 16 (Section 3.2) is not justified. Additionally, the hierarchical fusion module (Section 3.3) would benefit from a more intuitive explanation of how stepwise concatenation captures complex omics interactions compared to alternative fusion strategies. Providing pseudocode or a flowchart for the entire pipeline (beyond Fig. 1) could enhance reproducibility.
2. Experimental Rigor and Statistical Validation
The ablation study effectively demonstrates the superiority of hierarchical fusion over bilinear and concatenation methods. However, the statistical significance of performance differences (e.g., in Table 1) is not explicitly tested (e.g., via paired t-tests or confidence intervals). Furthermore, the choice of datasets (BRCA, BLCA, LIHC) lacks justification—why were these specific cancers selected, and how do they represent broader applicability? Including additional datasets (e.g., pan-cancer cohorts) would strengthen generalizability claims.
3. Biological Interpretation and Validation
The DeepSHAP analysis identifies key genes (e.g., MAP3K1, hsa-miR-363) as contributors to recurrence prediction. However, the biological relevance of these features is only briefly discussed. For instance, are these genes supported by prior literature as biomarkers for recurrence in the evaluated cancers? Incorporating pathway enrichment analysis or experimental validation (e.g., in vitro/in vivo studies cited) would strengthen the model’s biological interpretability.
1. Comparison to State-of-the-Art Methods
While the authors compare MultiCPM to LR, SVM, MOGONET, and DeepKEGG, recent advancements in multi-omics fusion (e.g., transformer-based models, graph neural networks) are not addressed. Benchmarking against methods like MOFA+ (Argelaguet et al., 2020) or similar attention-driven frameworks (e.g., Pan et al., 2023a) would better contextualize the novelty and superiority of MultiCPM.
2. Data Preprocessing and Limitations
The preprocessing steps (e.g., 10% missing value threshold, random forest imputation) are reasonable but require validation. For example, how does imputation impact model performance compared to other methods (e.g., k-nearest neighbors)? Additionally, the model’s dependency on high-quality, complete multi-omics data limits its clinical utility, as real-world datasets often suffer from missing modalities. A discussion of strategies to handle incomplete data (e.g., imputation, modality dropout) is needed.
3. Presentation and Language
The manuscript is generally well-written, but minor grammatical errors and ambiguous phrasing occur (e.g., “SNV data features” in Section 4.2 could be clarified as “SNV mutation profiles”). Figure 3’s axis labels in the PDF are illegible in the provided text layer; ensure high-resolution visuals with clear annotations.
The public GitHub repository is commendable. However, the manuscript should specify software dependencies, hyperparameter tuning protocols, and computational resource requirements (e.g., GPU memory, runtime) to facilitate replication. Including a Docker container or conda environment file would further enhance reproducibility.
Major Revisions required to address the above points. The proposed model shows promise, but methodological transparency, biological validation, and broader benchmarking are critical for acceptance.
The paper presents a multi-head attention mechanism for extracting the key information from different biological pathways. It includes a hierarchical fusion module to efficiently capture omics data interdependences. To validate the proposed mechanism, three datasets are used (BRCA, BLCA, and LIHC). In addition, to interpret the proposed approach, DeepSHAP has been used.
While the paper includes the related work, it is merged with the introduction. It better to move the related work to a separate section and include its analysis/limitation.
Figures and tables should be in its place, having it at the end of the paper affects the paper readability.
The deep learning models are data hungry. But, the used datasets are quite small. The biggest one with 403 instance which puts the models learning in a question. Larger datasets are needed for approach validation. In addition, there are a lot of features, do all are needed? (feature reduction in useful in this situation)
The results show the usefulness of the proposed approach. Its results are better than existing techniques.
The introduction misses the summary of the proposed approach and the obtained results. It also do shows the paper organization.
The manuscript presents MultCPM, a deep learning model for cancer recurrence prediction that integrates multi-omics data (mRNA, miRNA, SNV) using a multi-head attention mechanism and a hierarchical fusion strategy. By mapping omics features to biological pathways and extracting pathway-level information via attention, the model captures complex inter-omics relationships and correlations. MultCPM outperforms existing machine learning and deep learning models, including SVM, MOGONET, and DeepKEGG, across three TCGA datasets (BRCA, BLCA, LIHC) in terms of accuracy, AUC-ROC, F1-score, recall, and AUPR. They used DeepSHAP to bring interpretability to the model prediction. Overall, the idea is good. But the paper has serious limitations in terms of reproducibility, more specifically, reproducing clean data they have used as the input to the model though the results are reproducible with the given clean input. It also needs improvement in writing.
Reproducibility Issues:
Issue 1: Identifying positive (recurrence) and negative (no-recurrence) samples --- For example, they obtained 211 breast cancer samples (recurrence + no-recurrence) from TCGA repository. But TCGA has about ~1000 breast cancer samples. Authors need to explain how they selected only 211 samples for this study. It is better to create a summary table with three cancer data showing the breakdown of original samples, number of recurrence and no-recurrence along with the procedure for selection. This could go into the supplementary documentation.
Issue 2: Superficial description of pathway data --- “Finally, the extracted pathway data underwent systematic preprocessing, which included the following steps: (1) excluding pathways associated with specific cancer types; (2) retrieving pathway names from the KEGG database (Kanehisa et al., 2024) and complementing missing information; and (3) removing duplicate entries in the lists of pathway genes and miRNAs.”
To reproduce the outcome of processing, the authors need to provide (a) how many pathways are there in the original dataset? (b) how many pathways were removed? (c) what “missing information” the author is talking about? (d) how many duplicate genes and miRNAs were there?
Issue 3: Not possible to reproduce the outcome of preprocessing ---
“After preprocessing, the final dimensions of the multi-omics features were as follows:
(1) BRCA (mRNA=1000, SNV=100, miRNA=1000)
(2) BLCA (mRNA=1000, SNV=100, miRNA=1000)
(3) LIHC (mRNA=1000, SNV=200, miRNA=1000)”
Authors must provide a supplementary to explain step-by-step reduction in the numbers of features from the original features space, i.e., how they came up with these magic numbers – 1000 mRNA, 100 SNV, and 1000 miRNA. This is the most important data for developing the whole computational framework. Without knowing how to generate this data, nobody can use this framework for their benefit.
Issues in Introduction
Major Issue: Section with multi-omics integration via graph neural network missed some state-of-the-art approaches, such as Graph Convolutional Network-based approaches, MOGONET (Wang et al. 2021) and SUPREME (Kesimoglu and Bozdag, 2023) and Graph Attention Network-based approach MOGAT (Tanvir et al., 2024). So, including these three references will improve the introduction. Since the authors’ approach is based on attention, they must compare their results with MOGAT, which is also attention-based approach.
Minor Issues: Line 58-59: References should be provided.
Major Issue 1: Line 143-146: Need a clear explanation of how the relationship for individual sample is established for every omics and pathways. May be the notations are misleading. Pick an example (maybe draw a figure to explain) and establish your logic behind creating matrix A. It is better to create a supplementary to explain the “Biological Pathway Module.”
Major Issue 2: Line 193: Why is the number of attention head 16? Is there any rationale? The authors need to show some kind of evidence that 16 heads perform better than other number heads. HP tuning might help.
Major Issue 3: Line 200-204 is the biological basis for the proposed “Hierarchical fusion module.” Thus, each of the statements in this section should be substantiated by appropriate references.
Major Issue 4: Line 248: From figure 2 it’s not clear how did the authors traced back to the original features' contribution. Needs little bit more explanation.
Major Issue 5: No mention of HP tuning. If you have already done, please mention the process and report the best HPs in the paper so that other people can reproduce the results. If you have not done HP tuning, then it is better to do it. Also, if you do not intend to do the HP tuning, then provide the used HP values and the rationale behind using the values.
Minor issues:
Line 123: one of the data types is SNV; but in Figure 1 and figure 2 it is written SNA instead of SNV
Line 141: KGEE should be KEGG.
Line 148: What is the dimension of P^i? It should be mentioned for clarity of your architecture. It is better to show the input and output dimensions in the figure for better understanding.
Line 153-154: For the matrix, row size is the number of samples. But it is not clear what the column dimensions are for each omics, which needs to be mentioned for reproduction.
Line 162: “…the model learns specific dependencies between different omics data and……” What specific dependencies are you talking about?
Line 166-167: Confusing statement about Figure 1-B: “It utilizes the relationship matrix between multi-omics data generated by the biological pathway module and the pathways as input.” I do not see the pathways are input to 1-B. Please clarify.
Line 221-222: “The final integrated features not only consolidate all essential information from the multi-omics data, but also capture the complex interactions among different omics levels, …” The 2nd part of this statement seems not right since the overall fusion process is nothing but the concatenation of SNV, SNV, mRNA, and miRNA, meaning SNV is utilized twice. How does this concatenation reflect the complex interactions among different omics levels, need clarification. Rather the bilinear fusion used for comparing the proposed fusion reflects the complex interactions among different omics levels.
Line 299: It is better to write AUC-ROC instead of AUC and AUPR instead of UPR.
Major Issue 1: Authors compared their results with GNN-based multi-omics integration framework MOGONET, which used GCN architecture. They failed to compare their results with the recent multi-omics integration approach MOGAT based on Graph Attention Network (GAT). Since the authors’ approach is based on attention, they must compare their results with MOGAT.
Major Issue 2: Line 371: Figure 3: Results for LIHC cancer are not included. Why? We were able to reproduce the results in Figure 3. We also generated the results for LIHC cancer. It turned out that the single omics, SNV performs better than the combined multi-omics, which raises concerns in the preprocessing steps as mentioned earlier.
Major Issue 3: Ablation experiments: What is done in the paper is not “Ablation experiments,” rather it’s a comparison of different fusion techniques. Need to provide appropriate title for the section. Also, to reproduce the results of bilinear fusion, authors need to provide the details (maybe the code as well) since there are 3 omics.
Major Issue 4: Line 385: “…. Table 2, show that the hierarchical fusion method significantly outperforms direct splicing and bilinear fusion ….” Which is not true, specifically, compared to Concate, the improvement is marginal. My understanding is that bilinear fusion should reflect the complex relationship between two omics and produce better results compared to simple concatenation used in MultiCPM. Again, the reason could be the issues in preprocessing. The author needs to discuss why the fusion using simple concatenation produces better results than bilinear fusion.
Major Issue 5: Line 415-423: This section needs appropriate references to substantiate the claims.
Minor Issues:
1. Line 352: Shouldn’t it be “multi-omics data” instead of “multimodal data”?
2. Line 363: It should be Fig. 3 instead of Fig.2.
3. Line 392: Did you perform HP tuning? It may happen that after HP tuning the results would improve.
4. Line 407-409: Figure 4: It is better to bring one kind of figure to explain the same thing for different omics.
None.
All text and materials provided via this peer-review history page are made available under a Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.