Back to all available

A multimodal CNN model with adaptive modality weighting for classification of breast cancer using mammography images, ultrasound images, and clinical data: A cross-sectional study

algorithms-and-analysis-of-algorithms
artificial-intelligence
computer-vision
data-mining-and-machine-learning
neural-networks

Abstract

Background: Breast cancer remains one of the most prevalent malignancies globally, necessitating accurate, noninvasive diagnostic strategies. This study introduces a novel multimodal deep learning framework utilizing an Adaptive Modality Weighting Convolutional Neural Network (AMW-CNN) for classifying benign and malignant breast masses based on mammography, ultrasound, and clinical data.

Methods: This cross-sectional study included 92 women with biopsy-confirmed breast masses. Imaging data were preprocessed using the Segment Anything Model (SAM), and data augmentation increased the dataset to 10,000 images. We developed and evaluated five multimodal deep learning models (ANN, RNN, Transformer, CNN, and AMW-CNN), alongside their unimodal counterparts. Performance was assessed via 5-fold cross-validation using accuracy, sensitivity, specificity, precision, Matthews Correlation Coefficient (MCC), and Area Under the ROC Curve (AUC). Statistical analyses included McNemar’s test for model comparison and independent t-tests for age differences.

Results: The proposed AMW-CNN model achieved superior performance across all metrics, with an AUC of 99.93%, accuracy of 99.08%, sensitivity of 99.36%, specificity of 98.79%, precision of 98.82%, and MCC of 98.16%. It significantly outperformed other models (p < 0.05). Additionally, AMW-CNN achieved AUCs of 99.98% and 99.96% when used with ultrasound and mammography alone, respectively. Age was also found to be a significant factor distinguishing malignant from benign masses (p < 0.001).

Conclusion: The integration of multimodal imaging and clinical data, guided by adaptive weighting, substantially improves diagnostic accuracy in breast cancer classification. The AMW-CNN model shows robust potential for clinical implementation, although validation on larger, multicenter datasets is warranted.

Ask to review this manuscript

Notes for potential reviewers

  • Volunteering is not a guarantee that you will be asked to review. There are many reasons: reviewers must be qualified, there should be no conflicts of interest, a minimum of two reviewers have already accepted an invitation, etc.
  • This is NOT OPEN peer review. The review is single-blind, and all recommendations are sent privately to the Academic Editor handling the manuscript. All reviews are published and reviewers can choose to sign their reviews.
  • What happens after volunteering? It may be a few days before you receive an invitation to review with further instructions. You will need to accept the invitation to then become an official referee for the manuscript. If you do not receive an invitation it is for one of many possible reasons as noted above.
  • PeerJ Computer Science does not judge submissions based on subjective measures such as novelty, impact or degree of advance. Effectively, reviewers are asked to comment on whether or not the submission is scientifically and technically sound and therefore deserves to join the scientific literature. Our Peer Review criteria can be found on the "Editorial Criteria" page - reviewers are specifically asked to comment on 3 broad areas: "Basic Reporting", "Experimental Design" and "Validity of the Findings".
  • Reviewers are expected to comment in a timely, professional, and constructive manner.
  • Until the article is published, reviewers must regard all information relating to the submission as strictly confidential.
  • When submitting a review, reviewers are given the option to "sign" their review (i.e. to associate their name with their comments). Otherwise, all review comments remain anonymous.
  • All reviews of published articles are published. This includes manuscript files, peer review comments, author rebuttals and revised materials.
  • Each time a decision is made by the Academic Editor, each reviewer will receive a copy of the Decision Letter (which will include the comments of all reviewers).

If you have any questions about submitting your review, please email us at [email protected].