A comprehensive analysis and performance evaluation for osteoporosis prediction models

Current status

Many researchers have attempted using machine learning techniques to predict osteoporosis and showed promising results in this field.

A study conducted by Kranthi, Sailaja & Jyothi (2024) that uses deep learning methods, particularly convolutional neural networks (CNNs), to automate the interpretation and analysis of images by leveraging extensive medical image datasets. Their process includes converting raw image data into structured formats to facilitate accurate analysis. Their proposed model achieved an F1-score of 93.22% and a classification accuracy of 91.16%. However, the authors stated some limitations such as overfitting and the complexity of training the model. These issues highlight the need for optimization and substantial computational resources to enhance model performance and reliability.

Sisodia, Nayak & Boghey (2024) developed a hybrid CNN-deep neural network (DNN) model to predict stock and index prices for Bank Nifty, leveraging historical trading data for feature extraction and accurate forecasting and achieving a prediction accuracy of 97.48%. Despite the high accuracy, the authors highlight some of the limitations such as overfitting, the need for robust computational resources, and the complexity of training a hybrid model.

Another recent study conducted in 2021 highlighted the effectiveness of a learning network known as DeepDXA in evaluating bone mineral density (BMD) using pelvis X-rays (Ho et al., 2021). By examining data within the X-ray images, this model can predict BMD values with precision. DeepDXA presents an alternative in comparison to conventional methods like dual-energy X-ray absorptiometry (DXA), which necessitates specialized equipment and trained personnel. Moreover, DeepDXA has demonstrated performance in predicting osteoporosis from hip X-ray images with an accuracy of 88%, allowing for cost-effective identification of osteopenia and osteoporosis (Ho et al., 2021). The findings of this research could have an impact on the accuracy and ease of diagnosing osteoporosis, potentially improving outcomes by enhancing the precision and usability of osteoporosis diagnosis in clinical environments. Other related work can be summarized in Table 1. The purpose of this comparison to related work is to emphasize the advancements in machine learning and deep learning techniques applied to osteoporosis prediction and similar medical data analysis tasks. By doing so, the study aims to highlight the existing gaps and limitations in current methods that it seeks to address, showcasing the effectiveness and novelty of our approach.

Limitations

Recent studies have proven the significant advancement in osteoporosis prediction made by machine learning based on image and non-image medical data and questionnaires of patients. Fusing a wide range of clinical records, genetic data, and patient-reported outcomes has increased the predictive precision of ML models.

1. Data integration and standardization: Combining data integration from a mix of sources such as clinical records, genetic data, and questionnaires into one harmonious format remains a significant challenge due to discrepancies in data formats and standards. This requires effective techniques in data fusion (Ebbehoj et al., 2022).

2. Privacy and security: Medical data are highly sensitive and cannot be exposed to the public domain. Among these, data handling and processing for protection must comply with regulations such as HIPAA and GDPR (Vitabile et al., 2019).

3. Model interpretability: ML models, especially deep learning models, are considered a black box. The interpretability of such models and their predictions is quite crucial for clinical acceptance and trust (Cui et al., 2023).

4. Generalizability and bias: ML models being trained on specific datasets risk generalizing on other populations due to differences in demographic characteristics and clinical practices. Correcting biases in training data is essential, hence these models must be validated on diverse datasets (Amal et al., 2022).

Our study effectively addresses several key limitations identified in the current research for using machine learning (ML) techniques to predict osteoporosis based on non-image medical data and patient questionnaires.

1. Data integration and standardization: Several datasets of femur, spine, and patient questionnaire data obtained from the NHANES 2017–2020 were integrated into our research. By merging the different datasets and incorporating feature selection techniques such as recursive feature elimination (RFE) and mutual information (MI), this study makes sure that the data used is complete and relevant to our research. This approach is mainly instrumental in mitigating the issues that result from the heterogeneity and standardization of data within multi-modal data integration.

2. Privacy and security: The fact that NHANES data is an openly accessed dataset would mean that its uses should be subject to all data protection regulations. Future work may be strengthened by explicitly addressing how privacy and security are maintained, particularly in line with guidelines set by HIPAA and GDPR, to enhance trust and applicability for real-world clinical settings.

3. Interpretability of models: Our research makes use of sequential deep neural networks (DNN), convolutional neural networks (CNN), and recurrent neural networks (RNN) for prediction. The authors provide transparency in how such models work and which features are most influential by detailing the architecture and process of feature selection. This type of transparency improves the interpretability of models, which is a critical limitation in the deployment of ML in clinical practice.

4. Generalizability and bias: Using a large, diverse dataset from NHANES helps to enhance the generalizability of our findings. However, our study does point out the limitation of working with data from a particular demographic (the Hispanic population) which could not capture all variations in patient demographics. Future research may do this by targeting more diverse populations to ensure the robustness of the model in a different demographic group.

Data preparation

One of the first data preparation steps is to ensure there is no missing or incorrect data that could affect the decision-making process. The dataset size affects the process of processing and analyzing (Kopperdahl et al., 2019). The datasets used in this article were NHANES 2017–2020 (NHANES, 2020), and are of the Hispanic population who are more likely to have osteoporosis than any other ethnicity, making them a great selection for this study. The Femur, Spine, and Questionnaire datasets were collected from the CDC Center for Disease Control and Prevention. After first analyzing the data and comparing these three datasets, some data rows were discarded since some of those patients were not included in the questionnaire data. This could be because those patients refused to participate in the questionnaire, or their information was incorrectly collected. The Femur dataset contains 3,545 rows, while the Spine dataset contains 1,889 data rows.

After comparing these datasets to the questionnaire dataset, a few data rows were omitted from the resulting dataset as not all patients participated in the survey that was conducted by CDC center, National Center for Health Statistics (CDC, 2021b). Only 1,725 and 3,544 data rows for spine and femur datasets, respectively, remain in this study. The resulting datasets were only two datasets which were a result of the merging process of the Spine and Femur datasets with the Questionnaire dataset based on a common identifier, which is the patients’ sequence number. These two resulting datasets were SpineOsteo and FemurOsteo Datasets.

All NHANES 2017–2020 sample individuals 50 years of age or older were eligible. Pregnant women were not allowed to take the DXA test. Participants who were not pregnant but had been omitted from the test were eligible non-respondents. The following are the reasons for being disqualified from the DXA test:

• Pregnancy (confirmed by a pregnancy test).

• Self-reported prior week exposure to radiographic contrast chemicals such as barium or dyes.

• Weight exceeding 450 pounds that was measured (DXA table restriction).

The Shepherd Lab assessed and evaluated each participant and phantom scan using conventional radiologic methods and study-specific protocols created for the NHANES (NHANES, 2023). All femur scans performed between 2017 and March 2020 were analyzed using the Hologic software, APEX v4.0 (Hologic). The correctness and consistency of the findings were confirmed by the Shepherd Research Lab’s expert examination of 100% of the participant scans that were subjected to analysis (NHANES, 2023).

Feature selection

Selecting the best features that have a high correlation with the target variable is a very important step in the process of analyzing and building a classification model (Chen et al., 2019). There are many supervised and unsupervised algorithms for feature selection, as shown in Fig. 1, where the supervised approach uses the target label or variable, unlike the unsupervised algorithm, which doesn’t need access to the target variable (Oleszak, 2023).

• Recursive feature elimination (RFE) selects features in a recursive manner by training the model and eliminating the least significant feature(s) at each iteration (Samb et al., 2012). It assesses the importance of characteristics in relation to the model’s performance. The least significant characteristics were eliminated, and the classification features were repeatedly updated (Chen et al., 2018). It is a wrapper feature selection algorithm that uses a filter-based feature selection algorithm internally to build an efficient classifier. In this article, RFE is based on the random forest algorithm RF-RFE, which was presented by Breiman (2001) and was designed to select only 30 variables of the datasets for all models.

• Mutual information (MI) serves as a method for uncovering the connections between variables (Priscilla & Prabha, 2021). It explains how to determine the significance of a group of features with the target variable. When the MI value is greater than zero for two variables, they are considered statistically related; if the MI value is less than zero, they are deemed statistically independent. The MI is directly linked to the entropies of the variables (Estévez et al., 2009). In these models, only the 30 features with correlations were selected using this algorithm.

Deep learning models

Deep learning is a subset of artificial intelligence and machine learning. Recent years have seen major advances in this subfield, leading to the development of tools for the comprehensive analysis of complicated medical data. Applying deep learning to the medical sector began in the early 2010s (Litjens et al., 2017), with the highly successful application of DL to image recognition. Significant performances from CNNs have been reported for medical image analysis, allowing automatic detection and classification of diseases from medical imaging data like X-rays, MRIs, and CT scans (McKinney et al., 2020). The use of machine learning algorithms in the health sector has increased over time which indicates the powerfulness of these algorithms in analyzing medical data and feature selection techniques to assist in diagnosis and disease prediction models (Rana & Bhushan, 2023; Salehi et al., 2023). The importance of deep learning methods in various medical image analysis tasks such as segmentation and reconstruction has significantly improved over time by providing higher accuracy and adaptability compared to traditional methods (Gong et al., 2024). The models used are Sequential DNNs, CNNs, and RNNs. Each of these models has a unique set of strengths and applications, especially regarding medical diagnostics.

1. Sequential deep neural networks (DNN): This is a kind of neural network in which neurons are applied layer by layer and each layer processes input data to extract features and make predictions. It has a better performance for structured data and has also been used in much medical research works for predictive analytics (Denoyer & Gallinari, 2014).

2. Convolutional neural networks (CNN): Although majorly used in image analysis, they can still be applied for structured data. They use convolutional layers to detect local patterns, making them suitable for identifying intricate relationships within medical datasets (Mishra, 2020).

3. Recurrent neural networks (RNN): RNNs are created in a way to maintain a ‘memory’ of previous inputs to be able to handle sequential data. Long short-term memory (LSTM) units can capture long-term dependencies within data and are therefore extremely effective for time-series medical data (Jayawardhana, 2020).

This article discusses exploring the capabilities of learning methods such as DNN, CNN, and RNN in handling medical image data. Our study focuses on categorizing a target variable using a well-structured medical dataset. The constructed models will explore the selected variables from the dataset that are most correlated with the target variable, which in this article is the classification of patients with or without osteoporosis. The data were split into training, validation, and testing sets.

In this article, six models were designed to analyze these datasets using deep learning techniques that are most suitable for the analysis of medical data to develop classification models. These six models were Sequential DNN, CNN, and RNN, and each of these models used two different feature selection algorithms. Notably, all the models utilize the entropy loss function, employ the Adam optimizer, and measure performance using accuracy as the evaluation metric.

Extracted features

In this article, two feature selection algorithms were used to better understand which features have more connection with each other and would benefit the model by using the most suitable features out of 139 features in the datasets. These two algorithms were MI and RFE. The reasons for choosing these algorithms depend greatly on the type of our model and the datasets. RFE and MI methods are suitable for classification models and non-image datasets as they rank the importance of features based on the model’s performance. In addition, MI and RFE algorithms are suitable for analyzing complex and nonlinear data, such as the datasets in this research, since they take into account the nonlinear correlations between features and class labels.

These two algorithms were used to extract important features to build the three classification models. The extracted features are shown in Tables 2 and 3 from SpineOsteo and FemurOsteo, respectively.

Using deep learning to extract important features from the two datasets combined shows how important the role of deep learning algorithms is in this process to ensure that only important features are selected to be trained in the DNN, CNN, and RNN models in this study. The analysis shows that most selected features were from the Femur and Spine datasets (64%) with the RFE algorithm and (54%) with the MI algorithm, and the remaining features were from the Questionnaire dataset.

The existence of 19 similar selected features in both RFE and MI for the spine dataset and 21 similar selected features in both RFE and MI for the femur dataset indicates these features are more important or relevant in both datasets. These qualities are consistently identified as significant by both feature selection methods, indicating that they may give helpful information for the classification process. The overlap in selected features between the two algorithms implies that these features follow similar patterns and interact with the target variable across numerous feature selection procedures. This suggests that these features may have significant discriminating power and might be reliable markers in classification models for distinguishing between distinct groups. Figures 2 and 3 illustrate the selected features resulting from MI and RFE, respectively. As can be noticed from Fig. 2, the most important selected feature using the MI algorithm was if the fracture resulted from severe trauma, and when using RFE, the highest importance registered feature was if the patient ever took prednisone or cortisone daily, as shown in Fig. 3.

Proposed deep learning models

There were many attempts to create the best models that would better understand the data and discover new patterns and relationships between these features. The most crucial stage in building classification models is the feature selection procedure. After a considerable number of attempts to create the best models to correctly analyze the datasets, only six models were designed using two feature selection methods. It is important to state that this article tested the same architecture of the three models—Sequential DNN, CNN, and RNN—on both datasets, and it is proven that these models performed at their best for both datasets.

Each of these models was evaluated using confusion matrix, precision, recall, F1-score, and accuracy after training the models using the SpineOsteo and FemurOsteo datasets for both feature selection algorithms. These calculation metrics formulas are presented as follows:

1. Precision calculates the number of correctly classified patients that have osteoporosis out of all trained values.

   $Precision=\frac{TP}{TP+FP}$ (Japkowicz, 2006).

2. Recall measures the number of patients who had osteoporosis that had identified correctly.

   $Recall=\frac{TP}{TP+FN}$ (George, 2012).

3. F1-score is the mean value between precision and recall.

   $F1score=\frac{2\cdot TP}{2\cdot TP+FP+FN}=2\cdot \frac{precision\cdot recall}{precision+recall}$ (Fioravanti et al., 2018).

4. Accuracy is the proportion of correct predictions (including true positives and true negatives) to total forecasts.

   $Accuracy=\frac{TP+TN}{TP+TN+FP+FN}$ (Chicco & Rovelli, 2019).

Proposed model results

The feature selection algorithms were able to extract the importance and contribution of the features in the classification process. The feature selection process showed that the area measurement of L1 and L3 vertebrae, BMC of L3, L4, and Femur bone, and BMD value of L1 had the highest correlation with the target variable, which was whether the patients had osteoporosis or not. In addition to these values, other medical history information had a significant impact on the classification process, for example, if the mother ever had a broken hip, which was the highest value among other medical information, the patient’s age when the spine broke for the first time and whether he/she was under or above 50 years old, the age at which the patient’s father broke his hip, and if the parents ever had osteoporosis. The two most important features were the age at which the patient broke his/her wrist for the second time, with 27% importance, and the area measurement of the L1 vertebra with 23% importance. Table 19 shows the comparison between the six models where the predicted values are 0 (class of patients without osteoporosis) and 1 (class of patients with osteoporosis).

The models and feature selection methods with the best performance, according to F1-score, as it provides a balanced measurement between recall and precision, are summarized as follows:

* The best model for the SpineOsteo dataset is the CNN model with the MI feature selection algorithm, while the best model for the FemurOsteo dataset is the CNN model with the RFE feature selection. This concludes that the CNN model is the best performing model regardless of the dataset.

** In terms of feature selection algorithms, it appears that Mutual Information (MI) outperforms Recursive Feature Elimination in the majority of scenarios, as it consistently produces higher F1-scores. As a result, for these models and datasets, MI is the best feature selection technique.

Comparison

In comparison to other related work, Alalhareth & Hong (2023) proposed an intrusion detection system to identify attacks against the Internet of Medical Things (IoMT). They used five classifier models: logistic regression (LR), support vector machines (SVM), decision tree (DT), random forest (RF), and LSTM, with mutual information feature selection algorithms. The researchers analyzed and evaluated their models based on accuracy, precision, recall, and other metrics. Their approach was to increase the size of the feature set to identify the accuracy of their feature selection algorithm and how it affected their model’s accuracy by gradually increasing the number of features from 5 to 45 and noticing that the accuracy of all models increased respectively.

In our models, the number of features was set to 30, as increasing the size of the feature set produced overfitting of the models. The accuracy of the six models was at its best when using the Mutual Information selection algorithm, as shown in Table 19, where it can be noticed that models using the Mutual Information algorithm had higher accuracy and F1-scores than models using the RFE algorithm. Table 20 illustrates the comparison between the models presented in Alalhareth & Hong (2023) and our proposed model when using the MI algorithm.

As can be noticed from the comparison above, the accuracy of the proposed models was higher than the other models presented by Alalhareth & Hong (2023), which implies that the type of deep learning models and type of dataset affect the accuracy of models, since some feature types have a great impact and correlation with the target variable. The key point is that the Mutual Information feature selection may minimize the dimensionality of the input data while maintaining the essential information by selecting the most relevant and informative features, thus enhancing the performance and efficiency of our model.

Authors’ novelty

The authors’ novelty can be summarized as follows:

1. Non-image data: Most studies into osteoporosis prediction rely on image data, making non-image data the alternative one in most studies. This new approach significantly extends the range of data types usually used for predictive modeling in medical science.

2. First attempt: This is the first attempt at predicting osteoporosis utilizing a combination of medical records and patient questionnaire data. This new combination of innovative data sources is a new perspective on osteoporosis prediction.

3. Deep learning models: This study discovers how to use advanced deep learning models, namely, sequential deep neural networks, convolutional neural networks, and recurrent neural networks for osteoporosis prediction. This has been done novelly and opens the application of state-of-the-art deep learning techniques on non-image medical data.

4. Effect of feature selection: The study demonstrates how feature selection can help create a better accuracy model. A comparison of MI versus RFE methods used for feature selection is a way of finding possibilities for improvement when developing the predictive model in the health domain.

5. Family medical history: This study highlights the significant role of family medical history in Osteoporosis prediction. This remark points to the potential benefits of integrating detailed medical history information into predictive models in achieving improvements in accuracy and relevance.

These contributions exemplify how deep learning applied to medical data analysis is novel, leading to new research lines and clinical practice for osteoporosis prediction.