Predicting no-shows at outpatient appointments in internal medicine using machine learning models

Introduction

The non-attendance to medical appointments is considered highly relevant due to its multifaceted impact on the healthcare system. Firstly, this issue directly affects healthcare by causing delays in diagnosing and treating diseases, which can negatively affect patients’ health. Additionally, missing medical appointments represents a significant waste of medical resources, including healthcare staff time, medical facilities, and operational costs, affecting the efficiency and economy of the healthcare system (Triemstra & Lowery, 2018). The continuity of care is also compromised, especially for patients with chronic diseases, as regular planned care is disrupted. At the access level, this issue can hinder the availability of appointments for other patients, increasing waiting lists and impeding timely access to care, thus generating a significant economic impact on the healthcare system and patients by accumulating opportunity costs and their associated expenses (Loccoh et al., 2018). This phenomenon is a global concern. In 2018, a global meta-analysis of the rate of missed medical appointments was conducted. On average, the no-show rate was 23%, ranging from 4% to 79% depending on the treating medical specialty and the type of appointment (Dantas et al., 2018). To address and reduce the problem of missed medical appointments, hospitals have adopted multiple strategies, including sending appointment reminders through email, phone calls, and text messages (Liew et al., 2009). These strategies aim to improve the accessibility of healthcare and ultimately reduce missed medical appointments. However, despite multiple efforts and strategies implemented by healthcare institutions, the patient no-show rate continues to rise, posing challenges to healthcare service delivery.

This work’s main contribution is identifying a computational model to predict absences in outpatient appointments in the internal medicine department at Fundación Valle del Lili, a high-complexity hospital located in Colombia. The internal medicine department was chosen for its fundamental role in providing comprehensive and specialized medical care to adult patients, addressing a wide range of chronic and complex conditions requiring continuous and personalized attention. Additionally, this area has one of the highest patient flows within the institution, making it an ideal environment to implement and evaluate a computational model to predict absences in outpatient appointments. For this purpose, an institutional database containing historical records of patients treated in this specialty was used. This database includes clinical and sociodemographic characteristics and details of the last scheduled appointment, whose influence on absenteeism was validated by identifying p and q values. A data combination process was carried out during preprocessing to validate the computational model’s effectiveness.

The decision to address the problem using machine learning models is based on the structure of the available data, allowing for the analysis of complex and nonlinear relationships inherent to the clinical context. The Bagging RandomForest model, combined with the Synthetic Minority Oversampling Technique (SMOTE) oversampling technique, was selected for its high accuracy and performance in identifying absenteeism patterns, improving the representation of minority cases without introducing significant bias. Additionally, SHapley Additive exPlanations (SHAP) values were employed to enhance the interpretability of the model, identifying the most influential variables such as the number of previous absences, the day and month of the appointment, and diagnosed diseases. This approach not only strengthened the model’s accuracy but also provided key insights for developing strategies aimed at patients with a higher risk of absenteeism, ensuring a robust, transparent, and practical analysis for managing medical absenteeism. The results of this research provide a starting point for optimizing resource management and improving efficiency in the healthcare sector by developing strategies based on identifying users at high risk of missing their appointments.

Related work

Artificial intelligence techniques are now being used to identify patterns in medical non-attendance and detect patients at risk of missing appointments. These techniques include statistical analysis, machine learning, and deep learning, which can help develop programs to address these gaps. As a result, there is growing research and innovation in this field, with the development of applications that use artificial intelligence to identify patterns and patients prone to non-attendance and apply interventions to reduce this rate. The results of these efforts are highlighted below: Harvey and colleagues conducted radiology absence prediction using regression models with data extracted from electronic medical records. They recorded demographic, clinical, and health service use variables relevant to attendance. Descriptive statistics and logistic regression models were applied to determine whether these data could predict missed scheduled radiology exams. The regression models demonstrated predictive accuracy with an area under the curve of 0.754. The factors most strongly predicted absence were previous absences, days between scheduling and appointment, modality, and insurance type (Harvey et al., 2017). Chua & Chow (2019) developed a predictive model to stratify the risk of nonattendance in specialized outpatient clinics in a public hospital and improve efficiency in the use of resources. Univariate analyses were performed on 16 variables, including demographic data, appointment/visit records, and historical outpatient records. Multiple logistic regression was used to identify independent risk factors for no-shows. From univariate analysis, 11 variables were determined to be associated with no-shows, and six of these (age, race, specialty, delivery time, referral source, and prior visit status) remained independently associated with the model. The model’s predictive ability was assessed by receiver operating curve analysis, with an area under the curve of 72% (Chua & Chow, 2019).

Daghistani et al. (2020) applied predictors of outpatient absence through extensive data analysis using Apache Spark. After conducting several experiments and employing validation methods, it was found that gradient boosting (GB) performed the best, with an increase in accuracy and ROC curve to 79% and 81%, respectively. This study highlights the importance of exploring and evaluating the performance of machine learning models using multiple evaluation methods, as prediction accuracy can vary significantly (Daghistani et al., 2020). Nelson et al. (2019) evaluated complex machine learning models for predicting scheduled hospital attendance with artificial intelligence. They used models based on logistic regression, support vector machines, random forest, AdaBoost, and gradient boost machines, varying in complexity. The high-dimensional gradient boosting machine models performed best, with an receiver operating characteristic (ROC) value of 0.852 and an average accuracy of 0.511. To achieve this performance, 81 variables were required. This study highlights the need for more complex models to predict optimal attendance, reflecting the interaction of multiple patient, environmental, and operational causal factors (Nelson et al., 2019).

The results of these and other research studies not mentioned in this section provide ample information on using artificial intelligence techniques to predict patient absenteeism to medical appointments. Thus, there is a need to evaluate the computational parameters necessary for these algorithms to reduce the incidence rates of non-attendance and implement actions that subsequently support the shortening of this phenomenon. Therefore, it is necessary to parameterize the design and implementation processes of the computational models, in addition to identifying those critical factors that contribute to the increase of no-shows.

Materials and Methods

Data distribution

Historical data were collected from 18,587 patients who attended the Internal Medicine specialty between January 2019 and September 2022. Of this total, 12,153 patients attended scheduled medical appointments, while 6,434 recorded absences. The distribution of these figures is shown in Fig. 1A. It is evident that there is an imbalance in the classes, which could negatively impact the performance of computational models, biasing them towards the majority class.

In the exploration of the database, a variation was observed in the distributions of ages related to the outcome of the last medical appointment, revealing notable differences between different demographic groups. In particular, it was found that older adults show a lower prevalence of missed medical appointments compared to other groups (Fig. 1B). These findings highlight the complexity of factors affecting medical appointment adherence and emphasize the importance of considering multiple variables in healthcare analyses.

Results

Database preprocessing

Data balanced

To balance the distribution of classes in the dataset, three balancing techniques were implemented: RandomSplit, SMOTE, and ADASYN. This process generated three additional datasets. In the first dataset, created with RandomSplit, the number of instances of the Attendances class was randomly reduced to match that of the Absences class. Additionally, the minority class was increased using SMOTE and ADASYN (Fig. 2).

We conducted additional analyses using three datasets generated with Random Split, ADASYN, and SMOTE techniques to compare the distributions of patient characteristics with the original unbalanced dataset (Table 2). The corresponding summary tables are provided in Supplemental Information S1. While the distributions of demographic and clinical characteristics were generally comparable, the statistical significance of variables such as sex (original p = 0.2) and recent hospitalizations (original p = 0.5) changed in the balanced datasets. These changes suggest that the balancing methods could influence the statistical properties of the data. Therefore, these modifications should be carefully considered when interpreting the results and their generalizability to real clinical settings.

Correlation matrix

In Fig. 3, the correlation matrix of the dataset is presented, revealing the lack of linear correlation between the features and suggesting independence among them. This positive independence is observed when analyzing attributes that are expected not to be directly related. The correlation values were calculated using the Pearson correlation coefficient through the corr() method in Python (Molin, 2019). For numeric features, the Pearson coefficient measures the linear relationship between variables, ranging from −1 to 1. For multicategorical variables, these were first encoded into numerical representations before calculating the correlation, allowing their inclusion in the analysis. The low correlation indicates less-defined patterns, benefiting the model’s adaptability to the complexity of the dataset. Moreover, the lack of correlation helps prevent overfitting during model training by avoiding the capture of false relationships, thereby enhancing the model’s ability to generalize more effectively to new datasets (Nasution, Sitompul & Ramli, 2018).

Models

Hyperparameters

Hyperparameter optimization was applied for each combination concerning the database preprocessing, as specified in section Hyperparameters. Table S1 shows the results for each of the classes in these combinations.

Model metrics for each preprocessing method

The machine learning models were evaluated using the hyperparameters derived in the gridsearch process using metrics such as accuracy, recall, and F1-score, as well as both standardized and non-standardized data. Table 5 presents the evaluation of the models with the unbalanced database; Table 6 shows their metrics using the RandomSplit balancing technique; Table 7 presents the model metrics using the ADASYN balancing technique; and Table 8 displays the results with the SMOTE balancing technique. The data normalization process was carried out using the StandardScaler technique.

Table 5:

Results obtained for each model evaluated in two classes with the unprocessed database include the accuracy obtained in the 5-fold cross-validation, the F1 score, and the recall.

In addition, each result is presented for normalized (using the StandardScaler technique) and non-normalized data.

Unprocessed database
Model	Standard scaler			Non standard scaler
	Accuracy cross-validation (%)	F1-score (%)	Recall (%)	Accuracy cross-validation (%)	F1-score (%)	Recall (%)
Decision Tree	79.08 ± 0.51	84.33	87.87	79.06 ± 0.53	84.33	87.87
Random Forest	81.09 ± 0.39	86.16	89.62	81.12 ± 0.54	86.58	90.35
AdaBoost Decision Tree	81.08 ± 0.54	85.98	88.76	81.07 ± 0.54	85.96	88.76
Bagging Random Forest	81.44 ± 0.57	86.49	91.37	81.29 ± 0.57	86.66	91.82
Gradient Boosting	82.02 ± 0.31	87.35	90.15	82.06 ± 0.26	87.37	90.15
Extra Trees	81.25 ± 0.9	86.95	91.33	81.25 ± 0.9	86.95	91.33
SVM	76.83 ± 0.62	84.14	88.64	73.47 ± 1.25	82.25	91.04

DOI: 10.7717/peerj-cs.2762/table-5

Table 6:

Results obtained for each model evaluated in two classes with balancing by RandomSplit include the accuracy obtained in the 5-fold cross-validation, the F1 score, and the recall.

In addition, each result is presented for normalized (using the StandardScaler technique) and non-normalized data.

Random split balanced
Model	Standard scaler			Non standard scaler
	Accuracy cross-validation (%)	F1-score (%)	Recall (%)	Accuracy cross-validation (%)	F1-score (%)	Recall (%)
Decision Tree	75.9 ± 1.09	76.37	74.45	76.02 ± 0.99	76.26	74.22
Random Forest	79.12 ± 1.07	78.67	76.74	78.94 ± 1.02	78.90	76.51
AdaBoost Decision Tree	79.06 ± 1.06	78.16	76.82	79.08 ± 1.21	77.70	76.66
Bagging Random Forest	80.12 ± 1.01	79.42	77.81	79.74 ± 0.87	79.27	77.66
Gradient Boosting	80 ± 1.29	79.88	80.18	80 ± 1.3	79.68	79.65
Extra Trees	79.14 ± 1.03	79.13	76.89	79.14 ± 1.03	79.13	76.89
SVM	73.95 ± 1.47	75	74.83	71.74 ± 0.56	72.07	71.77

DOI: 10.7717/peerj-cs.2762/table-6

Table 7:

Results obtained for each model evaluated in two classes with synthetic balancing by ADASYN include the accuracy obtained in the 5-fold cross-validation, the F1 score, and the recall.

In addition, each result is presented for both normalized (using the StandardScaler technique) and non-normalized data.

ADASYN balanced
Model	Standard scaler			Non standard scaler
	Accuracy cross-validation (%)	F1-score (%)	Recall (%)	Accuracy cross-validation (%)	F1-score (%)	Recall (%)
Decision Tree	76.07 ± 0.31	78.63	75	76.33 ± 0.2	78.95	75.73
Random Forest	82.12 ± 0.81	83.07	79.23	82.29 ± 0.91	82.66	78.99
AdaBoost Decision Tree	84.23 ± 0.58	84.67	83.43	76.97 ± 0.49	79.49	75.73
Bagging Random Forest	83.53 ± 0.39	84.61	82.17	83.9 ± 0.44	84.01	80.86
Gradient Boosting	82.41 ± 0.86	81.07	84.80	82.39 ± 0.84	84.79	81.03
Extra Trees	82.38 ± 0.67	83.26	81.19	82.38 ± 0.67	83.26	81.19
SVM	76.56 ± 0.64	79.68	74.06	82.37 ± 0.28	78.75	84.73

DOI: 10.7717/peerj-cs.2762/table-7

Table 8:

Results obtained for each model evaluated in two classes with synthetic balancing by SMOTE include the accuracy obtained in the 5-fold cross-validation, the F1 score, and the recall.

In addition, each result is presented for both normalized (using the StandardScaler technique) and non-normalized data.

SMOTE balanced
Model	Standard scaler			Non standard scaler
	Accuracy cross-validation (%)	F1-score (%)	Recall (%)	Accuracy cross-validation (%)	F1-score (%)	Recall (%)
Decision Tree	$77.31\pm 0.59$	78.83	70.81	$77.34\pm 0.62$	78.7	70.64
Random Forest	$82.93\pm 0.73$	83.41	80.94	$82.93\pm 0.62$	83.27	80.66
AdaBoost Decision Tree	$84.39\pm 0.59$	84.22	84	$84.41\pm 0.65$	84.23	83.96
Bagging Random Forest	$84.80\pm 0.81$	84.75	83.02	$84.79\pm 1.03$	84.3	81.76
Gradient Boosting	$83.14\pm 0.84$	85.12	81.96	$83.23\pm 0.84$	85.11	81.92
Extra Trees	$83.17\pm 0.66$	83.6	82.21	$83.17\pm 0.62$	83.6	82.21
SVM	$77.76\pm 0.54$	80.44	75.45	$77.61\pm 0.47$	78.29	74.88

DOI: 10.7717/peerj-cs.2762/table-8

For better interpretability of the results, three computational methods (AdaBoost, Bagging, and GradientBoosting) were selected due to their relevant metric values and their stability in the relationship between them. The F1-score metric, especially relevant in scenarios with imbalanced classes, was analyzed. Figure 4 compares the F1-score across different balancing approaches and shows that, in general, the variations between them are minimal. However, the RandomSplit undersampling method shows lower F1-score performance for AdaBoost and Bagging. Since the metrics do not show significant differences, interpretability was prioritized to select the most appropriate model, valuing its ability to provide clear and useful insights in this context.

SHAP values highlight that the most influential variables in predicting medical appointment no-shows include the number of previous no-shows, the month and day of the appointment, the number of prescribed medications, and diagnosed diseases, among others. Gradient Boosting was discarded due to the instability and range of its SHAP values. At the same time, AdaBoost showed greater sensitivity to the variable “Number of Previous Attendance,” whose relatively higher impact could lead to erroneous interpretations. In contrast, Bagging demonstrated a more uniform and stable distribution of SHAP values, emphasizing key variables’ significant and consistent impact. These characteristics set Bagging apart, establishing it as the most reliable and aligned option with the study’s objectives. These results can be observed in the Supplemental Information S2.

Based on the metrics obtained (accuracy, F1-score, and recall) presented in Tables 5, 7 and 8, and the distribution of SHAP values (Fig. 5), SMOTE was selected as the best balancing strategy for the Bagging model. Specifically, SMOTE (Fig. 5A) provides a more uniform and consistent distribution of key variables, such as the number of previous no-shows, the day, the month, and the number of medications. These variables reflect a behavior most closely aligned with the phenomenon of no-shows, reinforcing the model’s reliability. In contrast, with ADASYN (Fig. 5B), variables like the day and time exhibited greater dispersion, while in the unbalanced dataset, the relevant variables showed a less defined impact. Therefore, SMOTE not only improved performance metrics but also ensures a more stable and accurate representation of the factors associated with medical no-shows, solidifying its choice as the optimal strategy for this context.

Figure 6 shows the results obtained with the Bagging RandomForest model with the standardized and SMOTE-balanced data. It includes the confusion matrix for the two classes and their respective AUC values for each combination and class type in the standardized data preprocessing.

Analysis of the confusion matrix reveals adequate discrimination of the class of interest. This differentiation capacity correlates positively with the behavior observed in the ROC curve, whose value (0.89) demonstrates a good class discrimination capacity. Supplemental Information S3 contains the class-wise metric reports of the Bagging RandomForest model for each data preprocessing combination, as well as the feature importances for each of them.

Figure 7 illustrates the feature importance of the Bagging RandomForest model, which align with the impact of the variables presented with the SHAP values.

Discussion

In our study, we evaluated seven computational models to address the issue of medical absenteeism in a dataset with discrete and categorical features. The models analyzed were random forest, decision tree, Bagging, AdaBoost, ExtraTrees, GradientBoosting, and SVM, whose results are presented in Tables 5, 6, 7 and 8. The results revealed that ensemble models such as Bagging, AdaBoost, ExtraTrees, and GradientBoosting outperformed simpler models like random forest and decision tree. We highlighted the metrics obtained by Bagging RandomForest, which demonstrated high precision, sensitivity, and specificity. This model excels in efficiently handling both continuous and categorical features and its robustness against class imbalance, making it a solid option for predicting medical absenteeism.

The Bagging RandomForest model uses the Bagging ensemble technique to build multiple independent decision trees from random subsets of the dataset. Each tree generates a prediction, and the model combines these predictions to produce a more robust final result. This approach reduces variance, improves generalization capacity, and decreases the risk of overfitting, especially in scenarios with imbalanced data. Furthermore, its ability to handle mixed data types (categorical and continuous) makes it particularly effective for complex problems like medical absenteeism.

Our analysis revealed that when the SMOTE balancing technique is combined with standardized data, the Bagging RandomForest model showed superior performance compared to other techniques for class 0 (no-shows). This finding was reflected in an improvement in evaluation metrics, as shown in Fig. 6 and Table 8. The effectiveness of the SMOTE technique lies in its ability to generate synthetic samples uniformly distributed within the feature space, preserving the structure of the minority class data. This is especially relevant given the nature of our dataset. Moreover, the introduction of greater diversity in the dataset improved the model’s generalization capacity. The inclusion of the datasets generated with Random Split, ADASYN, and SMOTE allowed us to evaluate the robustness of our findings under different balancing methods. While these techniques improved the representation of the minority class, they also altered the statistical significance of certain variables, such as sex and recent hospitalizations, which were not significant in the original unbalanced dataset (Table 2). This highlights their impact on the relationships between variables and outcomes. Despite these modifications, the consistency of predictive performance and the robustness of Bagging RandomForest as the selected model reinforce the reliability of the approach. However, it is crucial to interpret statistical associations derived from synthetic datasets with caution.

To enhance interpretability in the selection of computational models, SHAP values were applied. This technique allows for the evaluation of the individual impact of each variable on the model’s predictions. This analysis, complemented by the evaluation of feature importance from the computational model, confirmed the performance of the Bagging RandomForest model. The results showed that the impact of the variables in this model aligns more closely with the real behavior of medical absenteeism, reinforcing its relevance and reliability as a predictive tool in this context. Identifying the most impactful variables through SHAP values in the Bagging model, complemented by a statistical analysis of p-values that align with these results (Table 2), enables data-driven decision-making and provides valuable insights into the key factors in predicting medical appointment no-shows. Variables such as the number of previous no-shows, the day and month of the appointment, and diagnosed diseases emerge as critical determinants (Fig. 5A). These findings have significant practical implications, as they enable a detailed analysis of behavioral patterns associated with absenteeism and offer insights into the factors that determine this phenomenon. This knowledge facilitates the development of personalized strategies, such as reminders adjusted to days with higher absenteeism rates, appointment schedules tailored to identified patterns, and prioritization of follow-up for patients with a history of multiple absences. By focusing efforts on patients most likely to miss appointments, these interventions not only improve operational efficiency but also increase the effectiveness of resource allocation. Furthermore, these insights allow healthcare institutions to proactively address absenteeism, optimizing the patient experience through timely, personalized care and ensuring more efficient management of medical resources. This integration of data-driven insights into strategic planning ensures that interventions are targeted where they are most needed, enhancing overall healthcare outcomes.

Furthermore, leveraging these insights through a computational model supports decision-making and serves as a practical tool for the institution. This model strengthens the ability of institutions to act proactively against medical absenteeism, offering a reliable predictive tool that facilitates data-driven decision-making and optimizes strategic planning. By effectively segmenting the population based on the risk of absenteeism, institutions can focus resources where they are most needed, mitigating operational inefficiencies and improving the patient experience by ensuring more timely and personalized care. This integration of data-driven insights and strategic planning ensures that resources are directed toward critical areas, enhancing interventions’ effectiveness and improving healthcare system outcomes.

Conclusion

This comparative study evaluated machine learning models to classify medical absenteeism using various data encompassing clinical, sociodemographic, and specific characteristics of the patient’s latest medical appointment in the context of internal medicine. The validity of the variables in the structured database was examined by identifying statistical significance values (p and q), revealing the significant influence of several variables. Among these, patient age, type of medical insurance, number of diagnosed diseases, recent hospitalization history, the time, day, and month of appointment scheduling, and the number of previous absences stood out. These findings underscore the importance of various factors in predicting medical absenteeism, providing a more comprehensive understanding of the determinants influencing this phenomenon. Ensemble models outperformed traditional ones in precision and stability, with Bagging RandomForest, GradientBoosting, ExtraTrees, and AdaBoost Decision Tree standing out. Bagging RandomForest proved especially effective in identifying patients at risk of missing their appointments, even with imbalanced or balanced data using techniques such as RandomSplit, SMOTE, and ADASYN. The SMOTE technique allowed the generation of synthetic samples uniformly distributed, preserving the structure of the minority class data and reducing bias toward the majority class.

Despite the low correlation between variables, computational models were able to identify complex patterns and nonlinear relationships. Data standardization also improved accuracy by unifying categorical and discrete features, enabling more efficient processing and greater prediction accuracy. The methodology of the Bagging RandomForest model, which integrates SMOTE and data standardization, achieved a cross-validation accuracy of 84.80 $\pm$ 0.81% and an F1-score of 83.02%, confirming its effectiveness in predicting medical absenteeism in a clinical setting. Additionally, the SHAP values of the model helped identify the most impactful variables in predictions, aligning closely with the natural behavior of absenteeism. Variables such as the number of previous absences, the day and month of the appointment, the number of diagnosed diseases, and prescribed medications emerged as key determinants in the model’s predictions. This approach not only enhanced the model’s interpretability but also provided a solid foundation for developing personalized strategies targeted at the population of interest. For instance, by identifying patients with multiple absenteeism histories, medical institutions can implement personalized reminders, adjust appointment schedules based on specific patterns, or prioritize follow-ups for patients with complex conditions. These strategies allow resources to be allocated efficiently, maximizing intervention effectiveness and improving attendance rates. Knowledge of impactful variables strengthens the ability to make informed, data-driven decisions, adapting actions to real-world contexts and ensuring a more precise and effective approach to appointment management.

This study successfully identified a robust and efficient computational model for predicting medical absenteeism, opening opportunities for future research. It is recommended to include broader sociodemographic characteristics of patients, as these could capture regional behavioral patterns and provide a more complete perspective on the factors influencing absenteeism. Furthermore, scaling the model to other medical specialties would require analyzing the specific behaviors of these specialties and determining relevant variables of interest, as the determinants of absenteeism can vary significantly across clinical contexts. A computational model that identifies patients at higher risk of missing appointments represents a valuable tool for institutional support. This approach not only complements existing strategies to mitigate absenteeism but also helps optimize resource management, personalize interventions, and improve operational efficiency in healthcare systems. These future directions will not only strengthen the model’s applicability but also expand its impact on improving healthcare outcomes.

Supplemental Information