Predicting hotel booking cancellations using tree-based neural network

Datasets

We used the dataset provided by Antonio, de Almeida & Nunes (2019) for our study. They collected the dataset from a hotel chain in Portugal. For privacy and ethical reason, the name of the hotel chain was kept unexposed. The hotel chain consented to join and authorize the data collectors to access the Property Management System (PMS) data of their two accommodations. The first accommodation, termed H1, is a resort while the second one, termed H2, is a city hotel. Both accommodations are certified with four-stars and have more than 200 rooms. Data for the experiment were recorded during the period between July 2015 and August 2017. The PMS of the hotel chain has confirmed that its database tables do not contain any missing data. However, certain categorical variables, such as “Agent” and “Company”, include “NULL” as one of the categories. The “NULL” in this case is interpreted as “not applicable”, not “missing value”. The “Agent” field of a booking is blank, for instance means that the reservation was not made through a travel agent. The datasets H1 and H2 contain 40,060 and 79,330 booking samples with the same number of features (variables). For each original dataset, there are 31 variables, including 14 categorical variables, 16 numeric variables, and 1 date variable. The variables ‘AssignedRoomType’, ‘RequiredCarParkingSpaces’, and ‘ReservedRoomType’ were removed by using feature importance ranking. The ‘country’ variable was also taken out of the modeling because it caused information leakage since this information was only confirmed and corrected at check-in. Hence, Portugal was considered the country of origin when booking a hotel. Song & Li’s (2008) study found that the phenomenon of seasonality plays a crucial role in the tourism industry which affected the canceled booking ratio. Therefore, all variables related to time or one representing seasonality, including ‘ArrivalDateDayOfMonth’ (date), ‘ArrivalDateMonth’ (month), ‘ArrivalDateWeekNumber’ (yearly week number), ‘ArrivalDateYear’ (year), ‘ReservationStatusDate’ (status date of reservation), were eliminated from the modeling. Table 1 provides information of refined datasets used for model development and evaluation.

Figure 1 shows the annual cancellation rate in both accommodations. During the period, the cancellation ratios of resort H1 were unstable with wide fluctuations in the range from 15% to 40%. Similarly, after having a high cancellation ratio at the beginning of July and August 2015, the cancellation ratios of hotel H2 oscillated between 25% and 50%. In addition, while the overall cancellation ratio of resort H1 was lower than that of hotel H2, its trending line moderately increased. In contrast, the trending line of the hotel H2 slightly decreased. Figure 2 displays the annual cancellation rate in both accommodations. The figure indicates that seasonality played a crucial role in the cancellations behaviors in that period. The cancellation rate dropped to a minimum in the off-peak season months (November and January) and raised to a maximum from June to September.

Overview of the method

Figure 3 describes the main steps in developing our tree-based neural network model. Initially, original datasets were split into training sets (75%) and test sets (25%) using the time series cross-validator. The splitter function generates indices for dividing a dataset of time series data, which is collected at fixed intervals, into two subsets: a training set and a test set. It is important to note that, in each iteration of the splitting process, the test indices must be sequentially higher than those used in the previous split. As such, randomly shuffling the data during cross-validation is not appropriate in this context. In contrast to standard cross-validation techniques, the training sets in successive splits are supersets of the preceding training sets. Time series cross-validation was conducted on the training set to find the best-performing tree-based model. The selected tree-based model was then integrated into an artificial neural network (ANN) to create a tree-based neural network (TNN). Two training sets were used to develop the two TNN models (for H1 and H2). These models were evaluated using two corresponding test sets.

Tree-based learning algorithms

We constructed four models using four tree-based algorithms, including gradient boosting, extreme gradient boosting, Random Forest, and Extremely Randomized Trees, and examined these models using cross-validation to find the best-performing tree-based model which was then integrated into the deep neural network.

Tree-based neural network

Tree-based neural network (TNN) is a novel deep learning approach that combines the advantages of a tree-based learning algorithm and a deep neural network (DNN) to improve the performance of the conventional DNN in dealing with tubular data. Any tree-based learning algorithm can be integrated into DNN to create a TNN model. The detailed principle of TNN is clearly explained in Sarkar’s (2022) study. The TNN is designed with $n$ gradient-boosted layers in which the fully-connected (FC) layer uses feature importance computed by the tree-based algorithm as the neural network’s weights (Fig. 4). In each gradient-boosted layer, the flow of data follows three consecutive steps: (S1)–Training batch tree-based model, (S2)–Exporting batch feature importances from trained tree-based model and using them as weights of FC layer, and (S3)–Training FC layer. The $g^{th}$ gradient-boosted layer adopts outputs from $(g-1{)}^{th}$ layer as inputs. In the first gradient-boosted layer, the FC layer is initialized without weight and it uses computed feature importance as its weights. From the second gradient-boosted layer, the FC layer is initialized with random weights and it iteratively updates its weight by adding the scaled batch feature importance into its current weights. Except for the final gradient-boosted layer using sigmoid as its activation function, other gradient-boosted layers are activated by LeakyReLU. In this study, we used the learning rate of 1 $\times$ 10⁻⁴ and the minibatch of 128. To develop the TNN models, we used three simple DNN: (TNN1) Dense(32) $\leftarrow$ Dense(32) $\leftarrow$ Dense(1), (TNN2) Dense(64) $\leftarrow$ Dense(32) $\leftarrow$ Dense(1), and (TNN3) Dense(128) $\leftarrow$ Dense(64) $\leftarrow$ Dense(1).

Evaluation metrics

To evaluate the performance of the model, several metrics, includingspecificity (SP), recall (RE), balanced accuracy (BA), precision (PR), F1 score (F1), Matthews’s correlation coefficient (MCC), the area under the receiver operating characteristic curve (AUCROC), and the area under the precision-recall curve (AUCPR), were employed. These metrics were calculated using the True Positive (TP), False Positive (FP), True Negative (TN), and False Negative (FN) values.

Tree-base model selection

Table 2 gives information on the time series cross-validation performance of the four tree-based models. For dataset H1, the GB model is the best predictor, followed by the ERT model, RF model, and XGB models with the cross-validation AUCROC values of 0.7691, 0.7676, 0.7619, and 0.7589, respectively. For dataset H2, the RF model works more effectively compared to GB, XGB, and ERT models with corresponding cross-validation AUCROC values of 0.8082, 0.8016, 0.7914, and 0.7909, respectively. Based on the results, the GB model was selected as the tree-based model to be integrated into DNN for datasets H1 while the RF model was selected to be integrated into DNN for datasets H2.

Model evaluation

Table 3 provides results on the performance of the three TNN models compared to the four tree-based models (as baseline models) on the test set. The results show that all TNN models have better performance compared to all tree-based models in terms of AUCROC and AUCPR values. Among these TNN models, the TNN1 model is the best-performing model for dataset H1 while the TNN2 model is dominant over the others for dataset H2. For the dataset H1, the TNN1 model achieves an AUCROC value of about 0.86 and an AUCPR value of about 0.73, followed by the TNN2 and TNN3 models. For the dataset H2, the TNN3 model works more efficiently than the TNN2 and TNN3 models. From these results, there are two opposite trends found in which the simpler TNN model works better in dataset H1 while the more complex TNN model obtains higher performance in dataset H2. The difference in the number of training data may be one of the possible reasons explaining this observation.

Model benchmarking

Table 4 summarizes the results on the performance of the three TNN models compared to the four state-of-the-art models on the test set. These four algorithms include RB-NN (Li & Micchelli, 2000), DNN (Mittal, 2020), XBNet (Sarkar, 2022), Multilayer Perceptron (MLP) (Singh & Banerjee, 2019), and Bayesian Networks incorporated with Lasso regression (BN-Lasso) (Chen et al., 2023). The results reveal that all TNN models outperform the other models in terms of AUCROC. However, for AUCPR, TNN3 performs less effectively compared to the DNN and XBNet models. On the H1 and H2 datasets, TNN1 and TNN2 remain the best-performing models, with AUCROC values of about 0.86 and 0.84, respectively. The TNN1 model also outperforms the other models, as well as the two TNN designs, with AUCPR values of about 0.73 and 0.82, respectively. The XBNet and DNN models are competitive, as their performance is not significantly different from ours. The RB-NN models show the fastest training speed, indicating the most cost-effective methods. Our proposed TNN models take longer to complete one training epoch compared to the other models.

Limitations and future directions

Although combining the strengths of tree-based models and neural networks can enhance interpretability and performance through a unique optimization technique for tabular data, tree-based neural networks like ours, have several limitations besides their innovation and robustness. The combination of tree-based algorithms (e.g., Random Forest, eXtreme gradient boosting) and neural networks increases model complexity and computational intensity. It also raises the risk of overfitting, especially with smaller datasets, requiring careful regularization. Hyperparameter tuning is complex, as both models have numerous settings that need optimization. Data preprocessing can be challenging due to the differing requirements of tree-based algorithms and neural networks. Additionally, tree-based algorithms may not generalize well to unstructured data, and their performance gains over simpler models may be marginal, making the added complexity sometimes unjustified. Finally, there is limited theoretical understanding of how these models interact, leaving some aspects of their behavior underexplored.

To address these limitations, future directions could explore the use of data-centric AI approaches (Wang et al., 2024). Instead of focusing solely on refining the model architecture, data-centric AI emphasizes improving data quality, consistency, and relevance, which could mitigate overfitting and enhance generalization, particularly on smaller datasets (Hussain et al., 2020; Tran & Nguyen, 2024). This approach could streamline preprocessing by standardizing and optimizing data for both tree-based algorithms and neural networks (Wang, Chukova & Nguyen, 2023). Additionally, focusing on creating richer, more representative datasets may reduce the need for extensive hyperparameter tuning and complex model combinations, ultimately simplifying the pipeline while maintaining strong performance.