Dynamic temporal reinforcement learning and policy-enhanced LSTM for hotel booking cancellation prediction

Motivation for dynamic temporal reinforcement learning prediction techniques

• In modern business environments, especially in hotel management, accurate prediction of booking cancellation probabilities is crucial for optimizing resource allocation and enhancing customer satisfaction (Wu, Liu & Li, 2021; Song, Chen & Zhang, 2022). Traditional machine learning methods, such as decision trees, support vector machines, and basic neural networks, have been applied to address these prediction tasks. However, these methods often struggle with complex variations and dynamic factors present in real-world scenarios, necessitating more advanced techniques that can dynamically adjust decision strategies to better address these challenges (Zhang, Wang & Zhao, 2021; Ferreira, Silva & Santos, 2023).

• Dynamic temporal reinforcement learning prediction techniques use deep neural networks (such as multi-layer perceptrons) to approximate state-action value functions, optimizing decision strategies by minimizing errors. Unlike traditional supervised learning methods, this approach continually learns from interactions with the environment, allowing it to adapt to changes in complex environments. By dynamically adjusting policies to maximize cumulative rewards, it effectively predicts and optimizes hotel booking cancellation probabilities, outperforming static models in volatile conditions.

Mathematical derivation of dynamic temporal reinforcement learning prediction techniques

Firstly, we define the optimization objective of the problem as follows: (4)${\displaystyle {\theta }^{\ast }=arg\underset{\theta }{max}J\left(\theta \right).}$

Here, J(θ) represents the objective function to optimize, and θ denotes the model parameters. The objective function J(θ) can be expressed as the expected cumulative reward in reinforcement learning: (5)${\displaystyle J\left(\theta \right)={\mathbb{E}}_{{\pi }_{\theta }}\left[\right.\sum _{t=1}^T{\gamma }^{t-1}{r}_t+\frac{\alpha }{2}\sum _{t=1}^T\sum _{i=1}^N{\left(\frac{\partial log{\pi }_{\theta }\left(a_{i,t}|s_t\right)}{\partial {\theta }_i}\right)}^2-\beta \sum _{t=1}^T\text{KL}\left({\pi }_{\theta }\left(\cdot |s_t\right)\left|\right|{\pi }_{\text{prior}}\left(\cdot |s_t\right)\right)+\lambda \sum _{t=1}^{T-1}{\left(V_{\varphi }\left(s_t\right)-\sum _a{\pi }_{\theta }\left(a|s_t\right)Q_{\varphi }\left(s_t,a\right)\right)}^2\left]\right..}$

Here, π_θ is the model’s decision policy, r_t is the immediate reward at time step t, γ is the discount factor. Our goal is to maximize the cumulative reward J(θ) by optimizing the decision policy π_θ. To achieve dynamic temporal reinforcement learning prediction techniques, we need to address the key mathematical derivations. Firstly, we consider the update rule for the state-action value function Q(s, a): (6)${\displaystyle Q\left(s,a\right)={\mathbb{E}}_{{\pi }_{\theta }}\left[\right.r+\gamma \underset{a^{\prime }}{max}{\mathbb{E}}_{{\pi }_{\theta }}\left[\right.r+\gamma \underset{a^{\prime \prime }}{max}{\mathbb{E}}_{{\pi }_{\theta }}\left[\right.r+\gamma \underset{a^{\prime \prime \prime }}{max}Q\left(s^{\left(3\right)},a^{\prime \prime \prime }\right)\left|\right.s^{\left(2\right)},a^{\prime \prime }\left]\right.\left|\right.s^{\left(1\right)},a^{\prime }\left]\right.\left|\right.s,a\left]\right..}$

Here, r is the immediate reward at current state s and action a, s′ is the next state, a′ is the next action chosen according to policy π_θ. We iteratively update the state-action value function Q(s, a) to optimize the policy π_θ. We consider how to use deep neural network approximation in deep reinforcement learning algorithms to approximate the state-action value function Q(s, a) in dynamic temporal reinforcement learning prediction techniques. We use a multi-layer perceptron (MLP) as a function approximator to learn the state-action value function Q(s, a; θ): (7)${\displaystyle Q\left(s,a;\theta \right)\approx Q^{\ast }\left(s,a\right).}$

Here, θ represents the parameters of the neural network, and Q^∗(s, a) denotes the true value of the state-action value function. We update the neural network parameters θ by minimizing the mean square error to approximate the true state-action value function.

Here, θ^∗ represents the optimal parameters optimizing the objective function J(θ), which reflects cumulative rewards in reinforcement learning. π_θ denotes the model’s decision policy, r_t is the instantaneous reward at time step t, and γ is the discount factor. Our goal is to maximize cumulative rewards by optimizing π_θ and utilizing dynamic temporal reinforcement learning prediction techniques to enhance operational decision-making strategies for hotel booking cancellation predictions.

Finally, by employing policy-enhanced reinforcement learning optimization techniques, we refine the decision policy π_θ. We define the policy gradient update rule as follows: (9)${\displaystyle {\nabla }_{\theta }J\left(\theta \right)={\mathbb{E}}_{{\pi }_{\theta }}\left[\right.{\nabla }_{\theta }log{\pi }_{\theta }\left(a|s\right){\mathbb{E}}_{{\pi }_{\theta }}\left[\right.r+\gamma \underset{a^{\prime }}{max}{\mathbb{E}}_{{\pi }_{\theta }}\left[\right.r+\gamma \underset{a^{\prime \prime }}{max}{\mathbb{E}}_{{\pi }_{\theta }}\left[\right.r+\gamma \underset{a^{\prime \prime \prime }}{max}Q\left(s^{\left(3\right)},a^{\prime \prime \prime }\right)\left|\right.s^{\left(2\right)},a^{\prime \prime }\left]\right.\left|\right.s^{\left(1\right)},a^{\prime }\left]\right.\left|\right.s,a\left]\right.+\mu \sum _{p=1}^P\left(\frac{1}{N_p}\sum _{q=1}^{N_p}{\left(f_{{\theta }_i}\left(Z_{i,t+q}\right)-f_{{\theta }_i}\left(Z_{i,t}\right)\right)}^2\right)+\nu {\int }_{{\mathcal{X}}_i}\sum _{r=1}^R{\left|{\nabla }_{{\psi }_i}{f}_r\left(\mathbf{x};{\psi }_i\right)\right|}^{2}d\mathbf{x}+\xi \sum _{s=1}^S\left({\int }_{{\mathcal{Y}}_i}{\left|{\nabla }_{{\varphi }_i}{g}_s\left(\mathbf{y};{\varphi }_i\right)\right|}^{2}d\mathbf{y}\right)+\zeta \sum _{t=1}^T\left(\frac{1}{N_t}\sum _{u=1}^{N_t}{\left(f_{{\theta }_i}\left(Z_{i,u+t}\right)-f_{{\theta }_i}\left(Z_{i,u}\right)\right)}^2\right).}$

By computing the policy gradient ∇_θJ(θ), we update the neural network parameters θ to optimize the decision policy π_θ.

Here, r is the immediate reward at state s and action a, s′ is the next state, a′ is the next action chosen by policy π_θ. We use an MLP to approximate the state-action value function Q(s, a; θ).

Proof in the appendix.

Proof in the appendix.

Motivation for policy reinforcement learning optimization techniques

• Traditional reinforcement learning methods face challenges balancing cumulative reward optimization and policy stability (Wang & Other, 2021; Xue & Other, 2022). Maximizing cumulative reward alone often neglects policy gradient variance and model adaptability in changing environments, which can lead to suboptimal performance in dynamic settings (Zhang, Wang & Zhao, 2021; Sumiea et al., 2023).

• We introduce regularization based on KL divergence to maintain consistency between current and prior policies, enhancing model stability and generalization. By incorporating this regularization into the loss function of the action value network, we not only promote accurate value function estimation but also strengthen policy robustness and efficiency in long-term decision-making. This regularization technique helps mitigate overfitting to transient changes, ensuring that the model maintains both flexibility and reliability in complex hotel booking environments.

Mathematical derivation of policy reinforcement learning optimization

We define the optimization objective function J(θ), which encompasses the cumulative reward and its associated regularization terms: (12)${\displaystyle J\left(\theta \right)={\mathbb{E}}_{{\pi }_{\theta }}\left[\sum _{t=1}^T{\gamma }^{t-1}{r}_t+\frac{\alpha }{2}\sum _{t=1}^T\sum _{i=1}^N{\left(\frac{\partial log{\pi }_{\theta }\left(a_{i,t}|s_t\right)}{\partial {\theta }_i}\right)}^2\right].}$

Here, θ^∗ are the parameters that maximize the objective function J(θ), π_θ is the decision policy defined by parameters θ, r_t is the immediate reward at time step t, γ is the discount factor, and α controls the regularization term.We introduce KL divergence regularization to maintain similarity between the current policy π_θ and the prior policy π_prior: (13)${\displaystyle -\beta \sum _{t=1}^T\text{KL}\left({\pi }_{\theta }\left(\cdot |s_t\right)\left|\right|{\pi }_{\text{prior}}\left(\cdot |s_t\right)\right).}$

Here, KL denotes KL divergence, and β is a hyperparameter controlling the importance of the regularization term.We consider the squared error between the state value function V_ϕ(s_t) and the action-value function Q_ϕ(s_t, a) as an additional loss term to promote accurate value function estimation: (14)${\displaystyle \lambda \sum _{t=1}^{T-1}{\left(V_{\varphi }\left(s_t\right)-\sum _a{\pi }_{\theta }\left(a|s_t\right)Q_{\varphi }\left(s_t,a\right)\right)}^2.}$

Here, λ is a hyperparameter controlling the weight of the loss.To update the gradient of the optimization objective function J(θ), we compute the expected policy gradient for updating the policy parameters θ: (15)${\displaystyle {\nabla }_{\theta }J\left(\theta \right)={\mathbb{E}}_{{\pi }_{\theta }}\left[\right.\sum _{t=1}^T{\gamma }^{t-1}{\nabla }_{\theta }log{\pi }_{\theta }\left(a_t|s_t\right)\left(\sum _{t^{\prime }=t}^T{\gamma }^{t^{\prime }-t}{r}_{t^{\prime }}\right)\left]\right.={\mathbb{E}}_{{\pi }_{\theta }}\left[\right.\sum _{t=1}^T{\gamma }^{t-1}{\nabla }_{\theta }log{\pi }_{\theta }\left(a_t|s_t\right)\cdot \left(r_t+\gamma r_{t+1}+{\gamma }^2{r}_{t+2}+\dots +{\gamma }^{T-t}{r}_T\right)\left]\right..}$

Here, ∇_θ denotes the gradient of the objective function with respect to the parameters θ. In dynamic temporal reinforcement learning, we use the Q function to estimate the long-term returns for each state-action pair and update the Q function as follows: (16)${\displaystyle Q\left(s_t,a_t\right)\leftarrow \left(1-\alpha \right)Q\left(s_t,a_t\right)+\alpha \left(r_t+\gamma \underset{a^{\prime }}{max}Q\left(s_{t+1},a^{\prime }\right)+\frac{\beta }{2}\sum _{t=1}^T\sum _{i=1}^N{\left(\frac{\partial log{\pi }_{\theta }\left(a_{i,t}|s_t\right)}{\partial {\theta }_i}\right)}^2\right.-\gamma \sum _{t=1}^T\text{KL}\left({\pi }_{\theta }\left(\cdot |s_t\right)\left|\right|{\pi }_{\text{prior}}\left(\cdot |s_t\right)\right)+\lambda \sum _{t=1}^{T-1}{\left(V_{\varphi }\left(s_t\right)-\sum _a{\pi }_{\theta }\left(a|s_t\right)Q_{\varphi }\left(s_t,a\right)\right)}^2\left.+\eta \sum _{t=1}^T{\left({\nabla }_{\theta }log{\pi }_{\theta }\left(a_{i,t}|s_t\right)\right)}^2+\delta \sum _{t=1}^{T-1}{\left(V_{\varphi }\left(s_t\right)-\sum _a{\pi }_{\theta }\left(a|s_t\right)Q_{\varphi }\left(s_t,a\right)\right)}^3\right).}$

Here, α is the learning rate controlling the weight between old and new Q values. We update the policy parameters θ using the policy gradient method to maximize the objective function J(θ): (17)${\displaystyle \theta \leftarrow \theta +\eta {\nabla }_{\theta }J\left(\theta \right)+\alpha \sum _{t=1}^T\sum _{i=1}^N{\left(\frac{\partial log{\pi }_{\theta }\left(a_{i,t}|s_t\right)}{\partial {\theta }_i}\right)}^2+\beta \sum _{t=1}^T\sum _{i=1}^N{\left(\frac{\partial log{\pi }_{\theta }\left(a_{i,t}|s_t\right)}{\partial {\theta }_i}\right)}^3+\gamma \sum _{t=1}^T\text{KL}\left({\pi }_{\theta }\left(\cdot |s_t\right)\left|\right|{\pi }_{\text{prior}}\left(\cdot |s_t\right)\right)+\delta \sum _{t=1}^T{\left(\frac{{\partial }^{2}log{\pi }_{\theta }\left(a_{i,t}|s_t\right)}{\partial {\theta }_i^2}\right)}^4+ϵ\sum _{t=1}^T{\left(\frac{{\partial }^{3}log{\pi }_{\theta }\left(a_{i,t}|s_t\right)}{\partial {\theta }_i^3}\right)}^5.}$

Here, η is the learning rate controlling the step size for each update. Policy reinforcement learning optimization techniques in dynamic temporal reinforcement learning are particularly adept at optimizing decision policies in complex environments.

Here, θ^∗ represents the optimal parameters maximizing the objective function J(θ), π_θ is the decision policy, r_t is the immediate reward at time t, γ is the discount factor, and α, β, λ are tuning parameters.

Proof is provided in the appendix.

where θ^∗ denotes the optimal parameters of the objective function J(θ), π_θ is the decision policy, r_t is the immediate reward, γ is the discount factor, and α and β are parameters. By optimizing θ, we aim to maximize cumulative rewards while considering the variance of the policy gradient and the KL divergence term.

See the appendix for the proof.