Short-term wind power forecasting through stacked and bi directional LSTM techniques

Data collection

The actual data on wind farms is taken from the Kaggle Competition site (Hongtao, 2012). The dataset includes power, speed, and directional values for wind besides zonal (u), and meridional (v) components. The dataset is compiled at the wind farm level. The total number of instances available in each wind farm dataset are Wind Farm 1: 16,589, Wind Farm 2: 16,586, Wind Farm 3: 17,010, Wind Farm 4: 15,385, and Wind Farm 5: 16,531, respectively. A mapping of data over hours and wind power is shown in supplementary graphs (Figs. S1–S5). It is noteworthy to mention here that the actual dataset contains data from seven farms while only data from five wind farms is being considered. This is done to ensure a fair comparison between the proposed work and the base literature GPeANN (Zameer et al., 2017) and DNN-MRT (Qureshi et al., 2017). Each instance of the dataset contains a deterministic forecast, gathered twice a day, and each measured instance has a time-based resolution of 1 h and 2 days ahead. To calculate power, four power values provided by weather predictors, in the last 48 h having a time difference of 12 h are utilized. The dataset also contained temperature information which is not utilized to make a fair comparison with the base articles. The base articles did not utilize temperature for prediction.

Data pre-processing

The dataset contains weather forecasts for five distinct wind farms along with hourly wind power measurements. During the pre-processing stage, the date and time formats are standardized for observed weather forecasts and power measurements. Metrological forecasts like zonal (u), meridional (v), wind speed (WS), and wind direction (wd) are used as features to get power measurements. The dataset hour offset format is changed to calculate the exact hour of measuring power output. A simple arithmetic mean is applied to get a single candidate feature set for more than one metrological forecast against a specific power measurement,

Besides weather observations like wind speed and wind direction at specific time intervals, wind power also has a huge dependency on historical weather values. Significant improvement in wind power prediction was observed by Zameer et al. (2017) and Qureshi et al. (2017) when previous power values were also included in the feature set. To include some specific number of past observations in the feature set, a variable n is defined. This variable n is passed to a function to get a new feature vector that concatenates past n observations. The process is applied to the whole data set. The normalization procedure is considered good practice for inputting data before presenting data to the network (Hochreiter & Schmidhuber, 1997). To enhance the accuracy and to make scale-free comparisons, data is scaled down in the range of (0,1). With all the above tuning, the following feature set S(t) as shown in Fig. S6 is finalized.

In the figure, n will be the number of past values that will be included in set S(t). Its range will be from 1 to 24 h in the past. The value of n will be finalized during the training phase based on the performance of the model. Input to the LSTM network is a three-dimensional (3-D) array, these dimensions are the number of training samples to be fed, the number of time steps to be included in a single training example, and the number of features. Data were processed using the Python library e.g., Numerical Python (NumPy), and reshaped as a 3-D array. In our case, each sequence is dealt with independently and the number of the time step is 1. Several features will be as per feature set S(t).

Long short-term memory networks

LSTM is a specific RNN that is specially designed to deal with data having long-term temporal dependencies. LSTM is an evolved model of classical ANN. Figure S7 shows the architecture as well as the feature set of RNN. As shown in the figure, the output of the previous state is also fed as an input to the subsequent state along with the current input X_t. Input to an RNN is the feature vector X and Y will be the output vector containing all the hidden states Rt-1,…, Rt+1.

The recursive nature of RNNs makes them ideal when dealing with sequences. However, Hochreiter & Schmidhuber (1997) discovered that the learning process of RNNs becomes drastically slow due to the vanishing gradient problem. This led to the development of LSTM (Althelaya & Mohammed, 2018), a special RNN that uses memory units to store contextual information and gated units to supervise the flow of information. In LSTM repeating hidden unit of RNN is replaced with an LSTM cell. A brief overview of LSTM can be viewed in Hochreiter & Schmidhuber (1997).

The thing that differentiates LSTM from standard RNN is its memory cell shown as C_t as highlighted in Yu et al. (2019). The following steps elaborate on the working of LSTM.

• At some time, t, inputs to an LSTM cell are cell state at timestamp t-1 (C_t−1), the output of the previous hidden state R_t−1, and the input feature vector X_t.

• First, inputs X_t and R_t−1 are passed to the forget gate (F) along with associated weights that in turn apply the sigmoid activation function on it. Based on the output of the sigmoid function, F decides what information to retain or forget. As sigmoid output values range is between 0 and 1. Values that are closer to 0 will be forgotten and values closer to 1 will remain unchanged. The equation for F will be (1)${\displaystyle F=\partial \left(W^f{X}_t+W^f{R}_{t-i}\right).}$ Next, a two-step process is performed before updating the cell state. First inputs X_t, R_t−1, and associated weights Wⁱ are passed to input gate I which uses the sigmoid function to decide which values to update to the cell state. Values nearest to 1 are likely to be updated. In the second step, utilizing the same input, a candidate vector G is created using tanh that holds the new values to be added to the cell state. Tanh function returns a value between −1 and 1. This will help regularize the network and deal better with the vanishing gradient problem. Now we get the new candidate values through pointwise multiplication of I and G. The following equations describe how I and G will be calculated.

• To update the cell state, the previous cell state C_t−1 is multiplied by the output of F which is a forget vector. This results in the dropping of cell state values where relevant forget vector values are near 0. Next pointwise addition is applied to the output of the previous step which is (I⊙G), this results in the new cell state C_t. Mathematically it can be written as (4)${\displaystyle C_t=F\odot C_{t-1}+I\odot G.}$

• Now lastly output will be decided using the output gate (O). To calculate the output, feature vector X_t, hidden state of previous cell R_t−1, and relevant weights W^o are used. Based on these inputs O decides the next hidden state R_t. Hidden state R_t is the prediction based on previous inputs. First X_t and R_t−1 are passed to the sigmoid function of O. Then the output of O will be pointwise multiplied with the resultant vector of tanh(C_t) where C_t is the new cell state. This multiplication will help decide the relevant information that hidden state R_t should carry. Finally, the hidden state R_t and new cell state C_t will be passed to the next LSTM cell. (5)${\displaystyle O=\partial \left(W^o{X}_t+W^o{R}_{t-i}\right)}$ (6)${\displaystyle R_t=O\odot tanh\left(C_t\right).}$

The final output of an LSTM layer is a vector Y containing all the outputs (R_t−1, …R_t+1) and can be represented as Y = [R_t−1, …R_t+1].

Stacked LSTM

Stacked LSTM is the simplest form that enhances the capability of a single LSTM layer network by stacking one or more LSTM layers above it. Figure S8 shows the flow of a stacked LSTM network with a sequence length of 3. In the figure, three recurrent LSTM layers are interconnected where the output of the layer (L-1) i.e., R_tL − 1 is treated as input X_t to the subsequent layer (L). Now X_t in the mathematical representation of simple LSTM is replaced with to get the equation for Layer (L) in case of stacked LSTM.

Bidirectional LSTM

Bidirectional LSTM is another variant of RNN that is found in the literature (Gao, Xu & Yin, 2024). It extends the standard unidirectional LSTM to bidirectional thus enabling it to process input sequences in both time directions simultaneously i.e., backward (future to past) or forward (past to future). Here for each time direction, two separate networks are used to train the model. Bidirectional LSTM is applicable where all time steps of the input sequence are known. Results from both layers are then merged to help prediction. Work done by Althelaya & Mohammed (2018) on stock market prediction shows that bidirectional LSTM produced better results compared with stacked LSTM.

In bidirectional LSTM the working of the forward layer is the same as in the standard LSTM that traverses from time t = 1 to T. However, in the case of the backward layer, the input sequence is fed from t = T to 1. Therefore, a mathematical representation of the LSTM cell in the forward layer will remain the same whereas mathematical expressions of the LSTM cell in the backward layer denoted by the leftward arrow (←) at time t can be written as: (7)${\displaystyle \overrightarrow{F}=\partial \left(W^{\underset{f}{\leftarrow }}{X}_t+W^{\underset{f}{\leftarrow }}{R}_{t+i}\right)}$ (8)${\displaystyle \overrightarrow{I}=\partial \left(W^{\underset{i}{\leftarrow }}{X}_t+W^{\underset{i}{\leftarrow }}{R}_{t+i}\right)}$ (9)${\displaystyle \overrightarrow{O}=\partial W^{\underset{o}{\leftarrow }}{X}_t+W^{\underset{o}{\leftarrow }}{R}_{t+i}}$ (10)${\displaystyle \overrightarrow{G}=tanh\left(W^{\underset{g}{\leftarrow }}{X}_t+W^{\underset{g}{\leftarrow }}{R}_{t+i}\right)}$ (11)${\displaystyle {\overrightarrow{C}}_t=\overrightarrow{F}\odot \overrightarrow{C_{t-1}}+\overrightarrow{I}\odot \overrightarrow{G}}$ (12)${\displaystyle {\overrightarrow{R}}_t=\overrightarrow{O}\odot tanh\left(\overrightarrow{C_t}\right).}$

At each time step of bidirectional LSTM, two outputs will be generated, one by forwarding the LSTM layer (R_t) and one by the backward LSTM layer ($\overrightarrow{R_t}$). These two outputs will be merged to get the output y_t. Final output Y will be expressed as the combination of the outputs generated by individual LSTM layers. (13)${\displaystyle Y_t=R_t+{\overrightarrow{R}}_t}$ (14)${\displaystyle Y=y_{t-1}+y_{\mathrm{t}}+y_{\mathrm{t+1}}}$

Figure S9 shows the working of three-time step bidirectional LSTM. Both forward and backward layers work independently in opposite directions to preserve contextual information. The output will then be processed, based on both past and future values.

Construction of stacked LSTM model

The Stacked LSTM has four layers: an input layer, a first LSTM layer, a second LSTM layer, and an output layer with 32, 16, 16, and 1 neuron respectively. The input vector to the model is X that consists of features (u_t..u_t−1, v_t..v_t−1, ws_t..ws_t−1, wd_t..wd_t−1, p_t−n..p_t−1) of feature set S(t) with past n observations whereas feature(p_t), the wind power, is our output vector Y. The first LSTM layer uses the Rectified Linear Unit (Relu) as an activation function and returns the processed sequence to the second LSTM layer. Finally, the predicted sequence is passed to the densely connected output layer. The weight initializer parameter of the model is set as ‘RandomUniform’ which initiates the network weights with random and uniformly distributed values in the range of 0 to 0.9. Root mean square propagation (RMSprop) is used as a model optimizer with a learning rate of 0.001. The model is trained with epochs = 5 and a batch size of 32. Figure 1 describes the architecture and flow of the stacked LSTM model.

Construction of bidirectional LSTM model

The bidirectional Stacked LSTM used in this study has four layers: the input layer, the first bidirectional LSTM layer, the second bidirectional LSTM layer, and an output layer with 64, 32, and 1 neuron respectively. The remaining tuning parameters like weight initializer, activation function, optimizer algorithm, learning rate, batch size, and the number of epochs are the same as the stacked LSTM model. The architecture of the proposed bidirectional LSTM can be seen in Fig. 2. In this model, simple LSTM layers are replaced with bidirectional LSTM layers.

Forecast accuracy

To perform more realistic forecasts, a step time-series cross-validation technique as indicated in the research (Zhang et al., 2022) is used. Preserving the time dependency aspect of the time series data, on each iteration, this method uses the previously observed values as a training set and the next unseen observation as a test set. To frame data for one-step cross-validation, a window of previous n observations was included as a feature at the data preprocessing stage. So besides splitting the dataset into training and test data, a one-step time-series cross-validation is also performed to enhance the forecast accuracy.

Performance parameters

Metrics used to evaluate the performance of the developed LSTM models are mean absolute error (MAE), root mean square error (RMSE), and standard deviation error (SDE) defined as (15)${\displaystyle MAE=\left|\frac{\sum \left(Xactual.i-Xmodel.i\right)}{n}\right|}$

(16)${\displaystyle RMSE=\sqrt{\frac{\sum {\left(Xactual.i-Xmodel.i\right)}^2}{n}}}$

(17)${\displaystyle \left(SDE=\sqrt{\frac{1}{M}}\sum _{i=1}^M{\left(Error-MeanError\right)}^2\right)}$

where X_actual is a vector consisting of actual values of wind power and X_{model, i} is the model predicted value at time/place i. The objective of the developed model is to minimize the values of these parameters to attain higher performance. Whereas Wp_actual is the desired value of power while Wp_predicted is the predicted power value provided by the proposed technique. Error = Wp_actual −Wp_predicted, while the average value of error is called mean error.

Stacked LSTM model

The model was implemented using 80% of the data as training and the remaining 20% for testing purposes. Table 1 shows the parameter settings applied to train the model. From the performance perspective, it has been observed that the proposed techniques achieved optimum results with only five epochs, whereas existing techniques used 150 to 500 epochs. Training and testing results obtained from the model are shown in Table 2. During the training phase, optimum results were observed when the past five observations were included in the feature set. So, the value of n i.e., no of past values to be included in the feature set is tuned as five.

Figures 3 through 7 shows the results of actual vs predicted hourly wind power for all the wind farms. The results show that there is a close match between the actual and predicted wind power. Both actual and predicted wind power curves almost follow the same pattern. It has been observed through results that LSTM is a robust technique for time sequence problems. Results also support the fact that deep neural networks work best when the dataset is large as in our case dataset for Wind Farm 3 is comparatively large, therefore, significant improvement in terms of error reduction is observed.

Bidirectonal LSTM model

As for Stacked LSTM, the dataset for the bidirectional LSTM model is also divided into a proportion of 80% and 20% for training and testing respectively. The same parameter settings of stacked LSTM are used for bidirectional LSTM. Table 3 shows both training and testing results when bidirectional LSTM is applied to the dataset.

Figures 8 through 12 show the results of actual vs predicted wind power on each wind farm dataset when bidirectional LSTM is used. Results demonstrate that bidirectional LSTM performs slightly better than stacked LSTM, especially in the case of Wind Farm 3, the performance difference increases concerning the proportion of data available for training. So, results can be improved when applied to large datasets.

Results comparison with existing techniques

In this section, the results of the proposed models are compared with the existing techniques of GPeANN (Zameer et al., 2017) and DNN-MRT (Qureshi et al., 2017). Table 4 shows the results in terms of RMSE, MAE, and SDE of existing and proposed models on training and test datasets. Figures 13 to 15 shows a comparison of the results of different error metrics measured against each wind farm. The comparison clearly shows that the Stacked LSTM and bidirectional LSTM outperformed the other two methods in terms of RMSE and MAE. However, in the case of SDE, the proposed models only outperformed Windfarm 3. The reason could be that in Wind Farm 3, the variation of wind power is generally lower than in other datasets. Furthermore, the author calculates SDE as the standard deviation of predicted wind power values, which is highly influenced by the variation of true wind power.

It is also noteworthy that the proposed model effectively generalizes patterns from the training set to the test set. This indicates that the model has successfully learned the underlying patterns and can apply them to new, unseen data. The utilization of regularization techniques during training of the proposed models prevents overfitting, thereby enhancing the model’s ability to generalize to unseen data. Furthermore, the LSTM’s inherent architecture and ability to capture long-term dependencies contribute to better performance on the test set. The model has learned intricate temporal relationships within the data that aid in making more accurate predictions. Moreover, the test set may have characteristics or patterns that closely align with the training set, allowing the proposed LSTM model to perform well on unseen data that share similarities with the training data.