REDf: a deep learning model for short-term load forecasting to facilitate renewable integration and attaining the SDGs 7, 9, and 13

Background

The imperative transition towards a sustainable future necessitates integrating renewable energy sources into power grids, reshaping global energy systems (Al-Shetwi, 2022; Jenkins, Long & Wu, 2015). Aligned with the United Nations’ (UN) ambitious goal of universal access to affordable, reliable, and modern energy by 2030 (United Nations, 2023; He et al., 2022), this integration emerges as pivotal for sustainable development. The UN Sustainable Development Goals (SDGs) set the framework, with SDG 7 (Affordable and Clean Energy), SDG 9 (Industry, Innovation and Infrastructure), and SDG 13 (Climate Action) standing as critical benchmarks for an equitable and environmentally sustainable energy landscape (Busch & Wydra, 2023).

Deep learning models play a crucial role in achieving these SDGs, particularly in predicting energy demand with precision. This accuracy enables optimal management of renewable resources, diminishes reliance on fossil fuels, reduces greenhouse gas emissions, and fortifies resilient and efficient infrastructure. Achieving high shares of renewable energy, as emphasized by SDG 7, diversifies the energy mix, decreases costs, and enhances accessibility.

Extending the focus to SDG 9, the study underscores the importance of fostering sustainable infrastructure and promoting innovation in the energy sector. Additionally, SDG 13 becomes a compelling imperative as the transition to renewable energy significantly contributes to mitigating the impacts of climate change.

Despite the promise of renewable sources, challenges persist, notably the unpredictability of energy generation (Ali et al., 2021; Maier & Gemenetzi, 2014). Addressing this challenge requires accurate predictions of energy demand, aligning with SDG 7. Developing effective predictive models becomes paramount in realizing these objectives.

This study delves into the intricate landscape of facilitating greater integration of renewable energy by predicting aggregated energy demand, aligning with multiple SDGs crucial for constructing an equitable and sustainable energy future (Aydos et al., 2022).

Harnessing clean energy sources such as hydroelectricity, geothermal, biomass, solar, and wind power not only reduces energy prices but also assures a dependable and sustainable energy supply globally (Perera, Coccolo & Scartezzini, 2019; United Nations Department of Economic and Social Affairs, 2017). Addressing climate change through the grid integration of renewable energy becomes pivotal for reducing greenhouse gas emissions (Giri et al., 2022) and mitigating its adverse effects (Joseph & Inambao, 2020). The transition from fossil fuels to renewable energy stands as a tangible step in fulfilling climate commitments under the Paris Agreement, safeguarding the planet for future generations (Akaev & Davydova, 2020).

While the integration of renewable sources holds promise, challenges persist, notably the unpredictability of energy generation (Ali et al., 2021; Maier & Gemenetzi, 2014). Tackling this challenge necessitates predicting energy demand using deep learning for a smart power grid, reflecting SDG 7. The development of accurate predictive models is a critical enabler in achieving this.

Literature review

The rapid advancement of machine learning techniques has significantly transformed the energy sector, offering innovative solutions to complex challenges such as energy consumption prediction, load forecasting, and efficient energy management. As the demand for sustainable and efficient energy systems grows, researchers have increasingly turned to advanced computational models, including deep learning and hybrid approaches, to enhance the accuracy and reliability of energy-related predictions. This literature review explores recent developments in the application of machine learning techniques to energy systems, categorizing the reviewed studies based on their methodologies, applications, and contributions. From hybrid deep learning models for household energy consumption to long short-term memory (LSTM)-based frameworks for smart grid applications, the reviewed works highlight the transformative potential of machine learning in addressing critical energy challenges. The following subsections provide a detailed analysis of these studies, organized by their thematic focus and methodological innovations.

Problem statement

Machine learning techniques have gained prominence in the energy sector, showcasing innovations in predicting energy demand. Noteworthy models, such as a hybrid approach combining LSTM networks with stationary wavelet transform, address irregular behaviors but leave gaps in providing a standardized evaluation framework (Yan et al., 2019). Similarly, the effectiveness of ANNs in developing countries lacks comprehensive evaluation metrics, leaving uncertainties in their general applicability for load demand forecasting (Arnob et al., 2023; Giri et al., 2023).

Novel frameworks integrating ANN-based forecast engines and controllers exhibit potential for cost-effective energy management, but challenges persist in refining forecasting models and understanding feature relationships within smart grids (Hafeez et al., 2020). The comparative analysis by Amalou, Mouhni & Abdali (2022) reveals the superiority of GRU over RNN and LSTM, yet achieving accuracy in load forecasts remains a concern. The proposed hybrid deep learning models in the Saudi smart grid system show promise, but the need for further refinement to enhance forecasting accuracy is evident (Alrasheedi & Almalaq, 2022).

Studies predicting household energy consumption using LSTM-RNN architectures indicate higher precision but lack comprehensive reporting on metrics such as MAE and $R^2$ values, posing challenges in evaluating overall model performance (Shachee, Latha & Hegde Veena, 2022). Taleb et al.’s (2022) flexible energy demand prediction model shows effectiveness but raises questions about its adaptability across various contexts.

Residential load forecasting studies highlight the superior performance of LSTM models compared to traditional methods, yet discrepancies in reported metrics raise concerns about the robustness of these findings (Mubashar et al., 2022). Additionally, the multivariate prediction approach using CNN-LSTM models for energy time series lacks standardized benchmarks, hindering a comprehensive understanding of its effectiveness (Rosato et al., 2019).

Despite advancements, the accuracy and reliability of energy demand predictions remain inconsistent across various models, revealing research gaps in standardization of evaluation metrics and the need for further model refinement. The absence of a unified framework hampers comparability and generalizability of results, necessitating comprehensive research to address these gaps and enhance the overall efficiency and sustainability of energy systems.

Key contributions

In alignment with recent studies and the defined objectives, this research introduces a novel approach utilizing deep learning, a sub-field of machine learning tailored for analyzing sequential data, particularly time series data (Ismail Fawaz et al., 2019; Hossain et al., 2022). The proposed methodology leverages a LSTM network, a type of RNN explicitly designed to model sequential data with temporal dependencies (Hochreiter & Schmidhuber, 1997). Training the model involves utilizing historical energy demand data, and its performance is rigorously evaluated using metrics such as MAE, RMSE, and $R^2$.

In addition to its prowess in accurately predicting energy demand, our proposed method showcases robust generalization capabilities for previously unobserved data. The ability to forecast energy demand accurately can revolutionize power grid management, fostering more efficient distribution and utilization of renewable energy sources while diminishing reliance on nonrenewable alternatives. Moreover, the proposed method has the capacity to enhance overall power infrastructure efficiency, cut costs, and facilitate the seamless integration of renewable energy sources.

The key contributions of this work extend beyond technical advancements to encompass a positive impact on SDGs, particularly SDG 7 (Affordable and Clean Energy), SDG 9 (Industry, Innovation, and Infrastructure), and SDG 13 (Climate Action). These contributions align with the United Nations’ broader agenda for sustainable development. The specific contributions are as follows:

• Introduction of LSTM-based deep learning model: The proposed deep learning model addresses the pivotal SDG 7, ensuring affordable and clean energy for all. By enhancing the precision of energy demand forecasts, the model promotes efficient utilization of renewable energy sources, thus contributing to a sustainable and accessible energy future.

• Minimization of MAE: The emphasis on minimizing MAE in the energy demand prediction model directly supports SDG 9 by fostering innovation and infrastructure development. Accurate energy forecasts enable infrastructure optimization, paving the way for more sustainable and resilient energy systems.

• Comprehensive model evaluation: The rigorous evaluation of the proposed approach using various metrics, including MAE, RMSE, $R^2$, and MAPE aligns with SDG 13. By improving the accuracy of energy demand predictions, the model contributes to climate action efforts, reducing reliance on nonrenewable energy sources and mitigating environmental impacts.

In essence, this article’s primary contributions lie in technical advancements and in addressing global challenges outlined in the SDGs. The research aligns with the United Nations’ vision for a sustainable future, emphasizing the role of accurate energy demand forecasting in achieving key sustainable development objectives.

The remaining sections of this article are structured as follows. The study’s materials and methods, including data acquisition and pre-processing, model development, evaluation, and deployment, are described in detail in the following section. The experimental findings are then presented in the results and discussion section, along with a comparison to other existing methods. Finally, the study concludes with recommendations for future research.

Problem formulation

The problem formulation section provides the foundation and context for the proposed solution in the study by clearly defining the objective, scope, and challenges of the energy demand forecasting problem. The problem of predicting energy demand for a smart power grid can be formulated mathematically as follows:

Given a time series of historical energy demand data, represented by a sequence of vectors $<D=d_1,d_2,\dots ,d_n>$, where $d_i$ is a vector of energy demand values at time $i$, the goal is to predict the energy demand at a future time, represented by a vector $d_{n+h}$, where $h$ is the number of time steps ahead for which the prediction is made.

This problem can be represented as a function $f(D)=d_{n+h}$, where the function $f$ maps the historical energy demand data to the predicted energy demand.

The objective is to find the optimal function $f$ that minimizes the prediction error, which can be defined as the MAE between the predicted and actual energy demand values.

Data collection

Data collection is essential in predicting energy demand in a smart power grid. This section briefly describes the steps and techniques used for data collection in the proposed study.

The initial phase of data collection involves the identification of pertinent data sources, necessitating the recognition of data types essential for enabling precise predictive modeling, including variables such as energy consumption, meteorological conditions, and economic indicators. Data acquisition encompasses a diverse array of potential sources, ranging from utility providers and government agencies to publicly accessible datasets. In the context of this research, the study leverages hourly energy demand data procured from American Electric Power (AEP), a prominent electric utility enterprise within the United States, renowned for its extensive service coverage encompassing over 5 million customers across 11 states (AEP, 2023). The dataset in question encompasses a robust repository comprising 121,273 data entries, documenting hourly energy consumption spanning from December 2004 to January 2018. Furthermore, to assess the model’s performance comprehensively, the supplementary datasets COMED (https://www.comed.com/Pages/default.aspx), DAYTON (https://www.aes-ohio.com/about-aes-ohio), and PJME (https://pjm.com/) have been enlisted for evaluation. These datasets collectively serve as established benchmarks for scrutinizing the efficacy of energy demand forecasting models and are readily accessible through the following GitHub repository: https://github.com/panambY/Hourly_Energy_Consumption (Pereira, 2023).

COMED hourly energy consumption data refers to the hourly electricity consumption data for the Commonwealth Edison (COMED) service area, which covers the northern part of Illinois in the United States. The data provides information on the hourly electricity demand for residential, commercial, and industrial customers in the COMED service area. This dataset contains historical energy consumption data in an hourly fashion from December 2011 to January 2018, with a total of 66,497 data points.

DAYTON hourly energy consumption data refers to the hourly electricity consumption data for the Dayton Power and Light (DP&L) service area, which covers the city of Dayton, Ohio, and surrounding areas in the United States. The data provides information on the hourly electricity demand for residential, commercial, and industrial customers in the DP&L service area. This dataset contains historical energy consumption data in an hourly fashion from December 2004 to January 2018, with a total of 121,275 data points.

PJME hourly energy consumption data refers to the hourly electricity consumption data for the Pennsylvania-New Jersey-Maryland Interconnection (PJM) regional transmission organization in the United States. The data provides information on the hourly electricity demand for a large portion of the eastern United States, covering 13 states and the District of Columbia, and includes variables such as date, time, temperature, and electricity demand. This dataset contains historical energy consumption data in an hourly fashion from December 2002 to January 2018, with a total of 145,366 data points. Table 2 shows an overview of the datasets utilized to benchmark the proposed model.

After data has been collected, it has to be pre-processed to ensure that it is in a format compatible with the model. This includes removing any inconsistencies or errors from the data and transforming the data into a format that the model can use. The data pre-processing procedures described in the following section describe normalisation, feature scaling, and outlier removal.

Data pre-processing

Data pre-processing is essential in predicting energy demand in a smart power grid. It ensures that the data used to train and evaluate the model is clean, consistent, and in a suitable format for the model. This section will describe the steps and techniques used for data pre-processing in the proposed study.

The initial phase in data pre-processing is data cleansing. This includes eliminating any data inconsistencies, errors, or missing values. Common data cleansing techniques include removing duplicates, replacing absent values with imputed values, and converting data to a standard format. In our case, for all the datasets employed in this study do not contain any missing values and any outlier. To check and detect the missing values and outliers “isnull” and “IsolationForest” techniques are employed from “Scikit-learn” library.

The following phase transforms the data into a format the model can utilize. This includes the normalization, scaling, and encoding of categorical variables. Normalization is adjusting the data so that the mean is 0 and the standard deviations are 1. Scaling the data can prevent the magnitude of the data from affecting the model. In this study, the “sklearn.preprocessing.MinMaxScaler” is employed to normalize the data. Figure 3 illustrates an example using the PJME dataset, comparing the data in its original form (without normalization) and its transformed state (with normalization). The entire process of data pre-processing is presented in Algorithm 1.

The algorithm serves as a systematic framework for preparing raw data, $D_{raw}$, for subsequent analysis. It takes the raw data as input and generates pre-processed data, $D_{pre}$, as output. The procedure unfolds as follows: First, it loads the raw data into the program using the function $load\mathrm{\_}data()$. Next, it checks for missing values within the dataset and employs appropriate handling procedures, yielding the pre-processed data, $D_{pre}$. Subsequently, the algorithm identifies and addresses outliers within $D_{pre}$ through the function $handle\mathrm{\_}outliers()$. Following this, the data is normalized using the $normalize()$ function to ensure consistency and comparability among different features. The algorithm then partitions the pre-processed data into separate training and testing sets, denoted as $(D_{train},D_{test})$, using the $split\mathrm{\_}data()$ function. Finally, it returns the pre-processed data, $D_{pre}$, which is conditioned for further analysis.

Data pre-processing is a critical step in predicting energy demand in a smart power grid. It includes cleaning the data and transforming the data. These steps help ensure that the data used to train and evaluate the model is clean, consistent, and in a suitable format for the model.

Model development

This section describes developing a forecasting model to predict energy demand in a smart power grid. The proposed method employs a deep learning-based model, particularly a LSTM network, a RNN designed to manage sequential data with temporal dependencies. LSTM is a form of RNN architecture frequently used in deep learning applications for sequence modelling, including natural language processing, speech recognition, and time series forecasting.

In time series forecasting, LSTM is useful for capturing difficult-to-model long-term dependencies in the data. Predicting hourly energy demand is one possible application of LSTM in time series forecasting. The model is trained on historical data to use LSTM to predict hourly energy demand to understand the patterns and relationships between the input features and the target variable, in this case, energy demand in megawatts. Based on the input features, the model can predict future time steps.

LSTM comprises a set of nonlinear transformations that operate on the input and hidden states of the network, as well as gating mechanisms that regulate the passage of information through the network. The equations for a single LSTM cell are shown in Eq. (1) (Sherstinsky, 2020).

(1) $$\begin{array}{rl}& i_t=\sigma (W_{xi}{x}_t+W_{hi}{h}_{t-1}+b_i)\\ & f_t=\sigma (W_{xf}{x}_t+W_{hf}{h}_{t-1}+b_f)\\ & \tilde{C}t=\tanh (Wxc{x}_t+W_{hc}{h}_{t-1}+b_c)\\ & C_t=f_t\odot C_{t-1}+i_t\odot \tilde{C}t\\ & o_t=\sigma (Wxo{x}_t+W_{ho}{h}_{t-1}+b_o)\\ & h_t=o_t\odot \tanh (C_t)\end{array}$$

Here, $x_t$ is the input at time step $t$, $h_{t-1}$ is the hidden state of the previous time step, W and $b$ are the weights and biases of the network, and $\sigma$ and $\tanh$ are the sigmoid and hyperbolic tangent activation functions, respectively. The equations involve several gates, including an input gate $i_t$, a forget gate $f_t$, and an output gate $o_t$, which control the flow of information through the network. The cell state $C_t$ is updated based on the input and hidden states, and the hidden state $h_t$ is computed as a function of the cell state and the output gate.

The initial phase in model development is to collect and pre-process the data. As inputs to the model, we utilized historical energy demand data. The data was collected from the PJM data interface for the AEP zone and pre-processed to ensure it was in the correct format for the model. The pre-processing stages included data cleansing, handling missing values, and data normalization.

Next, the network structure of the LSTM model was defined. Input, multiple LSTM, dropout, and output layers comprise the LSTM model. The number of neurons in the input layer corresponds to the number of input features, which in this instance are the historical energy demand data. The LSTM layers are the basis of the model and are responsible for learning the temporal dependencies in the data. Multiple LSTM layers were layered on top of one another to improve the model’s ability to learn complex data patterns. A grid search cross-validation technique was used to ascertain the number of LSTM layers and the number of neurons in each layer. Grid search cross-validation entails creating a grid of possible values for each hyperparameter and evaluating the model’s performance on each combination of hyperparameters using a cross-validation strategy. Then, the model’s performance is compared across all possible combinations, and the hyperparameters that yield the highest performance are chosen. This computationally intensive technique offers a systematic and trustworthy method for choosing optimal hyperparameters for deep learning models. Algorithm 2 presents the grid search method employed in this study. A single neuron in the output layer represents the anticipated energy demand.

The LSTM model was implemented using the Keras (Chollet & Keras Collaborators, 2015) library in Python (Van Rossum & Drake, 2009). The model was trained using an Adam optimizer and mean squared error (MSE) as the loss function. The model was trained for a fixed number of epochs, and the training process was stopped when the model’s performance on a validation set stopped improving. Algorithm 3 presents the steps utilized in the forecasting model’s development. The architecture of the proposed model is shown in Fig. 4.

Here, timesteps are the number of time steps in the input data, features are the number of features in the input data, x is the number of units in the LSTM layer, y is the number of units in the fully connected layer, z is the number of epochs for training, and w is the batch size for training. X_train and y_train represent the training data, and X_test represents the test data. In the proposed model, the unit used in LSTM is 200, and the number of units used in the fully connected layer is 1. The training epoch of the model is 10, and batch_size is 1,000.

The algorithm starts by initializing the model using the Sequential() function from the Keras library, and the LSTM, dropout, and fully connected layers are added using the add() function. The model is then compiled using the compile() function and trained using the fit() function. Finally, predictions are made on the test data using the predict() function.

The trained model was then evaluated using performance metrics such as MAE, RMSE, and $R^2$ on a test set. These metrics were used to evaluate the model’s ability to predict energy demand accurately. The model was then deployed and used to predict energy demand.

To ensure model generalizability across datasets and prevent overfitting, several key strategies were implemented. Dropout regularization was applied to all LSTM layers with a 10% dropout rate, reducing reliance on specific neurons and enhancing the model’s robustness. A five-fold cross-validation approach was employed to validate performance across different training subsets, ensuring consistency. Additionally, early stopping was implemented to halt training once validation loss plateaued, preventing unnecessary parameter tuning. To further enhance adaptability, the model was trained on four distinct datasets: AEP, COMED, DAYTON, and PJME allowing it to capture regional variations in energy demand. These techniques collectively ensure optimal model performance, making it well-suited for real-world deployment in smart grid applications.

Model evaluation metrics

This section describes the process and metrics of evaluating the performance of the proposed LSTM model for predicting energy demand in a smart power grid. The model was trained using historical energy demand data and evaluated using three performance metrics: MAE, $R^2$, RMSE, and MAPE.

The MAE is a measure of the difference between the predicted energy demand and the actual energy demand. It is calculated as the average absolute difference between the predicted and actual values. Mathematically, it is defined as Eq. (2):

(2) $$MAE=\frac{1}{n}\sum _{i=1}^n|y_i-\widehat{y_i}|$$where $n$ is the number of test samples, $y_i$ is the actual energy demand, and $\widehat{y_i}$ is the predicted energy demand. The smaller the MAE, the better the model’s performance.

In a linear regression model, the $R^2$ represents the proportion of the variance in the dependent variable explained by the independent variables. R² values range from 0 to 1, with greater values indicating a superior model fit to the data. A higher R² value indicates a better fit and a stronger relationship between the independent and dependent variables when used to evaluate the current model. Mathematically, it is defined as Eq. (3):

(3) $$R^2=1-\frac{{\sum }_{i=1}^n{(y_i-\widehat{y_i})}^2}{{\sum }_{i=1}^n{(y_i-\overline{y})}^2}$$where $y_i$ is the actual value of the dependent variable, $\widehat{y_i}$ is the predicted value of the dependent variable, and $\overline{y}$ is the mean of the dependent variable.

Another evaluation metric used in this study is RMSE. It measures the average magnitude of the error in the predictions of a model. RMSE calculates the difference between a dataset’s actual and predicted values and then takes the square root of the average of those differences. A lower RMSE value indicates that the model better fits the actual values and has higher accuracy in making predictions. Mathematically, it is defined as Eq. (4):

(4) $$\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E}=\sqrt{\frac{1}{n}\sum _{i=1}^n{(y_i-\widehat{y_i})}^2}$$

Here, $y_i$ is the actual value, $\widehat{y_i}$ is the predicted value, and $n$ is the number of observations.

The MAPE is a commonly used metric to measure the accuracy of a forecasting model. It expresses the prediction error as a percentage of the actual values, making it useful for comparing different models regardless of the scale of the data. A lower MAPE value indicates a better fit of the model to the actual data. Mathematically, it is defined as Eq. (5):

(5) $$MAPE=\frac{1}{n}\sum _{i=1}^n\left|\frac{y_i-{\hat{y}}_i}{y_i}\right|\times 100$$where $n$ is the number of test samples, $y_i$ represents the actual energy demand, and ${\hat{y}}_i$ denotes the predicted energy demand. MAPE provides an intuitive measure of error in percentage terms, which can be beneficial for evaluating model performance in real-world energy demand forecasting applications.

Model deployment

Deploying the proposed LSTM model for securely predicting energy demand in a smart power grid requires appropriate tools and techniques. This section describes the technique of deploying the model securely.

First, TensorFlow (Abadi et al., 2015) is used to serve the model. TensorFlow provides several security features that can be used to protect the model during deployment. For example, TensorFlow can encrypt model data and communicate between the model and other systems. TensorFlow also allows the ability to authenticate users and devices accessing the model. Then, Apache Kafka (Apache Software Foundation, 2021) is used to deploy the model securely. Apache Kafka is a message queuing system that handles high-throughput data streams. It is used to send real-time energy demand data to the model and also receives predictions from it. Apache Kafka provides built-in security features such as encryption and authentication, which protect the data during transmission. To secure the communication between the model and other systems, Secure Sockets Layer (SSL; Corporation Netscape Communications, 1996) is used in the proposed model. SSL ensures that all data transmitted between the systems is encrypted and that only authorized systems can access the model. Figure 5 shows the proposed system’s architecture of the model deployment phase. The figure shows that Kafka handles the application, user query, communication between the model server, and prediction output. Users place the prediction requests in the Apache Kafka application, and then the Apache Kafka application forwards the requests to the TensorFlow model server. Then the model server returns the predicted result and handles the response to the Apache Kafka application, which is served to the users. All the communication between the Apache Kafka application and the TensorFlow model server is encrypted with the SSL service. The prototype front end of the application is shown in Fig. 6. This figure shows the interface for taking input and giving prediction output from the TensorFlow model server.

Deploying the proposed LSTM model for securely predicting energy demand in a smart power grid requires using appropriate tools and techniques such as TensorFlow and Apache Kafka. TensorFlow provides built-in security features such as encryption and authentication while serving the model. Apache Kafka also provides built-in security features such as encryption and authentication for data transmission on the application side. Secure communication protocols, such as SSL, are also used to encrypt the data transmitted between systems.

Experimental setup

We implemented our proposed approach using Python version 3.6.5 as the primary programming language. The experiments were conducted on a computer with a Ryzen 7 processor, 24 GB RAM, and a GTX 1650 GPU. The computer was running Windows 10 as the operating system. We used Jupyter as our integrated development environment (IDE) to develop and test the model.

To implement and train our model, we utilized TensorFlow and Keras, two popular deep-learning frameworks widely used in the research community. TensorFlow provided us with the tools to develop our deep learning model, and Keras allowed us to easily create, compile, and train the model. The EarlyStopping method was utilized to prevent the REDf model from being over-trained and three-fold cross validation was employed on all the models compared in this study.

Apache Kafka, an open-source distributed event streaming platform, facilitates data streaming. This helped us handle high volumes of data and ensure efficient data processing. For model deployment purposes, we used ModelServe, a high-performance model serving solution that allowed us to quickly and easily deploy our trained model to production environments.

Experimental result analysis

The deep learning model, REDf was trained using the LSTM network architecture with 200 units. The model was trained using the Adam optimization algorithm with a learning rate 0.001. The training process took approximately 25 min on the experimental machine.

The performance of the model was evaluated using three metrics: MAE, RMSE, ( $R^2$), and MAPE. The MAE measures how close the predicted values are to the true values, the $R^2$ measures how well the model fits the data, the RMSE measures the average magnitude of the error in the predictions of a model and MAPE expresses the prediction error as a percentage of the actual values, making it useful for comparing different models regardless of the scale of the data. The experimental results are presented in Table 3.

Based on the results presented in Table 3, it is evident that the proposed REDf model achieved high accuracy in the predictions, with MAE ranging from 1.4% to 1.5% across all evaluated datasets, $R^2$ ranging from 97.9% to 98.5% for different datasets, RMSE of approximately 0.02 across all evaluation datasets, and an exceptionally low MAPE value between 0.0008 and 0.0012. The significantly low MAPE score highlights the model’s robustness in producing accurate predictions with minimal relative errors across different datasets.

In contrast, the state-of-the-art SVR model exhibited a relatively good fit for the data, as indicated by $R^2$ values ranging from 72.6% to 98.2%. However, its performance was inconsistent across datasets, as reflected by the significantly high MAE and RMSE values. Moreover, the MAPE scores for SVR are notably higher (ranging from 0.0058 to 0.0581), suggesting that while the model aligns well with the trend, it produces higher relative percentage errors in energy demand forecasting.

Facebook’s Prophet model demonstrated poor performance across all datasets in all evaluation metrics. It achieved a positive $R^2$ value only for the AEP dataset, while obtaining negative values for others, indicating an inadequate fit to the data. The model also exhibited excessively high MAE and RMSE values, confirming its poor predictive capability. The MAPE scores for the Prophet model range between 0.1331 and 0.1607, signifying significant deviations from the actual energy demand values, making it unreliable for short-term load forecasting.

A similar trend is observed for the RFR model, where the $R^2$ score varies between 4.7% and 17%, reflecting poor model performance in capturing energy demand patterns. The high MAE and RMSE values further suggest that RFR struggles to model time series dependencies effectively. The MAPE values for RFR are also notably high, ranging from 0.1278 to 0.1570, reinforcing the model’s difficulty in accurately predicting energy demand fluctuations.

The proposed REDf model consistently outperformed all other models across all metrics, demonstrating the lowest MAE, RMSE, and MAPE values, alongside the highest $R^2$ scores. The exceptionally low MAPE values (below 0.0013) validate the model’s ability to minimize relative percentage errors, making it highly suitable for real-world energy demand forecasting applications. These results highlight the superior accuracy, stability, and generalizability of the REDf model in facilitating renewable energy integration within smart power grids. This discussion is also supported by the visual representation of the comparison of the results shown in Fig. 7. Figure 7 presents an integrated performance comparison of various forecasting models, including REDf, SVR, Prophet, and RFR, across multiple datasets (AEP, COMED, DAYTON, and PJME). The visualization combines a bar chart representing the $R^2$ scores and scatter plots for MAE, RMSE, and MAPE, effectively illustrating the differences in model accuracy and robustness.

The $R^2$ scores, displayed as blue bars, highlight the superior performance of the REDf model, which consistently achieves values above 0.97 across all datasets. This confirms the model’s strong predictive capability and ability to accurately capture energy demand patterns. In contrast, SVR exhibits moderate performance, with its lowest $R^2$ score dropping to 0.726 for the PJME dataset. The Prophet and RFR models demonstrate poor performance, with multiple instances of negative $R^2$ scores, indicating their failure to fit the data effectively.

The MAE values, represented by a gold scatter plot, further emphasize the accuracy of the REDf model, which maintains the lowest MAE values across all datasets. This signifies minimal deviation from actual demand values, making it a reliable forecasting tool. However, SVR and RFR models exhibit significantly higher MAE values, particularly for the PJME and AEP datasets, indicating substantial errors in their predictions. The Prophet model performs the worst in this metric, with MAE values exceeding 2,000, highlighting its unsuitability for short-term energy forecasting applications.

The RMSE values, visualized with a green dashed scatter plot, reinforce the reliability of the REDf model, as it achieves the lowest RMSE values across all datasets. High RMSE values observed for the SVR and RFR models indicate large deviations in their predictions, reducing their forecasting reliability. The Prophet model once again performs the worst, exhibiting excessively high RMSE values, further confirming its poor predictive accuracy.

Finally, the MAPE values, represented by a red dashed scatter plot, validate the high precision of the REDf model, which maintains the lowest MAPE values across all datasets. This confirms its ability to minimize relative percentage errors in energy demand forecasting. Conversely, the SVR and RFR models display significantly higher MAPE scores, reducing their reliability in precision forecasting. The Prophet model exhibits the highest MAPE values, exceeding 0.15, making it unsuitable for accurate short-term forecasting applications.

These comparative results highlight the superior performance of the REDf model across all evaluation metrics, demonstrating its stability, accuracy, and robustness. The significantly lower error values and higher $R^2$ scores confirm its effectiveness in real-world applications, enabling accurate short-term energy demand forecasting, improved renewable energy integration, and efficient smart grid management. The experimental results can be further corroborated with visual representations of the models’ actual vs predicted data plots. Figure 8 shows the actual vs predicted data plots for AEP dataset. Similarly the actual vs predicted data plots for COMED, DAYTON, and PJME datasets are presented in Figs. S1, S2, and S3 respectively. The high-resolution versions of these figures are also available at the following link, https://ping543f.github.io/ren_energy/.

The plots depict hourly energy demand over time, with the x-axis representing the time frame and the y-axis representing energy demand. The green line depicts the actual energy demand, while the red line shows the predicted energy demand by the proposed REDf model. For the SVR and RFR models, the red dots indicate actual data points, and the blue dots represent predicted data points. For the Facebook Prophet model, the black dots represent actual data, and the blue line shows the predicted data by the model. Analysis of the plots indicates that the difference between the actual and predicted energy demand is minimal for the proposed REDf model across all datasets. Conversely, the difference between the actual and predicted energy demand is significantly high for the SVR, RFR, and Prophet models in almost all datasets, except for the SVR model in the Dayton dataset. The actual vs predicted plots for the proposed REDf model demonstrate that the predicted energy demand values are very close to the actual values, indicating a good fit for the data and accurate forecasting of energy demand data. Furthermore, the proposed model showed no sign of overfitting and underfitting at the end of the training epochs. In this study, the performance of the developed model was carefully evaluated to assess the presence of overfitting and underfitting phenomena. Figure 9 shows the loss curves for training and validation phases of the REDf model for different datasets employed in this study for all three folds. Overfitting occurs when a model performs exceptionally well on the training data but fails to generalize to new, unseen data. On the other hand, underfitting happens when a model lacks the complexity to capture the underlying patterns in the data and performs poorly on both the training and test sets.

The model demonstrated robust performance in our experiments without showing signs of overfitting or underfitting. This was evident from the consistent and comparable performance metrics achieved on the training and test datasets. The absence of overfitting can be attributed to including dropout layers in the model, which regularizes the training process by randomly dropping a fraction of the units, preventing the model from relying too heavily on specific features. Additionally, by selecting an appropriate model architecture and hyperparameters through a systematic grid search technique, we ensured that the model had the necessary complexity to capture the underlying patterns in the data without excessive complexity that could lead to overfitting.

The absence of underfitting indicates that the model could adequately capture the relevant information from the training data, allowing it to generalize well to unseen data. This suggests that the chosen model architecture and hyperparameters were suitable for the given task and dataset, striking a balance between simplicity and complexity. The similar trend has also been observed in all the folds of training the model. An example for all three fold for the AEP dataset is presented in Fig. 10. Figure 10 presents the cross-validation results for the proposed REDf model applied to the AEP dataset. The figure consists of three rows, each representing a different fold in the cross-validation process, and each row contains three subplots: Training vs Validation Loss (left), Actual vs Predicted values (middle), and Error Distribution (right).

The training vs validation loss plots illustrate the convergence behavior of the REDf model across different folds. The loss function, measured in MSE, steadily decreases for both the training and validation sets over successive epochs. The absence of sudden spikes or divergence between the training and validation loss curves indicates that the model effectively generalizes without signs of overfitting or underfitting. The smooth convergence further reinforces the model’s stability in learning the temporal dependencies of energy demand.

The actual vs predicted plots compare the REDf model’s forecasts against actual energy demand for the first 100 test samples in each fold. The close alignment between the actual (blue line) and predicted (orange line) values across all three folds demonstrates the model’s capability to capture the fluctuations and patterns inherent in short-term energy demand forecasting. This strong correlation between actual and predicted values suggests that the REDf model is well-suited for accurately forecasting short-term electricity consumption in smart grid environments.

The error distribution histograms provide insight into the model’s predictive accuracy across different test sets. The narrow and symmetric distribution of errors, centered around zero, confirms that the model exhibits low variance and maintains consistent performance across all cross-validation folds. The limited spread of errors further highlights the model’s reliability in forecasting energy demand with minimal deviations from actual values. The similar trends have been obsreved for COMED, DAYTON and PJME datasets for the REDf model and presented in Figs. S4–S6.

These results validate the effectiveness of the REDf model in accurately predicting short-term energy demand while maintaining robustness across multiple test sets. The combination of low validation loss, strong agreement between actual and predicted values, and tightly distributed errors demonstrates the model’s superiority in short-term load forecasting. This further reinforces its applicability in real-world scenarios, enabling improved renewable energy integration and efficient grid management.

Based on the experimental results, the proposed REDf model is highly accurate in forecasting energy demand. The model’s predicted values align with the original energy demand values. This indicates that the model is a good fit for the data and can be relied upon for accurate predictions of energy demand. Demand forecasting accuracy plays a significant role in integrating renewable energy sources and achieving the SDGs in the smart grid. Accurate demand prediction allows grid operators to anticipate load variations, allocate resources effectively for frequency regulation services, and optimize resource allocation by aligning them with expected load patterns. This improves the scheduling and dispatch of frequency regulation resources, minimizes the need for corrective actions, and reduces the risk of frequency deviations. Moreover, accurate load forecasting enhances market participation, enabling market participants to make informed bidding decisions based on expected load fluctuations, resulting in more competitive and accurate bids. Ultimately, load forecasting accuracy contributes to overall grid stability and reliability by allowing proactive measures to balance supply and demand, thereby reducing the likelihood of frequency deviations and ensuring the reliable operation of power systems.

Comparative analysis

It is important to note that the results are based on a specific dataset and architecture, and the performance of the proposed approach may vary depending on the type of data and the specific implementation details. However, the results demonstrate the potential of the proposed approach for predicting short-term energy demand in a smart power grid. The results achieved from the experiment can be compared to other recent works in predicting energy demand in smart power grids.

One of the most closely related studies is by Amalou, Mouhni & Abdali (2022), who used a similar approach to deep learning with LSTM networks to predict energy demand. In omparison to this study, the proposed REDf model achieved better performance in terms of MAE, $R^2$, and RMSE. Specifically, the proposed model achieved a mean absolute error of 1.4%, which is significantly lower than the MAE of 0.021 reported in this study. Additionally, the proposed model achieved a higher $R^2$ value, indicating a better fit to the data, and a lower RMSE value, indicating better accuracy in predicting energy demand.

Similarly, compared to the study by Alrasheedi & Almalaq (2022), the proposed REDf model achieved better performance in terms of RMSE and $R^2$. Alrasheedi & Almalaq (2022) achieved an RMSE of 0.8168 and a $R^2$ of 0.973 on their test set, while the proposed REDf model achieved a lower RMSE and a higher $R^2$ value. This indicates that the proposed model is more accurate in predicting energy demand and better fits the data.

A recent study by Shachee, Latha & Hegde Veena (2022) proposed a hybrid deep learning-based model of LSTM and RNN that utilizes historical load data to predict energy demand. They achieved an RMSE of 0.6 on their test set. Our model achieved better performance in this evaluation metric compared to this study.

Taleb et al. (2022) presented a hybrid model that combines standard neural networks with an automatic weight update process, achieving a MAE of 372.08 in energy demand prediction. Mubashar et al. (2022) proposed a method for load forecasting using LSTM models and compared its performance with two commonly used techniques, ARIMA and exponential Smoothing. Their proposed method outperformed the other two, achieving an MAE of 2.44736176. Rosato et al. (2019) presented a novel deep learning approach using convolutional neural network and long short-term memory models, achieving the lowest RMSE of 2.252 for the baseline 1-day forecast. Nguyen, Duong & Le (2020) proposed an electricity demand forecasting method based on the LSTM deep learning model, achieving an RMSE of 9.63. Pramono et al. (2019) proposed a method for short-term load forecasting using a wavenet-based model that employs dilated causal residual CNN and LSTM layers, achieving RMSE and MAE equal to 203.23 and 142.23 for ENTSO-E dataset 1 and 292.07 and 196.95 for ENTSO-E dataset 2. The proposed methods have demonstrated their potential for supporting energy management and demand response programs in hybrid energy systems.

The performance of the proposed model was compared with other recent studies in predicting energy demand in smart power grids. The proposed model outperforms the other models regarding MAE, $R^2$, and RMSE in all the evaluated datasets. The LSTM-RNN (Shachee, Latha & Hegde Veena, 2022) model did not report the MAE and $R^2$ values mentioning only RMSE. The CNN-GRU (Alrasheedi & Almalaq, 2022) model achieved a high $R^2$ score but did not report the MAE and RMSE values. On the other hand, the LSTM and GRU models reported MAE and $R^2$ values, but their performance was inferior to the proposed model. Apart from these, most studies did not report $R^2$ values.

Various deep learning-based methods have been proposed to accurately forecast energy demand, including standard neural networks with an automatic weight update process, LSTM models, CNN and LSTM models, and wavenet-based models. These models have been shown to outperform traditional methods such as ARIMA and exponential smoothing, achieving lower MAE and RMSE values. However, our proposed model significantly outperforms all the existing models, achieving an exceptionally low MAE of 0.015 and RMSE of 0.02, demonstrating its potential to revolutionize the energy sector by providing more accurate energy demand forecasting.

The proposed model for predicting the demand for energy in a smart grid can also assist in realizing substantial environmental advantages while advancing several SDGs. First and foremost, this paradigm can help fulfill SDG 7: Access to Affordable and Clean Energy. Utility companies can better manage their renewable energy resources and lessen their dependency on fossil fuels by precisely anticipating the demand for renewable energy. This will result in a more ecologically friendly and sustainable energy system. This may facilitate the transition to a low-carbon economy, improve air quality, and cut greenhouse gas emissions. As a result, people’s access to and affordability of energy, particularly in low-income areas, may improve.

In addition, this demand model for renewable energy might help with SDG 9: Industry, Innovation, and Infrastructure. Utility companies and other stakeholders can build sustainable infrastructure and encourage innovation in the energy industry by offering accurate and trustworthy estimates of the demand for renewable energy. New technologies and business models may be created as a result, which might hasten the uptake of renewable energy sources and encourage their use in a sustainable and efficient manner.

Thirdly, this model can help achieve SDG 13: Climate Action. Predictive models for renewable energy demand can aid in the fight against climate change and its effects by encouraging renewable energy sources and lowering dependency on fossil fuels. To mitigate the effects of climate change, such as more frequent and severe weather events; this can involve lowering greenhouse gas emissions, enhancing air quality, and enhancing air quality.

Achieving various sustainable development objectives relating to access to affordable and clean energy, innovation and infrastructure, and climate action can be facilitated by developing precise forecast models for renewable energy demand in smart grids. Our proposed model, which uses deep learning, LSTM networks, and data pre-processing approaches, performed better than recent research in this sector. This shows that our method can be an efficient way to estimate energy demand in smart power grids and might have significant economic and environmental benefits by encouraging the adoption of renewable energy sources and lowering dependency on fossil fuels. As a result, our work contributes significantly to the ongoing efforts to create sustainable energy systems that can support a more equitable future and less harmful to the environment.

Alignment with sustainable development goals

The REDf model aims to contribute to the SDGs, particularly SDG 7 (Affordable and Clean Energy), SDG 9 (Industry, Innovation, and Infrastructure), and SDG 13 (Climate Action). This section discusses the model’s potential impact while recognizing the need for further research to quantify certain aspects.

Environmental benefits

Using machine learning methods like deep learning to predict energy consumption has the potential to improve smart power grids significantly. Power grid administrators may maximize the distribution and use of renewable energy sources, minimizing reliance on non-renewable sources and encouraging the integration of clean energy by accurately projecting energy demand. As a result, customers and the environment could benefit from decreased prices, increased efficiency, and better control over power networks. The generalization and prediction abilities of the suggested method have been shown to be strong. This strategy can support the international effort to combat climate change and achieve sustainable development by incorporating the principles of SDGs 7 (affordable and clean energy), 9 (industry, innovation, and infrastructure), and 13 (climate action).

Challenges

Although the proposed method for predicting energy demand using deep learning has the potential to improve the integration of renewable energy sources and optimize the efficiency of power infrastructures, it is not without obstacles. The data availability and quality required for training deep learning models is a significant challenge. It is possible that historical energy demand data are unavailable or insufficient, which can compromise the accuracy of the model’s predictions. Another obstacle is the high computational requirements and lengthy nature of deep learning model training. This can be problematic when working with enormous datasets or multiple variables. In addition, the interpretability of the model can be problematic, as the inner workings of deep learning models can be challenging to comprehend, limiting their transparency and accountability. It is crucial to successfully address these obstacles to implement the proposed approach in smart power infrastructures.

The results of this work can be summed up as follows: (1) The suggested model outperformed other recent efforts in anticipating energy consumption in smart power grids in terms of performance. (2) Deep learning with LSTM networks and data pre-processing methods successfully anticipated energy consumption in smart power grids. (3) The suggested model has the ability to help achieve environmental benefits and sustainable development goals, as evidenced by its accuracy in anticipating energy consumption in smart power networks. (4) The suggested model can aid in the more effective use of renewable energy sources by better forecasting energy demand and lowering the requirement for environmentally hazardous non-renewable energy sources. (5) A more sustainable use of natural resources can be achieved by reducing energy waste and using energy more efficiently.

Validity

The validity of the proposed LSTM-based deep learning model for forecasting short-term energy demand is established through several key aspects:

• Data sources: The model was trained and validated on four distinct datasets from reputable energy distribution companies, including AEP, COMED, DAYTON, and PJME. These datasets provide a comprehensive and diverse range of historical energy demand data, ensuring that the model is tested on various consumption patterns.

• Model evaluation: The model’s performance was rigorously evaluated using widely accepted metrics such as MAE, RMSE, and $R^2$. These metrics provide a robust assessment of the model’s predictive accuracy and its ability to generalize to unseen data.

• Comparative analysis: The proposed model’s performance was compared with three other state-of-the-art forecasting algorithms: SVR, RFR, and Facebook Prophet. The REDf model consistently outperformed these models across all evaluation metrics, further validating its effectiveness in predicting short-term energy demand.

• Model architecture and training: The model was developed using a systematic approach, including data preprocessing, model selection, and hyperparameter tuning through grid search cross-validation. This comprehensive approach ensures that the model is optimized for the specific characteristics of the energy demand data, enhancing its validity.

• Consistency across datasets: The model’s high performance across all four datasets, with minimal variation in MAE, RMSE, and $R^2$ values, indicates that the model is not overfitted to a specific dataset and can generalize well to different energy consumption patterns.

• Absence of overfitting and underfitting: The loss curves for training and testing phases, as well as the consistency of performance metrics, suggest that the model does not suffer from overfitting or underfitting. This further supports the validity of the model as it demonstrates the ability to generalize effectively to new data.

The combination of these factors: rigorous evaluation, comparison with other models, and consistent performance across diverse datasets supports the validity of the proposed model as a reliable tool for forecasting short-term energy demand in smart power grids.

Limitations

While the proposed LSTM-based deep learning model demonstrates high accuracy in forecasting short-term energy demand, several limitations must be acknowledged.

• Data dependency: The model’s performance is highly dependent on the availability and quality of historical energy demand data. Inconsistent or incomplete data can significantly affect the model’s accuracy. Additionally, the model was trained and validated on datasets from specific energy distribution companies, which may limit its generalizability to other regions or datasets with different characteristics.

• Computational complexity: The LSTM model requires substantial computational resources, particularly for training on large datasets. This high computational demand may pose challenges when deploying the model in real-time applications or resource-constrained environments.

• Interpretability: Deep learning models, including LSTM networks, often function as “black boxes,” making it difficult to interpret the decision-making process. This lack of transparency can be a drawback, especially when model decisions need to be explained to stakeholders or when accountability is required.

• Exclusion of external factors: The model primarily relies on historical energy demand data and does not explicitly account for external factors such as weather conditions, economic activities, or policy changes, which can also influence energy demand. Incorporating such variables could potentially enhance the model’s accuracy and reliability.

Besides the above mentioned points, reducing MAE enhances forecasting precision, but it comes with certain trade-offs. Achieving a lower MAE often requires deep model architectures and extensive hyperparameter tuning, significantly increasing computational costs. Additionally, optimizing solely for MAE can make the model overly sensitive to short-term fluctuations, potentially compromising its ability to capture long-term trends reliably. To address this, we ensured a balanced evaluation by maintaining low RMSE values and high $R^2$ scores, promoting overall model stability. Future research should explore multi-objective optimization strategies to strike a balance between MAE, RMSE, and model robustness.

Addressing these limitations in future research could involve incorporating additional data sources, such as real-time weather data or economic indicators, enhancing model interpretability, and optimizing the model’s computational efficiency for broader applicability in smart grids.

Future work

To further enhance predictive accuracy and adaptability, future research will explore reinforcement learning (RL) for multi-period forecasting and federated learning (FL) for privacy-preserving energy demand prediction. RL-based models can dynamically adjust forecasts based on real-time grid feedback, enabling automated demand response management and improved adaptability to sudden fluctuations (Li et al., 2023). By integrating Deep Q-Networks (DQN) and Proximal Policy Optimization (PPO), RL can optimize long-term forecasting while ensuring efficient energy allocation.

Meanwhile, FL offers a decentralized approach to model training, ensuring privacy protection by allowing energy providers to collaborate without sharing raw data. FL enhances scalability, security, and efficiency, making it suitable for distributed smart grid networks (Li, Wang & Yang, 2021). Integrating techniques such as federated averaging (FedAvg) and differential privacy mechanisms will further strengthen data security while maintaining high forecasting accuracy.

A promising direction is a hybrid RL-FL framework, where FL trains decentralized models securely, and RL optimizes real-time energy distribution. This approach can create a secure, intelligent, and adaptive forecasting system, supporting the transition to a sustainable and AI-driven smart grid. Future studies will focus on implementing these methodologies to improve forecasting precision, grid stability, and privacy protection in large-scale energy networks.