Cloud-based real-time enhancement for disease prediction using Confluent Cloud, Apache Kafka, feature optimization, and explainable artificial intelligence

Data description

This study utilized UCI’s machine learning repository to aggregate the baseline data set on chronic kidney disease (Rubini, Soundarapandian & Eswaran, 2015). The data set includes 400 cases of CKD, 150 of which are negative and 250 of which are positive. The data set is composed of 24 features that are classified into 13 categorical features and 11 numeric features, with a class label that has two values: 1 and 0. The details of the characteristics of each feature are shown in Table 2.

Data preprocessing

We applied various data preprocessing steps:

• Data encoding: the dataset includes categorical and numeric features. The Scikit-learn library is used to encode all categorical features into numerical data.

• Missing values can be filled in with several statistical methods, depending on how much data is missing and how important the missing feature. Mean, maximum, and mode are good statistical techniques when 5% to 10% of values are missing. Our study has a low number of missing values, which is handled using mean.

Splitting datasets

The dataset was split into different cases and we performed experiments in different cases, including Experimental 1 (80% training set and 20% testing set), Experimental 2 (70% training set and 30% testing set) and Experimental 3 (60% training set and 40% testing set).

Feature selection methods

Feature selection can be characterized as selecting a smaller selection of appropriate characteristics (variables, attributes) from a more excellent range of available features (Sánchez-Maroño, Alonso-Betanzos & Tombilla-Sanromán, 2007; Jović, Brkić & Bogunović, 2015). Deleting redundant features reduces the dimensionality of machine learning models and increases performance and efficiency. PSO and GA are feature selection methods used. The advantages of applying GA and PSO are: GA acquired from its characteristics, which gave it the strength and efficiency necessary to be effective in finding optimal local solutions (Alam et al., 2020). Also, its flexibility gave it the ability to adapt to the various problems that it may encounter while selecting different features, especially when dealing with big data environments. Its characteristics gave it the power to choose the best features based on the extent of their contribution to the predictive distribution clearly and distinctly. Therefore, it combines exploration to find good solutions with the ability to improve existing solutions effectively (Alam et al., 2020). PSO is characterized by ease of implementation due to the limited number of its tuning parameters compared to similar algorithms (Houssein et al., 2021). This will lead to its remarkable speed in reaching good solutions due to its inherited effectiveness from its characteristics in finding a rapid convergence to the desired solution. Therefore, although there are many algorithms available for feature exploration, both PSO and GA have shown speed and efficiency in reaching rapid convergence of the available and optimal solutions in feature sets (Yadav & Anubhav, 2020).

• •

  PSO is a bio-inspired search and optimization system that uses a population-based strategy to find the best solution for a given problem (Xue, Zhang & Browne, 2012). In PSO, a population of potential solutions, known as particles, traverses the search space in search of the optimal or near-optimal solution. Each particle represents a potential solution and progresses through the search space based on its experience and the experience of the best performing particles in the population (Tran, Xue & Zhang, 2018). The process begins by populating the search space with randomly distributed particles. Every particle has a position as well as a velocity. The position represents a potential solution, while the velocity specifies the direction and speed of the particle’s movement (Marini & Walczak, 2015). The particles modify their positions and velocities during each generation or iteration of the algorithm, which is an iteration based on their individual experiences and the experiences of the top-performing particles in the population.

• •

  GA is a computational tool for addressing difficult search and optimization tasks and extracting or choosing relevant features (Mitchell, 1998; Kramer & Kramer, 2017). GA iteratively builds a population of chromosomes (solutions) and their genes through selection, crossover, and mutation, with the fittest solutions prevailing. Individuals in the population are evaluated using an objective function or heuristic, which is utilized to choose individuals to reproduce on each repeat (Forrest, 1996; Wright, 1991). Those samples that perform better in the objective function will have a higher chance of reproduction. The best individual, picked after many generations, is the end result (Wright, 1991). GA encodes the optimization function as bit arrays that resemble chromosomes, and Genetic operators customize strings to discover a near-optimal solution to the problem at hand. This is accomplished by following the methods illustrated in Fig. 3 (Fong et al., 2014):

  1. Coding the objectives or cost functions.

  2. Creating a fitness function.

  3. A generation is the process of producing a person (solution).

  4. Evaluating the fitness function of individuals in the population.

  5. Creating a new population by crossover and mutation, fitness-proportionate growth, and then replacing the previous population and looping with the new population.

  6. Decoding the outputs to resolve the inquiry.

• Crossover, mutation, and selection are the three fundamental genetic operators in genetic algorithms (Al-Asasfeh, Hamdan & Abo-Hammour, 2013):

  • –

    Crossover: The swapping of chromosomes or other solutions representing sections of a solution. The primary goal is to give subspace convergence and solution mixing.

  • –

    Mutation: The unplanned alteration of the constituent parts of a single solution enhances population variety and provides a technique to avoid a local optimum.

  • –

    Selection of the fittest is passing on options with high fitness to future generations, usually by selecting the most acceptable solutions.

Baseline machine learning models

Different ML models are applied:

• Support vector machine (SVM) is a supervised learning technique that is most commonly used for classification tasks. To categorize data points, the algorithm creates a hyperplane or a series of hyperplanes in a high-dimensional feature space (Wang, 2005). The objective of SVM is to maximize the margin, which is the distance between the hyperplane and the nearest data points in each class. It can handle both linearly and non-linearly separable data by employing various kernels or mapping functions (Meyer & Wien, 2001).

• Naive Bayes (NB) algorithm (Webb, Keogh & Miikkulainen, 2010) is a probabilistic machine learning technique commonly used for classification tasks. NB is founded on Bayes’ theorem, which describes the likelihood of an event given prior knowledge or evidence. The assumption of feature independence is made by Naive Bayes, which means that the presence or absence of one feature is unrelated to the presence or absence of any other feature (Leung, 2007).

• Decision tree (DT) is a supervised learning algorithm that is adaptable and interpretable. DT can be utilized for classification and regression tasks to create a tree-like structure by recursively splitting the data based on distinct attributes and their values (Kotsiantis, 2013). Each internal node corresponds to a feature test, and each leaf node corresponds to a class label or a predicted value. Decision trees can capture complicated linkages and interactions between elements (Kingsford & Salzberg, 2008).

• Random Forest (RF) (Rigatti, 2017) is a collaborative learning method that makes predictions by combining numerous decision trees. It builds an ensemble of decision trees by training each tree on a random subset of the training data and characteristics. During prediction, each tree in the forest predicts the output separately, and the final forecast is established by majority vote (Cutler, Cutler & Stevens, 2012).

• Logistic regression (LR) is a statistical ML technique used for binary classification tasks to predict the likelihood of an instance falling into a specific class. The logistic function (sigmoid function) describes the relationship between the characteristics and the binary outcome in logistic regression. Predictions based on LR yield discrete values best suited for binary categorization (Boateng & Abaye, 2019).

The stacking model

The ensemble machine learning technique combines multiple models as base learners to provide predictions or judgments. It uses the diversity and accumulated knowledge of the base learners to improve the overall prediction performance (Zounemat-Kermani et al., 2021). Multiple models are combined in model stacking, and a meta-learner model is used to improve model predictions. The meta-learner minimizes the weaknesses of each model while maximizing its strengths. Using the k-fold cross-validation technique, a meta-model is built from the predictions of several ML base models (weak learners). Finally, the meta-model was trained using the “final estimator” or “final learner” ML model (Cui et al., 2021). As a result, overall performance can be improved and a model superior to any intermediate model can be obtained (Sikora, 2015).

The stacking model develops during two levels as shown in Fig. 4.

In level 1, heterogeneous weak learners learn in parallel using a training set. Then, the output of the training set is combined in stacking training, and the production of the testing set is combined in a stacking test.

In level 2, a meta-learner is trained using stacking training and evaluating using testing stacking to predict the final output.

Explainable artificial intelligence

XAI concerns developing machine learning and artificial intelligence (AI) algorithms that justify their predictions or conclusions (Barredo Arrieta et al., 2020). The gap between the complicated inner workings of AI models and the desire for openness and interpretability in their outputs will be bridged with the help of XAI (Ngo, Beard & Chandra, 2022). One advantage of the XAI approach is understanding why an AI system made a particular decision, improving trust, accountability, and the ability to detect distortions or faults (Tjoa & Guan, 2020).

Ensemble models, in their nature, combine multiple algorithms with improving predictive performance, which makes them complex and opaque, making it challenging for stakeholders and users to understand how decisions are made (Rane, Choudhary & Rane, 2024). XAI plays a crucial role in addressing this issue by providing frameworks and techniques that reveal the interpretability of these models in a comprehensible manner by breaking down the contributions of individual models within the ensemble, highlighting how each one influences the final prediction (Dwivedi et al., 2023). XAI plays an essential role in improving the interpretability of ensemble models by analyzing information: XAI techniques can measure the contribution of each in the ensemble model’s predictions by identifying global feature importance and local importance. This helps users understand which of the most essential features play a vital role in decision-making in the ensemble. XAI tools can decompose the final output into the contributions of each base model in the ensemble, demonstrating how individual models (Chamola et al., 2023).

• Global exploitability aims to fully understand the behavior of an AI model in its entire input space (Ding et al., 2022). It focuses on discovering the model’s biases, habits, or rules. Global exploitability approaches can be used to understand the structure of the model, its learned representations, and the relative importance of various aspects (Ding et al., 2022).

• Local explainability is concerned with providing reasons for specific predictions or decisions that an AI model makes. Attempts to discover the variables or features that contribute to a particular outcome. Local explainability techniques can illuminate the model’s decision-making process by emphasizing the significance or contribution of various features for a single instance (Ding et al., 2022).

• SHAP (SHapley Additive exPlanations) is an XAI method that attempts to deliver information about machine learning models’ predictions. It is based on the cooperative game theory notion of Shapley values. SHAP’s fundamental principle is to assign a relevance score to each feature in a forecast. These scores represent how much each feature adds to the prediction compared to a reference baseline (Marcílio & Eler, 2020). Understanding the feature contributions provides insights into the model’s decision-making process and allows for improved interpretation of its predictions (Marcílio & Eler, 2020).

Evaluation models

Classification performance is typically measured using precision, recall, F1-score, and accuracy, Matthews correlation coefficient (MCC) and Cohen Kappa. True positive (TP), false positive (FP), true negative (TN), and false negative (FN) are calculated as true positives, false positives, and false negatives, respectively. While TN showed a negative result, it actually returned a positive result, while TP showed a negative result, but it actually returned a positive result. TP indicates that the result is actually positive, whereas TN indicates that the result is actually negative.

(1) $$Accuracy=\frac{TP+TN}{TP+FP+TN+FN}.$$ (2) $$Precision=\frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}}$$ (3) $$Recall=\frac{TP}{TP+FN}$$ (4) $$F1\text{-}score=\frac{2\cdot precision\cdot recall}{precision+recall}$$ (5) $$MCC=\frac{TP\cdot TN-FP\cdot FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}$$ (6) $$CohenKappa=\frac{(TP+TN)-E}{N-E}$$where: N = TP + TN + FP + FN and $E=\frac{(TP+FP)(TP+FN)+(FN+TN)(FP+TN)}{N}$

The receiver operating characteristic (ROC) is a graphical curve that evaluates the performance of binary classification at various thresholds. It shows the trade off between TPR and specificity.

Big data platforms

Online prediction pipelines were developed using Apache Kafka and the confluent platform.

• •

  Kafka was used to build streaming apps and pipelines for real-time data. The Kafka system can receive large amounts of data in real-time and with low latency in addition to being real-time, fault-tolerant, and scalable. It (Garg, 2013) Streaming data can be read from sensors and ports. Kafka allows you to read and write streams of data. The development of real-time stream processing applications is possible. The data streams are stored in a distributed, replicated, and fault-tolerant cluster. There are two main libraries for APIs: Producer APIs and Consumer APIs. Kafka topics can be created using the Producer API. Kafka topics are subscribed to, and stream records are processed using the Consumer API.

• •

  The Apache Kafka stream processing engine is an open-source, distributed system that offers the ability to filter, aggregate, and join data streams based on predefined patterns as an IaaS (Infrastructure-as-a-Service) service to generate predictive inferences (Garg, 2013). It supports publishing and subscribing to data streams, storing them fault-tolerantly, and processing them in real-time. Streaming engines process sensor data streams in real-time to predict event patterns based on sensor observations/readings and correlate the data with predefined/preset thresholds. As data streams from heterogeneous producers are processed via APIs in a fault-tolerant manner in a producer-publish, consumer-subscribe model, the platform is like an enterprise messaging system (Garg, 2013). The data streaming pipeline is processed and analyzed between Apache Kafka Connect APIs and heterogeneous systems or applications (Cao et al., 2015). KSQL is a streaming SQL engine to executes queries in real-time without pushing the data stream to the database. A data stream is a continuous flow of data records analyzed through the stream processing engine to generate an output stream (Garg, 2013). Sensors and IoT are utilized in smart cities, healthcare, and energy management to generate streaming data that is used to develop real-streaming predictive systems (Cao et al., 2015). A cluster of Apache Kafka servers can be installed locally or on the cloud, and each producer connects to the cluster through the Kafka Connector APIs. Topics are categories in which data records are stored. The records consist of a key, a value, and a timestamp. Input streams are converted into output streams using Kafka stream processing, which consumes input streams from available topics and produces output streams.

• Kafka includes sets of features

  • –

    Topic: The Kafka cluster categorizes messages into topics based on their similar attributes. In an ordered list, topics are divided into partitions determined by an incremental identifier called an offset, which identifies messages/data in each partition. The Consumer reads the messages based on the chronological order or sequence of arrival generated by the Producer, where the messages are written to the appropriate topics (Garg, 2015).

  • –

    Producers: Apache Kafka producers are the sources of data and messages that write to topics, which are then written to partitions and stored on brokers. Sensors, devices, and manual or automatic systems produce events, which are a type of event producer (Garg, 2015).

  • –

    Consumer: A consumer subscribes to a topic to ingest or consume the messages/data generated by the producer (Narkhede, Shapira & Palino, 2017). Consumers are configured to read streaming data by automatically connecting to a suitable broker with read access. Consumers automatically fetch data from the next available broker when there is a failure of one of the brokers in the cluster. According to the offsets, data/messages are consumed in order. Streaming data offsets provide consumers with information about where to start reading a topic’s streaming data. Data/messages are read from multiple partitions of the same topic in parallel when there is more than one partition (Narkhede, Shapira & Palino, 2017).

• •

  The confluent platform is used to develop fully streaming pipelines based on Apache Kafka (Confluent, 2014). It provides real-time streaming of data on a scalable platform, making the information flow seamless. Apache Kafka is integrated with it, allowing efficient data processing, storage, and streaming. In addition, it provides scalable solutions to accommodate growing volumes of data, making it possible for the platform to handle more workloads and a greater number of demands over time (Confluent, 2014). A fault-tolerant mechanism is integrated into the Confluent platform, improving workflow reliability and resilience. The streaming platform provides connectors and APIs for integrating data sources and systems. Platform administrators can effectively monitor and manage the streaming infrastructure with management and monitoring tools (Confluent, 2014).

Online prediction pipelines steps

Numerous data sources in the healthcare industry continuously produce real-time data from linked equipment, including medical devices: Wearable sensors, such as glucose and heart rate sensors, and smart homes, which are IoT smart appliances, security cameras, and thermostats. Kafka is used to consume and produce these data. Different types of pre-processing steps will be applied, such as filling in missing values, converting data formats to maintain uniformity and normalize numerical values. In our work, online prediction pipelines consist of the following steps:

The Confluent Cloud and Apache Kafka service are integrated with a Python application to develop online prediction pipelines. Python applications can produce and consume messages using a Confluent Cloud and Apache Kafka cluster. The process involves creating an account on Confluent Cloud, setting up a Kafka cluster, and creating Kafka topics. API keys are then generated for authentication purposes. These keys enable Python applications to securely connect to the Confluent Cloud and Apache Kafka clusters.

• Set up Confluent Platform: We create an account on the Confluent Platform, then create a cluster and topics, and generate keys. These keys are used to connect to topics, allowing a Python script to produce and consume messages using the keys.

• A Python script is being developed to generate streaming health attributes. A vast range of health attribute data is generated and pushed into a Kafka topic in using the Producer API (confluent_kafka.Producer). The script buffers incoming data streams from the producers and transfers the data to the topic. This results in improved fault tolerance due to load balancing in the event of component failures.

• A Python script has been developed to consume health attributes from a Kafka topic using the Consumer API (confluent_kafka.Consumer). The consumed health attributes are then applied different pre-processing steps Impute missing values mean, median, mode and then converted to the same format as the input to the model. Subsequently, the model is loaded and applied to the health attributes for real-time prediction. Finally, the prediction result is saved to a topic.

Experimental 1

Table 5 shows a comprehensive evaluation of ML models (RF, LR, DT, SVM, and NB) and stacking models based on two different types of FS methods, namely PSO and GA, on the task of predicting CKD with a set of tests 20% using evaluation metrics: accuracy, precision, recall and F1-score and AUC and two statistical analysis Kappa and MCC. Based on the GA, the stacking model achieves 100 as the best results compared to other models and scores 100 on both Kappa and MCC because it enhances results by integrating the output of base models with meta-learner to strengthen performance and generalization. NB records the worst performance with 92.50 accuracy, 92.60 F1-score, and 84.615 Cohen Kappa for GA, 93.75 accuracy, 93.83 F1-score and 87.09 Cohen Kap for PSO. The second-best performance is conducted by RF and SVM for GA with accuracy, precision, recall, and F1-score at 98.75, 98.79, 98.75, and 98.75, respectively, because RF combines multiple decision trees to reduce overfitting and enhance performance. Based on GA, RF, SVM, scores very high scores (97.3–97.4) on both Cohen and Kappa MCC.

Figure 5 shows the ROC curves for these algorithms based on the area under the curve(AUC) metric. As the AUC value increases, the model’s generalization ability increases, as indicated on the ROC curve by the curve close to the upper-left corner. NB performed the worst, with an AUC value of 94 for GA and 95 for PSO. The stacking model had the highest AUC value of 100 for GA.

Experimental 2

Table 6 compares the effect of using two different FS methods, GA and PSO, based on various models: stacking, RF, LR, DT, SVM, and NB. with a 30% testing set. The comparison relies on measuring each model’s achieved accuracy, precision, recall, and F1-Score, and two statistical analysis Kappa and MCC.

For GA, the stacking model achieves the best performance compared to other models, with accuracy at 99.17%, precision at 99.18, recall at 99.17, and F1-score at 99.17, Cohen Kappa at 98.23, and MCC at 98.24. DT and LR record approximately the same percentage of each score at 95%. DT records lowest at 95.83, 95.09, 95.83, and 95.78 of accuracy, precision, recall, and F1-score, respectively. For PSO, the stacking model records the best performance compared to ML models with 97.80 accuracy. DT, LR and NB record approximately the same percentage at 94%. NB with PSO records the worst performance compared to other models at 93.44, 93.01, 93.44, and 93.01, 84.34, and 84.66 in accuracy, precision, recall, F1-score, Cohen Kappa and MCC respectively.

Figure 6A displays the ROC curve, and AUC values of each model with GA and PSO. It can be observed that stacking achieves the highest AUC at 99.33 for GA, indicating the best discriminatory ability in distinguishing between the positive and negative classes. While NB had the lowest AUC at 91 for PSO.

Experimental 3

Table 7 shows a comprehensive evaluation of ML models (RF, LR, DT, SVM, and NB), stacking models based on two different types of FS methods, namely PSO and GA, on the task of predicting CKD with a set of tests 40% using evaluation metrics and two statistical analysis: Cohen Kappa and MCC. Based on the GA, the stacking model achieved 98.75, 98.77, 98.75, 98.75, 97.333 and 97.333 as the best results of accuracy, precision, recall, F1-score, Cohen Kappa, and MCC compared to other models because it enhances results by integrating the base models with meta-learner to strengthen performance and generalization. The second-best performance is conducted by RF for GA with accuracy, precision, recall, F1-score, Cohen Kappa, and MCC at 96.88, 96.89, 96.88, 96.88, 93.355, and 93.364, respectively, because RF combines multiple decision trees to reduce overfitting and enhance performance. DT records the worst performance with 91.88 accuracy 91.95 F1-score, and 81.818 Cohen Kappa for GA. Based on PSO, the stacking model records the best performance with 96.25 accuracy, 96.26 F1-score, and 92.053 MCC. DT records the worst with 90.50 of accuracy, 90.70 of F1-score, and 80.07 of MCC.

Figure 7 displays the ROC curve and AUC values of each model with GA and PSO. It can be observed that stacking achieves the highest AUC at 98.667 for GA, indicating the best discriminatory ability in distinguishing between the positive and negative classes. DT had the lowest AUC at 90 for PSO.

Local exploitability

The waterfall plots in SHAP provide descriptive information about individual predictions. In SHAP waterfall plots, red bars indicate positive contributions, while blue bars indicate negative contributions. On the right, you can see the importance of the feature, and on the left, you can see the value of the feature. Figure 16 illustrates the prediction of the model as $\mathrm{f}(\mathrm{x})$, and the positive bias or intercept is displayed below the plot as $\mathrm{E}[\mathrm{f}(\mathrm{x})]$. Then, for each prediction, a red or blue line shows how the positive (red) or negative (blue) contribution of the feature affects the model output.

In Fig. 16A, the length of the corresponding bar represents the importance of each feature in the observation. Hemoglobin has the highest contribution with a SHAP value of +2.49. Specific gravity has the second-highest contribution with a SHAP value of +2.22. Serum creatine has the third-highest contribution with a SHAP value of +1.69.

In Fig. 16B, the red bars show the features that affect positively predicted values. Hemoglobin has the highest positive contribution with a SHAP value of +1.92. Hemoglobin has the second-highest positive contribution with a SHAP value of +1.38. The blue bars show the features that affect negatively predicted values. Albumin has the highest negative contribution with a SHAP value of −2.23. Hypertension has the second-highest positive contribution, with a SHAP value of −1.09.