Bayesian network model structure based on binary evolutionary algorithm

The Bayesian network model

It is not feasible to apply the joint distribution directly for multivariate probabilistic inference, and the computational complexity is super-exponential with the variables. The Bayesian network decomposes the joint probability by using conditional independence between variables, which reduces the number of parameters and is a powerful tool for probabilistic reasoning. The Bayesian network inference includes the posterior probability problem, the maximum posterior probability problem (Maximum a Posterior estimate, MAP), and the maximum probable explanation problem (most probable explanation, MPE) (Chiribella et al., 2010). Generally speaking, probabilistic reasoning refers to the problem of the posterior probability. At present, two heuristic algorithms, the maximum potential search, and the minimum missing edge search, are commonly used. Shafer & Shenoy (1990) proposed the clique tree propagation algorithm, which uses step sharing to save at least half of the time to complete the posterior inference. In essence, the clique tree propagation algorithm and the variable elimination method are the same, and both are used for exact inference (Henckel, Perkovi & Maathuis, 2022). When the network nodes are numerous and dense, the computational complexity is high. Thus, the approximate reasoning algorithm is usually used. The typical approximate reasoning methods include the random sampling method, including repetitive sampling and Markov Chain Monte Carlo (MCMC) sampling; variable distribution method, including naive average length method and loop propagation method; the model simplification method, including removing variables with less influence and reducing the state space of variables; including search-based algorithm, top-N algorithm, deterministic algorithm, MPE approximation algorithm based on an ant colony, etc. (Sood, 2019).

It uses an adjacency matrix or adjacency list to represent a directed acyclic graph. In the process of finding the final solution, every Bayesian network is a candidate, and the Bayesian network is expressed as: (1)${\displaystyle \lambda =\left(\alpha ,\beta \right)}$

where λ is the number of nodes in the network. The topological structure of a Bayesian network can be expressed as the following matrix form when it means that there is a directed edge between node α and node β. When the direction is α, it means that there is no edge between node α and node β  (Saengkyongam & Silva, 2020). The topology is displayed in Fig. 1.

Bayesian networks adopt an adjacency matrix to represent the topological structure of Bayesian networks, which can transform the changes in the abstract topological structure into the changes in the elements of the matrix. The optimization algorithm is to optimize the topology of the Bayesian network, in fact, it is to optimize its adjacency matrix (Khalifa Othman et al., 2022).

Two-tuple classifier

A two-tuple, α represents a network topology with a set of attributes $\gamma =\left({\gamma }_1,{\gamma }_2,\dots ,{\gamma }_n\right)$ and class variables β (Xiangyuan et al., 2021). α is a DAG (Directed acyclic graph) as shown in Fig. 2.

In a directed acyclic graph, the vertices in the graph represent any attribute, and the edges in the graph represent the probabilistic dependencies between attributes.

The β in the two-tuple refers to a set of parameters used to quantify the entire Bayesian network (Liu et al., 2021). Bayesian network classifier is a special type of Bayesian network in classification problems. Given an unlabeled instance $\gamma =\left({\gamma }_1,{\gamma }_2,\dots ,{\gamma }_n\right)$, the Bayesian network classifier λ assigns a class label φ to γ with the maximum posterior probability by Eq. (2) (Duan et al., 2020). (2)${\displaystyle \Delta {\phi }_{\lambda }=\mathrm{argmin}{\rho }_{{\phi }^{\prime }}\sum _{i=1}^n\rho \left|{\alpha }_i{\beta }_i\right|}$

where, ρ represents the set of variables other than class variable for attribute in.

Binary improved algorithm

The constraint-based structure learning algorithm is to find out the conditional independence relationship between nodes. If the nodes are conditionally independent, there is no edge, otherwise, there is an edge between nodes, such as the PC (Peter-Clerk) algorithm (Srivastava, Chockalingam & Aluru, 2020). The structure learning algorithm based on the scoring search is to list all possible Bayesian network structures including all nodes. Then, score each network structure with a specified scoring method. Finally, construct the Bayesian network structure according to a certain search strategy and scoring criteria, such as the GES (Optimal Structure Identification With Greedy Search) algorithm. Mixture-based structure learning algorithm refers to the use of a conditional independence test to reduce the complexity of the search space. Then, score search to find the optimal Bayesian network, such as MMHC algorithm (Wang et al., 2019).

In this article, a binary adaptive Lasso algorithm called NS-DIST is proposed, in which the first stage performs neighborhood selection. Then, the second stage estimates the DAG by a discrete improved tabu search (DIST) algorithm based on steepest descent, loop elimination, and tabu list in the selected neighborhood.

Binary clustering

Inspired by static group behavior and dynamic group behavior, the binary will be divided into many subgroups and operated on different regions, which corresponds to the spatial exploration stage. Binaries will also cluster together to form a large group of operations along a direction, which corresponds to the stage of local mining. Binary behavior follows the principles of grouping, separating, pairing, staying away from attacks, and being attracted to the source of the target. Each binary in the population represents a solution to the exploration space (Zhang, Petitjean & Buntine, 2020). Binary group movements are determined by a number of basic operations: separation, alignment, cohesion, being attracted to the target and staying away from the attack (Yosuf et al., 2022). Wherein the target node is the individual position of the optimal fitness value explored by the population in history. The attacking node is the individual position of the worst fitness value explored by the population in history. Separation refers to the static collision that occurs in order to avoid separation from the individual next to it. Formation means that the speed of an individual matches the speed of other individuals around him. Clustering refers to the tendency of individuals to go to the center of the neighborhood (Nouri-Moghaddam, Ghazanfari & Fathian, 2021).

In the process of binary clustering, as the domain radius of other individuals around an individual becomes larger, they often cluster into small groups, cluster within the domain radius, or expand the range of activities to stay away from attacks (Jumelet et al., 2021).

The behavior of the binary is influenced by the combination of these five modes. To update the position and simulated movement of a binary in the search space, a step vector and a position vector are introduced (Atoui et al., 2019).

The step vector represents the direction of binary movement, which is defined as: (3)${\displaystyle \Delta f_t=\left[q{W}_i+y{R}_i+p{U}_i+g{K}_i+v{H}_i\right]\varpi \Delta {\varphi }_t}$

where Δf_t is the mathematical modeling of the segregation behavior of the i th individual in the subpopulation.

y is the mathematical modeling of the formation behavior of the i th individual in the subgroup.

P is the target weight, which is the mathematical modeling of the behavior of the i th individual attracted by the target.

One of the functions of v, which is the exchange function of the behavior of the i th individual in the subgroup away from the attack, is to balance exploration and exploitation.

If the transformation function is unchanged during the iteration process, the probability will be calculated in the same way throughout the optimization process. Changing the transformation function may better balance exploration and development (Min et al., 2020).

Penalty function

Penalized likelihood under the assumption of equal variance of latent variables, the final form of the score function is: (4)${\displaystyle {\lambda }_{\left(x\right)}=\sum _{i=1}^n\left|{\varphi }_n^k-{\varphi }_m^j\right|\cdot {\left(\frac{1}{{\mu }_{i}t}\right)}^2.}$

In the score function, ${\varphi }_n^k$ is a data matrix of $\alpha \times \beta .{\varphi }_m^j$ is a data vector of the jth variable. $\left(n,m\right)$ is a coefficient vector, and $\left(i,j\right)$ is a column vector of the jth row of the matrix ϕ. Where, μ_ij is estimated by canonical lasso regression and the initial penalty parameter is given by Eq. (5): (5)${\displaystyle {\mu }_{i}t=max\sum _{i=1}^n{\left(\frac{1}{\left|{\theta }_{ij}\right|}\right)}^2{\left(\frac{1}{{\left|F\right|}^{-1}}\right)}^2}$

where θ_ij is the coefficient vector, and F represents the vector group of the undirected neighborhood matrix.

Simulation platform construction

In this article, a federated transfer learning of a Bayesian network based on an improved binary algorithm is proposed. By removing the restriction, each client must use k2 to learn the Bayesian network algorithm. It eliminates the need for expert knowledge such as node order. Based on the improved NOTEARS dual ascent method, the problem is that the non- k2 algorithm will produce a ring after the binary is solved. Therefore, in the experimental part of this article, the effect of network loop removal is firstly verified, compared with the original NOTEARS algorithm applied to the loop removal process; secondly, the performance of the improved binary algorithm is analyzed, and the NOTEARS algorithm and the improved NOTEARS are compared by taking the particle swarm optimization algorithm used by all participants as an example.

The configuration of the experimental platform is displayed in Table 3.

In this article, the NOTEARS algorithm proposed in (Lee & Kim, 2019) and the EM algorithm proposed in (Wang, Ren & Guo, 2022) are adopted for horizontal comparison. These two comparison methods are the mainstream algorithms at present.

1. The NOTEARS algorithm is improved, and the loop removal operation is transformed into a constrained optimization problem. Let W be the adjacency matrix of a graph structure with k cycles. There will be non-zero elements on the k-power locus of its adjacency matrix.

2. EM algorithm, as an iterative algorithm, is a classical method for data completion in the case of missing data (Saarela, Rohrbeck & Arjas, 2022). As a single-step iterative algorithm, the main idea is to sample the current node with the knowledge of other nodes, that is:

where: Tⁿ is the nth iteration of the data T, W is the number of data samples, and $U\left({\alpha }_i\left|{\overrightarrow{\alpha }}_j\right.\right)$ represents the sampling values of other data except j.

Analysis of experimental results

Analysis of ring removal effect

The number of participants in federated transfer learning is 100, and the characteristics of each participant are missing with a probability of 0.2. The Bayesian network generated by particle swarm optimization is adopted to analyze the results.

The weight matrix represents the selection of each side by the participants. The larger the weight is, the more votes are obtained. More such sides are expected to remain. The evaluation index is as follows: (7)${\displaystyle \delta =\frac{max\left({\varphi }_i+{\varphi }_j\right)}{{\eta }_t}}$

The binary algorithm proposed in this article is compared with the NOTEARS algorithm. The calculation results are displayed in Fig. 4.

Figure 4 presents the results of the algorithm in this article and the NOTEARS algorithm on the binary matrix with ALARM and INSURANCE data sets. It can be seen from Fig. 4 that the number of edges retained by the NOTEARS algorithm in the ring removal process will gradually decrease to zero with the increase of its parameters. Its value requires a heuristic strategy. However, the edges retained by the algorithm in this article will gradually converge to a fixed value with the increase of K. Table 4 indicates the final convergence of the edges of the algorithm in this article in an average of 10 experiments on ALARM, INSURANCE, CHILD and ASIA networks.

The proposed algorithm outperforms NOTEARS algorithm on almost all data sets and outperforms the centralized learning method by a large margin on all given data sets. In this article, we propose a Bayesian network federated transfer structure learning based on the improved binary algorithm, which eliminates the limitation of the fixed learning algorithm in the application of the binary algorithm in Bayesian network federated transfer learning and makes the binary algorithm more universal. The performance of the proposed algorithm is verified by experiments.

Influence of the number of AP clusters on the ensemble

Among the 200 Bayesian regression sub-networks sorted by the validation set, the number of clusters is set to 2-16 by changing the preference P of the AP clustering algorithm. The prediction uncertainty weighting method is adopted to generate conclusions. The traditional NOTEARS algorithm is introduced, and the error of the EM (Expectation-Maximum) algorithm and the proposed algorithm is analyzed in the case of different cluster numbers. Figure 5 presents the clustering error scatter.

Table 4:

Convergence data set of each network.

Index	ASIA	CHILD	INSURANCE	ALARM
Number of binary participants	200	200	200	200
Original sides	7	17	25	42
Number of edge convergence	8	20	34	47

DOI: 10.7717/peerjcs.1466/table-4

In Fig. 5, X-Y is the prediction error accuracy range, where the Y-axis “-” value is the unattracted node. The algorithm will eliminate this part, so only the “+” value is taken; O is the best accuracy. O1 is the origin. B is the value radius, and A is the threshold range.

The comparative analysis data is shown in Table 5.

The error dispersion of the prediction model of this algorithm is better than other methods in the comparison of clustering accuracy. The prediction model of the traditional NOTEARS algorithm has better dispersion within the error threshold, but the dispersion convergence outside the threshold is poor. The prediction model of the EM algorithm has the lowest clustering accuracy.