A non-linear optimization based robust attribute weighting model for the two-class classification problems

Dataset

Four separate datasets were used in the study. Two are the Liver Disorders dataset and the Iris dataset obtained from real observations, while the Full Chain dataset and Two Spiral dataset are artificial datasets. Linear classifiers and sets that are difficult to distinguish were preferred in selecting artificial datasets. All datasets consist of real numerical attributes.

Liver Disorders dataset

The Liver Disorders dataset was prepared by Bupa Medical Research. It is a dataset consisting of six attributes, five of which are attributes obtained by blood tests and one of which is the amount of alcohol consumption. It contains a total of 345 observations. It has positive and negative classes (BUPA Medical Research Ltd, 1990; McDermott & Forsyth, 2016). In Fig. 4, given data distribution for three attributes. The figure shows that the class represented by red crosses and blue circles are intertwined.

Iris dataset

The Iris dataset is frequently used in machine learning studies. The dataset includes 150 observations of three different iris plants. It has four attributes. The species names with class output are numbered 0 − 1 − 2 (Fisher, 1936; Fisher, 1988).

Full chain dataset

The Full Chain dataset is an artificially generated dataset with two nested rings. The dataset we produced for the study contains three attributes and two classes. One thousand observations were produced. In Fig. 5, data distribution for three attributes is provided. The figure shows that the class represented by red crosses and blue circles is in a nested ring structure.

Two Spiral dataset

The Two Spiral dataset is an artificial dataset produced as two spirals nested within each other. It was produced as 2,000 observations with two attributes and two classes. In Fig. 6, data distribution for two attributes is provided. The figure shows that the class represented by red crosses and blue circles is in a spiral structure.

Optimization algorithms

This section briefly explains the optimization algorithms used in the study. Each algorithm looks for possible solutions with the fitness function described in the mathematical model section. Solutions close to the best possible result are marked as the best. In the study, optimization algorithms look for weight values. Therefore, the initial dimensions are equal to the product of the total number of attributes (m) and the number of classes (C) as given in Eq. (1). Each algorithm uses the initial values given in Table 1. The literature’s most used common values are used for initial values.

As can be seen in Table 1, the number of iterations, which are the stopping criteria, the number of solutions to be calculated in a single iteration, and the lower and upper limits for the weighting coefficients are given jointly for the four algorithms. The specific starting parameters of each algorithm are explained in the headings below.

Particle swarm optimization algorithm (PSO)

The particle swarm algorithm is an algorithm that models the movements of animals living in a herd during foraging (Arican & Polat, 2020). In the PSO algorithm, each particle and the structure created by them is called a flock. Among the initial parameters given in Table 1, c1 is the particle coefficient, and c2 is the social coefficient. It determines the effect of individual and herd on total movement. w is the moment of inertia, which causes each particle to move somewhat, even if it has the best result.

Bat algorithm (BAT)

The bat algorithm is a metaheuristic algorithm that models the way-finding behavior of bats with resonance (Yang, 2010a). From the initial conditions given in Table 1A shows the sound intensity, and r0 shows the propagation rate of the signal. The values of a and are constant values. Selection is made according to the (0) <a <1 and γ>0.

Gravitational search algorithm (GSA)

It is a heuristic algorithm modeled on the basis of Newton’s laws of gravity and motion (Rashedi, Nezamabadi-pour & Saryazdi, 2009). Each piece in the search space is considered a mass, so it is an artificial mass system. All masses exert a force of attraction to each other in the search space and move in the search space under the influence of these forces. The ɛ given in Table 1 refers to a small, fixed number entered by the user.

Flower pollination algorithm (FPA)

FPA is a metaheuristic model based on flowering plants’ pollination and reproduction process (Yang, 2014). It uses biotic reproduction, the reproductive characteristic of flowering plants globally, and abiotic reproduction, as local exploration. P given in Table 1 is a fixed probability value between 0 and 1 for local and global pollination.

The mathematical model of the optimization-based dynamic attribute weighting model

The mathematical model used in determining the weighting coefficients in the study will be explained in this section. At the same time, this model is the fitness function of optimization algorithms.

An example of in-class and between-class distance measurements is given in Fig. 7 for two classes and two attributes. Here dc shows the Euclidean distances of the attribute pairs belonging to the tenth observation for the o c’th grade to their class center. d’c, o shows the Euclidean distances of the attribute pairs belonging to the observation number o for class c to other class centers. Figure 8 gives the flowchart of the proposed model.

Equation (2) calculates the sum of the averages of the distances for the whole class. Here O gives the number of in-class observations. C gives the total number of classes. (2)${\displaystyle WithinClassDistance=\sum _{c=1}^C\frac{\sum _{o=1}^O{d}_{c,o}}{O_c}}$ With Eq. (3), the sum of the averages of the distances between all classes is calculated. (3)${\displaystyle BetweenClassDistance=\sum _{c=1}^C\frac{\sum _{o=1}^O{d}_{c,o}^{\prime }}{O_c}}$ Optimization algorithms looking for weight values by using the fitness function in Eq. (4) perform the weighting process with the equation given in Eq. (5). Here m is the total number of attribute s, and C is the total number of classes. (4)${\displaystyle fitness=\frac{WithinClassDistance}{BetweenClassDistance}}$

Optimization algorithms looking for weight values using the fitness function in Eq. (4) perform the weighting process with the equation given in Eq. (5). Here m is the total number of attribute s, and C is the total number of classes. ${\displaystyle w_1\ast f_1,w_2\ast f_2,\dots ,w_m\ast f_m}$ ${\displaystyle w_{m+1}\ast f_1,w_{m+2}\ast f_2,\dots ,w_{2m}\ast f_m}$ (5)${\displaystyle ⋮}$ ${\displaystyle v{w}_{n-m+1}\ast f_1,w_{n-m+2}\ast f_2,\dots ,w_n\ast f_m}$ ${\displaystyle n=C\ast m}$

Classifiers

Support vector machine and linear discriminant analysis, which perform linear classification, were preferred to see the effect of weighting attributes independently of the classifier. Thus, it can be shown when problems are solved linearly. This section briefly mentions the classifiers; detailed information can be viewed from the references.

Support vector machine (SVM)

It is a supervised learning system used in classification problems. It uses a linear line found at the maximum distance to the data of the two classes to separate the classes. Due to its linearity, its performance decreases in nested datasets (Kim et al., 2003; Sharma et al., 2023; Nour, Senturk & Polat, 2023). SVM are a supervised learning method used in classification problems. Draws a line to separate points placed on a plane. It aims to have this line at the maximum distance for the points of both classes. It is suitable for complex but small to medium datasets. To make the classification, a line separating the two classes is drawn, and the region between ±1 of this line is called the margin. The wider the margin, the better the separation of two or more classes (Yang, 2010b). Figure 9 shows the flowchart of the SVM model.

For a dataset with n elements; (6)${\displaystyle \left(x_1,y_1\right),\dots \dots \dots ,\left(x_n,n\right),x\in R^d,x\in \left\{+1,-1\right\}}$ The training data can be separated with the coefficients w and w₀ and the separator plane to be provided by the equation given in Eq. (7). (7)${\displaystyle D\left(x\right)=\left(w\times x\right)+w_0}$

The separating plane must fulfill the conditions given in Eq. (8). (8)${\displaystyle D\left(x_i\right)=\left\{\begin{array}{c}\left(w\times x\right)+w_0\ge +1,if{y}_i=+1\hfill \\ \left(w\times x\right)+w_0\le -1,if{y}_i=-1\hfill \end{array}\right.i=1,2,\dots ,n}$

Equation (9) is obtained by combining Eq. (8). (9)${\displaystyle y_i\left[\left(w\times x_i\right)+w_0\right]\ge +1,i=1,2,\dots \dots ,n}$

The distance between the dividing plane and the nearest vertex is called the boundary and is denoted by Γ. The larger the limit value, the better the classifier generalizes. The condition given by Eq. (10) must be satisfied for all training examples. (10)${\displaystyle \frac{y_k\times D\left(x_k\right)}{∥w∥}\ge \Gamma ,k=1,2,\dots \dots ,n\mathrm{ve}{y}_k\in \left\{-1,+1\right\}}$

The condition that must be met for the best discrimination value is to set the limit value Γ to be the highest. Because the weight w can take an infinite number of values. The data points on the boundary are called support vectors.

Linear discriminant analysis (LDA)

LDA looks for vectors that best separate classes rather than the one that separates data best. It creates a linear combination that gives the largest mean differences between the desired classes. It is defined by a score function defined by Fisher (Ye, Shi & Shi, 2009).

It was first developed by Fisher (Sayad, 2019). The analysis defines the score function given in Fisher Eq. (12). The problem in this equation is to find the coefficients that will maximize the score. (11)${\displaystyle Z={\beta }_1{x}_1+{\beta }_2{x}_2+.\dots \dots .+{\beta }_n{x}_n}$ (12)${\displaystyle S\left(\beta \right)=\frac{{\beta }^T{\mu }_1+{\beta }^T{\mu }_2}{{\beta }^{T}C\beta }}$ (13)${\displaystyle S\left(\beta \right)=\frac{\overline{Z_1}-\overline{Z_2}}{GruplardakiZvaryansı}}$ (14)${\displaystyle \beta =C^{-1}\left({\mu }_1-{\mu }_2\right)}$ (15)${\displaystyle C=\frac{1}{n_1-n_2}\left(n_1{C}_1-n_2{C}_2\right)}$

Here β shows the model coefficients, C covariance matrices, C mean vectors.

Data distributions

Liver Disorders dataset

Figure 11 shows the data distributions of the Liver Disorders dataset. In Fig. 11, blue and red-colored icons show data for each class. Figure 11A shows the data distribution multiplied by the weights obtained by the PSO algorithm. Similarly, the data distributions were obtained by the BAT algorithm (Fig. 11B), the FPA algorithm in (Fig. 11C), and the GSA algorithm in (Fig. 11D). When the distribution of the weighted data is examined, it is seen that the data approach each other within the class and diverge between the classes. Figure 12 shows the graph of the fit functions calculated by each optimization algorithm in iterations. Figures 12A–12D, respectively, refer to the PSO, BAT, FPA, and GSA algorithms.

Full Chain dataset

Data distributions of the Full Chain dataset are shown in Fig. 13. In Fig. 13, blue and red-colored icons show data for each class. Figure 13A shows the distribution of the data multiplied by the weights obtained by the PSO algorithm. Similarly, the data distributions were obtained by the BAT algorithm in (Fig. 13B), the FPA algorithm in (Fig. 13C), and the GSA algorithm in (Fig. 13D). When the distribution of the weighted data is examined, it is seen that the data approach each other within the class and diverge between the classes. Figure 14 gives the graph of the fit functions calculated by each optimization algorithm in iterations. Figures 14A–14D, respectively, refer to the PSO, BAT, FPA, and GSA algorithms.

Two Spiral dataset

Data distributions of Two Spiral datasets are shown in Fig. 15. In Fig. 15, blue and red-colored icons show data for each class. Figure 15A shows the distribution of the data multiplied by the weights obtained by the PSO algorithm. Similarly, the data distributions were obtained by the BAT algorithm (Fig. 15B), the FPA algorithm (Fig. 15C), and the GSA algorithm (Fig. 15D). When the distribution of the weighted data is examined, it is seen that the data approach each other within the class and diverge between the classes. Figure 16 gives the graph of the fit functions calculated by each optimization algorithm in iterations. Figures 16A–16D, respectively, refer to the PSO, BAT, FPA, and GSA algorithms.

Table 2:

Classification accuracy rate for PSO algorithm (%).

Dataset	Non-Weighted		Our weighted algorithm
	LDA	SVM	Optimization	LDA	SVM
Iris	98.0	97.3	PSO FPA BAT GSA	100.0 100.0 100.0 100.0	100.0 100.0 100.0 100.0
BUPA Liver Disorders	69.0	68.4	PSO FPA BAT GSA	100.0 100.0 100.0 100.0	100.0 100.0 100.0 100.0
Full Chain	66.6	66.9	PSO FPA BAT GSA	82.8 82.8 70.5 82.7	95.2 92.2 70.2 96.4
Two Spiral	64.6	65.3	PSO FPA BAT GSA	79.8 79.3 80.2 79.6	69.2 69.2 71.4 69.5

DOI: 10.7717/peerjcs.1598/table-2

Experimental results

Table 2 gives the accuracy rates for all datasets and weighting processes. As can be seen, when the results obtained were examined, an increase was observed in the performances of all datasets. Especially in the real dataset Iris and the Liver Disorders dataset, a 100% accuracy rate has been reached. Furthermore, accuracy rates exceeding 95% from 65% to −70% have also been achieved in artificial datasets. Therefore, regardless of the algorithm used, it is considered appropriate to determine the weighting values of the proposed mathematical model.