Using machine learning-based binary classifiers for predicting organizational members’ user satisfaction with collaboration software

Previous literature on predictive research using machine learning

Machine learning-based binary classifier is a method that classifies a set of elements into two categories using classification rules (Read et al., 2021). In the context of artificial intelligence, more and more scholars have conducted research related to prediction using machine learning-based binary classifiers (Khandani, Kim & Lo, 2010; Fu, Sawang & Sun, 2019; Dastile, Celik & Potsane, 2020; Yoo & Rho, 2020; Ho, Cheong & Weldon, 2021; Cocco, Tonelli & Marchesi, 2021; Naeem et al., 2021; Park, Kwon & Jeong, 2023). For instance, doctors predict the health of diabetes and cancer patients using random forest classifiers and NB classifiers. Practitioners use case datasets and similar disease features to classify and predict future patient health (Fu, Sawang & Sun, 2019). Moreover, scholars use machine learning-based binary classifiers to perform statistics and construct prediction methods to provide appropriate strategies for combating and managing the spread of epidemics like COVID-19 (Naeem et al., 2021). In the financial industry, people can predict Bitcoin prices through a framework based on a machine learning-based binary classifier and provide trading strategies for industry practitioners (Cocco, Tonelli & Marchesi, 2021). Bank lenders can use machine learning to analyze data from past loan officers and build credit risk prediction models. Predictive models can determine future loan applicants’ repayment ability and help banks decide whether to lend and reduce losses (Khandani, Kim & Lo, 2010; Dastile, Celik & Potsane, 2020). In education-related research, machine learning was used to construct a method to predict teacher job satisfaction and student satisfaction in remote learning during the COVID-19 pandemic (Yoo & Rho, 2020; Ho, Cheong & Weldon, 2021).

Overall, machine learning-based binary classifiers are highly feasible and superior for prediction due to their ability to use both categorical and numerical predictors and evaluate the importance of each predictor.

Data source

This study utilized publicly available data from the “Smart Work Fact-Finding Report 2020” conducted by the Korea National Information Society Agency. The report (Approval Number: NIA III-RSE-A-20010) is based on an online survey of 1,900 employees from September 1 to September 30, 2020, across 17 metropolitan cities and regions in South Korea. The survey questionnaire covered a range of topics, including perceptions of smart work, the status of smart work, the work environment, the effects of smart work, obstacles to smart work, government support measures, and respondent information. Table 2 provides an overview of the questionnaire’s content.

The data used in this study are publicly available and can be accessed by anyone who applies. The Korea National Information Society Agency provides users with reusable information and allows for both for-profit and non-profit use. Data used in this study can be obtained by applying at (https://www.nia.or.kr/site/nia_kor/03/10303040200002016092710.jsp).

Since the focus of this study is on predicting user satisfaction with collaboration software, only data related to practitioners who use collaboration software for smart work was used for predictive analysis, according to the type of smart work in the public data. The dataset included a total of 1,002 observations related to the use of collaboration software. The demographic characteristics of the participants are presented in Table 3.

Predictor variable

In this study, we used public data that consisted of typical Likert items with five response options: strongly disagree, disagree, neither agree nor disagree, agree, and strongly agree. To improve prediction accuracy, we discretized the predictor variable of user satisfaction. This variable is derived from the question item “How satisfied are you with the collaboration software you used?” in Table 2, PART D.

We created a discrete predictor variable from the mean of user satisfaction, assigning values of 0 and 1 to represent low and high user satisfaction, respectively. Out of the 1,002 participants who used collaboration software, 401 (40.02%) reported low user satisfaction, while 601 (59.98%) reported high user satisfaction. Discretization is a crucial step in machine learning-based research, as it simplifies data representation and enhances understanding (Liu et al., 2002; Tsai & Chen, 2019).

Explanatory variable

To enhance prediction accuracy, we employed XGBoost’s feature importance algorithm to identify the most crucial explanatory variables for our predictive model. Feature importance algorithms enable the elimination of redundant and irrelevant features, thereby improving the performance of classifiers (Jiang et al., 2022).

This specific process is depicted in the first part of Fig. 1. We initially removed variables related to personally identifiable information and missing data from the original 1,002 datasets, yielding 44 explanatory variables. Subsequently, we utilized XGBoost’s algorithm to calculate the importance scores of these 44 explanatory variables, enabling us to pinpoint the most critical factors influencing prediction outcomes. Feature importance is assessed by the information gain before and after splitting a DT based on a particular feature. The more a feature can reduce the uncertainty in predicting the target variable, the greater its importance (Li et al., 2021). From the 44 explanatory variables, we selected the top 10 based on their importance scores for subsequent prediction and classification accuracy computations.

Prediction model

To identify the most effective binary classifier for predicting user satisfaction with collaboration software, we compared the accuracy of several mainstream machine learning-based binary classifiers. The classifiers evaluated in this study included the NB, LR, XGBoost, SVM, KNN, and DT. By analyzing the performance of these classifiers, we were able to determine the most accurate and effective approach for predicting user satisfaction with collaboration software.

The data analysis phase of the research process, depicted in Fig. 1, outlines the accuracy prediction process. We divided the original data into a 70% training set and a 30% test set. The binary classifier first trains using the training set, and then it analyzes the accuracy using the test set.

In previous studies, researchers employed various accuracy metrics such as accuracy, precision, recall, and F1-score, depending on their experimental objectives. Although metrics for judging classifier performance are a crucial issue in machine learning, there is no broad consensus on a unified standard (Chicco & Jurman, 2020). In this study, we use accuracy, the most popular metric in binary classifier tasks, to judge the performance of the binary classifiers. Accuracy is the most intuitive performance measure, and it is simply a ratio of correctly predicted observations to the total observations (Mali et al., 2022). The formula for accuracy is shown below.

$$\left(\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}\right)={\displaystyle \frac{TP+TN}{TP+FN+FP+TN}}$$

where:

– True positives (TP) are the instances that were correctly classified as positive by the model.

– True negatives (TN) are the instances that were correctly classified as negative by the model.

– False positives (FP) are the instances that were incorrectly classified as positive by the model when they were actually negative.

– False negatives (FN) are the instances that were incorrectly classified as negative by the model when they were actually positive.

The accuracy rate ranges from 0 to 1, where 0 indicates that none of the instances were classified correctly, and 1 indicates that all instances were classified correctly. A higher accuracy rate indicates better predictive performance of the model.

NB classifier

The NB classifier is a widely adopted machine learning technique rooted in Bayes’ theorem, used primarily for classification problems. Despite its simple nature, it excels in handling extensive datasets and has been successfully applied in various fields such as text classification, spam detection, and sentiment analysis. The term “naive” stems from the algorithm’s core assumption that all features are conditionally independent of each other given the class label—an assumption that may not always hold true in real-world scenarios (Sun, Li & Fan, 2021).

Nonetheless, the naive Bayes classifier has consistently demonstrated strong performance in diverse forecasting and prediction tasks, thanks to its straightforward nature, computational efficiency, and ease of implementation. The algorithm computes the posterior probability of each class label considering the feature values and subsequently assigns the instance to the class with the highest probability. In specialized forecasting processes, the naive Bayes classifier can offer valuable predictions by employing its probabilistic approach to estimate the likelihood of various outcomes based on available data.

In conclusion, the naive Bayes classifier is an effective and resource-efficient machine learning algorithm for tackling classification and forecasting challenges. While it relies on the simplifying assumption of feature independence, it has proven to be remarkably effective in a range of applications, such as text classification, spam filtering, and sentiment analysis. The algorithm’s probabilistic nature enables it to make reliable predictions in specialized forecasting processes by gauging the likelihood of different outcomes according to the input data.

The formula for Bayes’ theorem is:

$$\mathrm{P}\left(\mathrm{C}|\mathrm{X}\right)={\displaystyle \frac{\mathrm{P}\left(\mathrm{C}\right)\mathrm{P}\left(\mathrm{X}|\mathrm{C}\right)}{\mathrm{P}\left(\mathrm{X}\right)}}$$where:

$\mathrm{P}(\mathrm{C}|\mathrm{X})$ is the posterior probability of class C given the feature set X.

$\mathrm{P}(\mathrm{X}|\mathrm{C})$ is the likelihood of observing feature set X given class C.

P(C) is the prior probability of class C.

P(X) is the probability of observing the feature set X.

In the naive Bayes classifier, the assumption is made that all features are conditionally independent given the class label. Thus, the likelihood term $\mathrm{P}(\mathrm{X}|\mathrm{C})$ can be calculated as the product of the probabilities of each feature given the class:

$$\mathrm{P}(\mathrm{X}|\mathrm{C})=\mathrm{P}(\mathrm{x}1|\mathrm{C})\ast \mathrm{P}(\mathrm{x}2|\mathrm{C})\ast \dots \ast \mathrm{P}(\mathrm{x}\mathrm{n}|\mathrm{C})$$

For each class, the classifier calculates the posterior probability $\mathrm{P}(\mathrm{C}|\mathrm{X})$ and assigns the instance to the class with the highest probability.

LR

LR is a widely used statistical method and machine learning algorithm for predicting the probability of an event occurring based on one or more predictor variables. It is particularly suitable for binary classification problems, where the outcome has two possible classes. The technique has been employed in various fields, including medicine, social sciences, and economics, to model the relationship between a binary dependent variable and one or more independent variables, which can be either continuous or categorical (Friedman, Hastie & Tibshirani, 2000; Lever, Krzywinski & Altman, 2016).

The primary concept behind LR is to model the probability of the event of interest (e.g., class 1) by fitting a logistic function to the predictor variables. The logistic function, also known as the sigmoid function, maps any real-valued input to a probability value between 0 and 1. The logistic function is given by:

$$\mathrm{P}(\mathrm{Y}=1|\mathrm{X})=1/(1+{\mathrm{e}}^{(-\mathrm{z})})$$where:

– $\mathrm{P}(\mathrm{Y}=1|\mathrm{X})$ is the probability of the event of interest (class 1) given the predictor variables X.

– z is the linear combination of predictor variables, represented as $\mathrm{z}=\mathrm{\beta }0+\mathrm{\beta }1\times 1+\mathrm{\beta }2\times 2+\dots +\mathrm{\beta }\mathrm{n}\times \mathrm{n}$.

– β0, β1, …, βn are the regression coefficients that need to be estimated.

The estimation of the regression coefficients is typically done using the maximum likelihood estimation (MLE) method. Once the coefficients are estimated, the logistic function can be used to predict the probability of the event of interest for new instances.

In summary, LR is a widely employed statistical method and machine learning algorithm for binary classification problems that models the probability of an event using a logistic function. The algorithm estimates the regression coefficients by fitting the logistic function to the predictor variables and uses the resulting function to predict the probability of the event of interest for new instances.

XGBoost

XGBoost, short for eXtreme gradient boosting, is a powerful and efficient machine learning algorithm designed for solving classification, regression, and ranking problems. It is an extension of the gradient boosting algorithm, which combines the strengths of multiple weak learners, typically decision trees, to create a more accurate and robust model (Atef, Elzanfaly & Ouf, 2022). XGBoost has gained widespread popularity due to its superior performance, scalability, and versatility across a variety of domains, including finance, healthcare, and natural language processing.

The core principle behind XGBoost is the iterative process of building a strong learner by optimizing an objective function that comprises a loss function and a regularization term. The loss function measures the difference between the predicted and true outcomes, while the regularization term prevents overfitting by penalizing complex models. XGBoost employs gradient boosting, which is an additive model that updates the weak learners by minimizing the loss function using gradient descent.

The specific forecasting process using XGBoost involves the following steps:

• 1) Initialize the model with a constant value or a simple model that minimizes the objective function. This serves as the base model.

• 2) For each iteration (t = 1, 2, …, T):

  1. Compute the gradient and Hessian of the objective function with respect to the current model's predictions. These values indicate the direction and magnitude of change needed to minimize the objective function.

  2. Build a new decision tree to fit the gradient and Hessian, which approximates the optimal structure for minimizing the objective function.

  3. Determine the optimal step size (learning rate) and update the model by combining the base model and the new decision tree.

  4. Regularize the model to control complexity and prevent overfitting.

• 3) Combine the base model with the results from all iterations to produce the final prediction.

The XGBoost algorithm can be represented mathematically using the following formula: ${\mathrm{F}}_{\mathrm{t}}(\mathrm{x})={\mathrm{F}}_{\mathrm{t}-1}(\mathrm{x})+\mathrm{\eta }\ast {\mathrm{h}}_{\mathrm{t}}(\mathrm{x})$

Here, ${\mathrm{F}}_{\mathrm{t}}(\mathrm{x})$ denotes the prediction at iteration t, ${\mathrm{F}}_{\mathrm{t}-1}(\mathrm{x})$ represents the previous prediction, $\mathrm{\eta }$ is the learning rate, and $h_t(x)$ is the new decision tree that fits the gradient and Hessian.

In conclusion, XGBoost is a powerful and versatile machine learning algorithm with numerous advantages, including efficiency, scalability, and the ability to handle missing values. By incorporating gradient boosting techniques, regularization, and a specific forecasting process, XGBoost can effectively address various prediction tasks.

SVM

SVM is a widely used supervised machine learning algorithm designed for classification and regression tasks. Its decision boundary is the maximum-margin hyperplane that solves the learning sample (Fu, Sawang & Sun, 2019). It has been successfully applied in various domains, such as bioinformatics, finance, and image recognition, due to its ability to handle linear and non-linear problems efficiently and accurately.

The key idea behind SVM is to find the optimal hyperplane that separates the data points of different classes with the maximum margin. The margin is defined as the distance between the hyperplane and the closest data points from each class, known as support vectors. A larger margin signifies a better separation between classes, resulting in a more accurate and robust model.

For linearly separable data, SVM constructs a linear hyperplane that can perfectly separate the classes. However, in cases where the data is not linearly separable, SVM employs the kernel trick to transform the input data into a higher-dimensional space, where a linear separation is possible. Commonly used kernel functions include linear, polynomial, radial basis function (RBF), and sigmoid kernels.

SVM is known for its robustness against overfitting, especially in high-dimensional spaces, as it only considers support vectors for model construction. Additionally, the algorithm allows for fine-tuning of model complexity through the use of hyperparameters, such as the cost parameter C and the kernel-specific parameters.

In summary, support vector machine is a versatile and efficient supervised machine learning algorithm suitable for classification and regression tasks. By finding the optimal hyperplane that maximizes the margin between classes, SVM can handle linear and non-linear problems effectively. Its robustness against overfitting and flexibility in adjusting model complexity through hyperparameters make it a popular choice among data scientists and machine learning practitioners.

The primary optimization problem for SVM in its primal form can be written as:

$$\begin{array}{ll}& \mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}1/2{\parallel \mathrm{w}\parallel }^2+\mathrm{C}\sum \left({\xi }_{\mathrm{i}}\right)\\ & \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}{\mathrm{y}}_{\mathrm{i}}(\mathrm{w}\cdot {\mathrm{x}}_{\mathrm{i}}+\mathrm{b})\ge 1-{\xi }_{\mathrm{i}}\mathrm{a}\mathrm{n}\mathrm{d}{\xi }_{\mathrm{i}}\ge 0\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{i}=1,\dots ,\mathrm{n}\end{array}$$where:

– w is the weight vector, which is orthogonal to the hyperplane.

– ${\mathrm{x}}_{\mathrm{i}}$ are the data points.

– ${\mathrm{y}}_{\mathrm{i}}$ are the class labels, either −1 or 1.

– b is the bias term.

– ${\xi }_{\mathrm{i}}$ are the slack variables, which allow for some misclassification in non-linearly separable cases.

– C is the cost parameter, a user-defined parameter that controls the trade-off between maximizing the margin and minimizing the classification error.

In the dual form, the optimization problem becomes:

$\begin{array}{ll}& \text{maximize}{\displaystyle \sum \left({\mathrm{\alpha }}_{\text{i}}\right)}-1/2{\displaystyle \sum \left({\displaystyle \sum \left({\mathrm{\alpha }}_{\text{i}}{\mathrm{\alpha }}_{\mathrm{j}}{\mathrm{y}}_{\mathrm{i}}{\text{y}}_{\mathrm{j}}\mathrm{K}\left({\mathrm{x}}_{\mathrm{i}},{\mathrm{y}}_{\mathrm{i}}\right)\right)}\right)}\\ & \mathrm{subject}\mathrm{ }\mathrm{to}\mathrm{ }0\le {\mathrm{\alpha }}_{\text{i}}\le \mathrm{C}\mathrm{for}\mathrm{i}=1,...,\mathrm{n}\mathrm{and}{\displaystyle \sum \left({\mathrm{\alpha }}_{\text{i}}{\mathrm{y}}_{\mathrm{i}}\right)}=0\end{array}$where:

– ${\alpha }_{\mathrm{i}}$ are the Lagrange multipliers.

– $\mathrm{K}({\mathrm{x}}_{\mathrm{i}},{\mathrm{x}}_{\mathrm{j}})$ is the kernel function that maps the input data into a higher-dimensional space.

KNN

The KNN algorithm is a theoretically mature method and one of the simplest machine learning algorithms. The idea of this method is: in the feature space, if most of the recent k samples near a sample belong to a specific category, then the sample also belongs to this category (Liu et al., 2019). KNN operates on the assumption that similar data points are more likely to belong to the same class or have similar output values. Its primary advantage lies in its ability to adapt to the underlying data distribution, making it suitable for a wide range of applications.

The KNN algorithm consists of the following steps:

• 1) Choose the number of neighbors, k.

• 2) Calculate the distance between a query point and all data points in the training set. Common distance measures include Euclidean, Manhattan, and Minkowski distances.

• 3) Select the k nearest neighbors to the query point based on the calculated distances.

• 4) For classification, assign the class label that occurs most frequently among the k neighbors to the query point. For regression, assign the average output value of the k neighbors to the query point.

The distance between two data points ${\mathrm{x}}_{\mathrm{i}}$ and ${\mathrm{x}}_{\mathrm{j}}$ can be calculated using the Euclidean distance formula:

$$\mathrm{d}({\mathrm{x}}_{\mathrm{i}},{\mathrm{x}}_{\mathrm{j}})=\sqrt{\sum \left({({\mathrm{x}}_{\mathrm{i}}-{\mathrm{x}}_{\mathrm{j}})}^2\right)}$$where:

– $\mathrm{d}({\mathrm{x}}_{\mathrm{i}},{\mathrm{x}}_{\mathrm{j}})$ is the Euclidean distance between points ${\mathrm{x}}_{\mathrm{i}}$ and ${\mathrm{x}}_{\mathrm{j}}$.

– ${\mathrm{x}}_{\mathrm{i}}$ and ${\mathrm{x}}_{\mathrm{j}}$ are data points in a multi-dimensional feature space.

In summary, the K-nearest neighbors algorithm is a straightforward and flexible supervised learning method used for classification and regression. It relies on the principle that similar data points are more likely to have similar output values and calculates the distance between data points to identify the k closest neighbors. The output for a query point is determined by the majority class label or average output value of its k nearest neighbors.

DT

DT are a popular and interpretable supervised learning method used for classification and regression tasks. DT recursively split the input feature space into regions based on feature values, ultimately leading to a predicted class or continuous output value at the leaf nodes.

It is a graphical method that uses probability analysis intuitively. Supervised learning is given several samples, each with attributes and a category. These categories are determined in advance. Then a classifier is obtained through learning. This classifier can give the object the correct classification (Moreno-Bote & Mastrogiuseppe, 2021). The DT is easy to understand and implement. It can directly reflect the characteristics of the data. If it is explained, it can understand the meaning expressed by the DT. For DT, data preparation is often uncomplicated or unnecessary and can be done simultaneously. Works with both datatype and general-type attributes to produce feasible and effective results on significant data sources in a relatively short period (Patel & Prajapati, 2018; Charbuty & Abdulazeez, 2021; Lee, Cheang & Moslehpour, 2022).

The construction of a DT involves the following steps:

• 1) Choose a feature and a split point to create a decision node, which will partition the data into two subsets.

• 2) Calculate the impurity (e.g., Gini impurity, entropy) for each subset.

• 3) Select the feature and split point that result in the largest impurity reduction.

• 4) Recursively repeat steps 1–3 for each subset until a stopping criterion is met, such as a maximum depth, a minimum number of samples per leaf, or an insignificant impurity reduction.

The impurity reduction can be calculated using the following formula:

$$\mathrm{I}\mathrm{m}\mathrm{p}\mathrm{u}\mathrm{r}\mathrm{i}\mathrm{t}\mathrm{y}\mathrm{R}\mathrm{e}\mathrm{d}\mathrm{u}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}=\mathrm{I}(\mathrm{p}\mathrm{a}\mathrm{r}\mathrm{e}\mathrm{n}\mathrm{t})-\sum (({\mathrm{n}}_{\mathrm{c}\mathrm{h}\mathrm{i}\mathrm{l}\mathrm{d}}/{\mathrm{n}}_{\mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}})\ast \mathrm{I}(\mathrm{c}\mathrm{h}\mathrm{i}\mathrm{l}\mathrm{d}))$$where:

– I(parent) is the impurity of the parent node.

– ${\mathrm{n}}_{\mathrm{c}\mathrm{h}\mathrm{i}\mathrm{l}\mathrm{d}}$ is the number of samples in the child node.

– ${\mathrm{n}}_{\mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}}$ is the total number of samples in the parent node.

– I(child) is the impurity of the child node.

– The summation is over all child nodes.

In summary, decision trees are a widely-used supervised learning method for classification and regression tasks. They recursively partition the feature space based on feature values, leading to predicted outputs at the leaf nodes. Decision trees are valued for their interpretability and ease of implementation, making them a popular choice for various applications.

Feature importance results

In this study, we first used the XGBoost feature importance algorithm to obtain scores for all indicators. Figure 2 is the bar chart of the feature importance scores of all features. We then selected the top 10 indicators based on their feature importance scores for the subsequent prediction work. Figure 3 shows a bar chart of the feature importance scores for the 10 indicators, while Table 4 provides detailed information about these indicators. The article classifies the 10 indicators with the highest feature importance scores into four predictable dimensions for predicting user satisfaction with collaboration software. The details are as follows, Institutional and guidance (E2_1, E2_5, E2_6), ICT environment (C1_1, E2_4), Company culture (C1_2, C1_4, A2_3), and demographics (SQ2, F6). The high feature importance scores of the indicators at these levels demonstrate their criticality in predicting employee satisfaction with collaboration software.

Prediction accuracy results

In this study, we focused on relevant data related to critical indicators presented in Fig. 3 to evaluate the prediction accuracy of various machine learning-based binary classifiers. The prediction accuracy of these classifiers is presented in Table 5. Our analysis indicated that the NB classifier achieved the highest prediction accuracy of 0.780, followed by LR (0.767), XGBoost (0.744), SVM (0.744), KNN (0.704), and DT (0.637). Therefore, the NB classifier is the preferred predictive model for employee satisfaction with collaboration software in this study. This result further demonstrates the numerous advantages of NB classifiers in predicting panel data. For instance, it is simple and efficient, capable of handling missing and noisy data, and exhibits good interpretability.

Limitations and future research

While this study has provided insights into business management and forecasting, it is important to acknowledge that the timing of the data collection may present some limitations. The data were collected during the COVID-19 pandemic when many people were working remotely and using collaboration software to prevent the spread of the virus. This unique situation may have influenced user satisfaction with the software. As we move towards a post-pandemic era, it is possible that important indicators for predicting user satisfaction with collaboration software may shift, making it necessary to continue collecting relevant data and conducting predictive analyses.