Evaluation of predictive maintenance efficiency with the comparison of machine learning models in machining production process in brake industry

Gaps in the literature

Conspicuous by the lack of data diversity and widespread applicability, most studies use narrow datasets based on a single site or facility, leading to uncertainty of performance in new conditions (Abbas, Al-haideri & Bashikh, 2019; Gougam et al., 2024; Amihai et al., 2018; Borgi et al., 2017; Tambake et al., 2024; Paszkiewicz et al., 2023; Merkt, 2019). Most existing IoT prototypes and ML studies use only a single model (e.g., artificial neural networks (ANN) or simple regression) and lack comprehensive benchmark and hyperparameter optimization (Justus & Kanagachidambaresan, 2022; Kannadaguli, 2020; Gougam et al., 2024; Abbas, Al-haideri & Bashikh, 2019; Taş, 2018). Many studies use only training accuracy or a single hold-out split, while cross-validation, discrete test set and comprehensive metric comparisons are often neglected (Aydin & Guldamlasioglu, 2017; Abbas, Al-haideri & Bashikh, 2019; Selvaraj & Min, 2023; Aydın et al., 2021). The developed prognostic curves and classification results often remain in laboratory conditions and are not integrated into real-time systems (Amihai et al., 2018; Amruthnath & Gupta, 2018; Çekik & Turan, 2025; Soylemezoglu, Jagannathan & Saygin, 2010; Abu-Samah et al., 2015; Kannadaguli, 2020; Shearer, 2000). Most studies only report technical performance and do not provide infrastructure recommendations for the integration of results into decision support modules or similar platforms and active use by maintenance teams (Kasiviswanathan et al., 2024; Çekik & Turan, 2025; Selvaraj & Min, 2023). Existing studies mostly collect data using different IoT-based sensors and methods (Nikfar, Bitencourt & Mykoniatis, 2022; Soylu et al., 2022; Yeardley et al., 2022), but rarely present a unified, standardized protocol or end-to-end data flow infrastructure. There has been no comprehensive study focusing on changes in model performance results by comparing ML models with changing ratios of learning and test data in existing research. Following the research, the question “Can the efficiency of predictive maintenance applications be increased by using different ML algorithms based on sensor data?” was determined as the research question.

In this work, we collect 30,622 CNC sensor and operating system records in real-time with the MTConnect protocol and generate a balanced and meaningful feature set with undersampling and SelectKBest after extensive preprocessing steps such as missing data cleaning, outlier suppression, and categorical transformations. We maximize the generalizability and reliability of the models by benchmarking nine classical machine learning algorithms (decision tree (DT), naive Bayes (NB), k-nearest neighbors (KNN), support vector machine (SVM), Adaptive Boosting (AdaBoost), random forest (RF), Categorical Boosting (CatBoost), Extreme Gradient Boosting (XGBoost) and Light Gradient Boosting Machine (LightGBM)) and an artificial neural network with manual hyperparameter optimization, both with 80–20% holdout and 10-fold cross-validation. Finally, by streaming live data with MTConnect Agent and integrating the results into the decision support module, we not only provide a highly accurate classification, but also an end-to-end solution that supports real-time maintenance decisions.

Amihai et al. (2018) and Borgi et al. (2017) each worked with limited data from a single plant or 155 profiles, which posed a risk of generalizability. This study used a richer and more diverse database with 30,622 samples collected from different sensor types (temperature, oil levels, spindle speed data, axis load, etc.) and strengthened the data quality with outlier removal and feature selection steps.

In conclusion, this study highlights how ML, feature engineering and advanced learning paradigms can directly contribute to Industry 4.0 goals. By transforming the brake component production line into a data-driven maintenance environment, the study provides practical insights into the integration of these technologies into smart factories.

In conclusion, this study highlights how ML, feature engineering and advanced learning paradigms can directly contribute to Industry 4.0 goals. By transforming the brake component production line into a data-driven maintenance environment, the study provides practical insights into the integration of these technologies into smart factories.

Understanding the business

The Mazak CNC machine is used for high-precision brake lifting part production. It operates through automated processes including design, material placement, and precise cutting, leveraging advanced motion systems.

The production begins with part design using CAD software and G-code programming. Raw materials are securely placed for machining, and the CNC machine follows programmed paths with precision, ensuring optimal speed, depth, and accuracy. After machining, quality inspections are conducted before shipment.

The machine is equipped with sensors that monitor critical parameters like oil and coolant levels, motor temperatures, and X, Y, Z positions, recording data in real-time for performance optimization.

Data collection

The MTConnect data collection process integrates multiple components to standardize and transmit data from Mazak CNC machines:

Mazatrol is the control system of Mazak CNC machines that creates processing programs, controls machine movements, and performs CNC operations. Adapter serves as an intermediary, connecting the Mazatrol control system to an MTConnect-compatible data collection system. MTConnect Agent receives data from the adapter, processes it into an MTConnect-compatible XML format, and facilitates data transfer to other systems. Client Software allows users, such as operators and managers, to access and monitor real-time MTConnect data. The data flow starts with Mazatrol operations transmitted via an API to the adapter using TCP. The adapter collects this data and forwards it to the MTConnect Agent, where it is converted into a standardized XML format. Finally, the data are transmitted to a user-accessible environment via HTTP. The MTConnect data transfer process is shown in Fig. 2.

Data set

Data recording from Mazak CNC machines occurs from two different areas. One method consists of data automatically generated from the operating system. The other data recording method involves sensors connected to the device. Table 1 lists the variables associated with the operating system and their explanations.

Table 2 shows the variables recorded by the sensors and their descriptions. The relevant sensor parts consist of level measurement and temperature sensors. These sensors were installed in the device by the company.

The data collected through MTConnect is used in comma-seperated values (CSV) format. The dataset consists of 22 variables, including 16 continuous variables, six categorical variables, one time variable and 30,622 records in the CSV document created. Continuous variables represent sensor readings such as grease oil level, spindle speed, and electric panel temperature. Categorical variables include descriptors such as System Status (the target variable), Spindle Condition, and Emergency Status. The time variable, DateTime, provides timestamps for each observation. This data is instrumental in capturing real-time operational details like temperature, oil levels, load conditions on axes, and overall machine status. The dataset records 18 distinct system states under the system status variable, which reflect the operational conditions of the machinery. These states are categorized into two main groups: normal state and faults or warnings. The normal state accounts for 16,201 observations, representing approximately 65% of the dataset, and indicates the machine is functioning without any issues. Faults and warnings are categorized based on their frequency and type. Frequent faults include Warning: Memory Protection, observed in 193 records (~0.78%), and Warning: Invalid Format, appearing in 176 records (~0.71%). Rare faults, such as Fault: Overload, Warning: Soft Limit, and Warning: Collision Warning, occur in fewer than 10 observations each. Fault types are further classified into overloads, memory-related faults, invalid format errors, and emergency status warnings. The relevant records consist of data between 27.02.2023 and 05.05.2023. As can be seen in Table 3, there are two types of data structures in the data set: categorical and numerical. Table 4 also shows the number of filled values in each data set.

The variable representing the system status is designated as the target variable and represents the main element that the model is trying to predict. This critical parameter describes the system’s current or future performance or state. All other variables are considered independent variables that influence the target variable.

Correlation analysis was conducted to examine the relationship between the variables, and the results are visualized in Fig. 3. Dark red indicates negative correlation, while dark blue represents a positive correlation, with darker the colors signifying stronger relationships. For example, a strong positive correlation is observed between M51 Motor Temperature and Bench Transformer Temperature, shown by a dark blue shade in the Fig. 3.

The results of the correlation analysis between the numerical variables in the dataset are visualized in a heat map. Correlation coefficients range between −1 and 1; where values near 1 indicate a strong positive correlation, and values near −1 indicate strong negative correlation. Variables with a coefficient of 0 have no significant relationship.

The analysis revealsa moderate positive correlation between axis loading (e.g., X Axis Loading, Y Axis Loading) and Spindle Temperature and Spindle Speed (e.g., 0.64 between Y Axis Loading and Spindle Temperature). This indicates that an increase in axis loading can have an impact on spindle temperature and speed. However, low or non-significant correlations were found between some variables.

In particular, Anamil Coolant Oil Level, Apparatus Hydraulic Oil Level and Hydraulic Oil Level were left blank in the correlation matrix because these variables contain constant values and therefore their correlations cannot be calculated. It is foreseen that these fixed variables will make a limited contribution to the modeling process.

Following the correlation table indicating elevated values among most entities, a deeper analysis of the underlying causes and consequences of these findings was conducted. The correlation table derived from the dataset reveals that the correlation coefficients among the majority of items are 1.00, signifying perfect linear relationships. The outcome was further investigated to ascertain its underlying reasons.

Initially, variables including X Axis Loading, Y Axis Loading, and Z Axis Loading demonstrate substantial correlations owing to their restricted variability and analogous patterns across data. These variables probably denote interconnected physical processes or quantify overlapping phenomena, leading to markedly comparable behavior. Furthermore, it was noted that variables such as Spindle Loading and Spindle Speed exhibit a substantial correlation with axis loads, indicating a functional or deterministic relationship intrinsic to the examined system. This is anticipated in systems where mechanical or physical interdependencies link variables.

These relationships were also revealed by preprocessing. All missing values were removed during data purification, removing imputation artifacts. This method improved data integrity but may have increased predictable patterns by reducing noise and variability. Despite perfect correlations, similarity measures like cosine similarity may vary due to magnitude or distribution differences. Variables with similar trends but different magnitudes can have high correlation but low similarity. This shows that correlation measures linear relationships, while similarity indices measure other data patterns.

Data preprocessing

To ensure accurate and trustworthy ML models, the dataset was preprocessed to treat missing values, manage outliers, and resolve class imbalance, a major concern in this work.

As shown in Table 5, missing values occurred primarily due to internet outages during data recording. The Emergency Status variable had the highest number of missing entries. After identifying rows with missing values, the dataset was reduced from 30,622 to 19,472 records by removing 11,150 incomplete rows. Following this process, all columns were verified to contain no missing values.

Certain variables, such as Grease Oil Level, Anamil Coolant Oil Level, Apparatus Hydraulic Oil Level, and Hydraulic Oil Level, consistently contained only zero values. This issue was communicated to the relevant company.

Outliers, which can distort model performance, were identified and corrected to ensure data reliability. Figure 4 illustrates the distributions of numerical variables prior to outlier processing.

For outlier detection, values outside the 0.05 and 0.95 range of each variable’s distribution were identified and adjusted to the nearest limit. This process effectively eliminated all outliers in the dataset.

The system state, used as the target variable, represent the class frequencies, constitutes the target variable in the data set used. The class frequencies, which are detailed in Table 4.

In line with the discussions with the company, normal values represent the optimal operating conditions of the machine, while other values indicate the need for maintenance. Two models were created: one for all system frequencies and another grouping system status as 1 for normal values and 0 for others, using feature engineering.

Data normalization uniformizes information with different scales, reducing model accuracy issues in ML and statistical analysis. This study used min-max normalization to adjust data to 0–1. This method normalized all dataset values.

The dataset used in this study has a class imbalance, with 16,201 “Normal” observations and 8,648 “Fault” observations. This discrepancy could prejudice ML algorithms, lowering minority class performance. We randomly selected 8,648 “Normal” observations to equal the “Fault” class count using undersampling. This change ensured fair class representation, improving model learning. The goal variable was divided into two groups: “Normal” (1, optimal operation) and “Fault” (0, fault conditions). This binary classification optimized modeling by distinguishing normal and incorrect scenarios. The final balanced dataset has 8,648 observations per class after these modifications, providing a solid training and assessment base.

Feature selection was performed to eliminate unnecessary or ineffective variables and identify those that improve model performance. The feature selection process was completed by selecting five variables to be modeled with the SelectKBest method. Related variables are DateTime, Spindle Temperature, Animal Coolant Oil Temperature, M51 Motor Temperature, Bench Transformer Temperature.

Data preprocessing steps, including missing data analysis, normalization, and outlier managementwere applied to optimize the data set for ML algorithms. Various algorithm were used and compared during modeling, including DT, NB, kNN, SVM, Adaptive Boosting (AdaBoost), RF, CatBoost, XGBoost and LightGBM.

During the training and evaluation of the model, train-test split and k-fold cross-validation methods were used to improve model performance. Cross-validation played a critical role in evaluating the overall performance of the model and minimizing problems such as overfitting.

A two-stage approach was adopted for the target variable. In the first stage, the model treated the target variable was simplified into two categories: “Normal” values (1) and “Abnormal” values (0), redefining the problem as binary classification. This dual approach was used to evaluate classification challenges at different levels and determine the most effective strategy.

The examination of the dataset indicated a substantial class imbalance: 16,201 instances in the “Normal” class and 8,648 in the “Abnormal” class. To rectify this, undersampling was implemented, decreasing the “Normal” class to 8,648 observations to align with the “Abnormal” class count. This produced a balanced dataset including 8,648 observations per class, so ensuring rigorous training and assessment.

Binary vs multiclass classification

Binary classification distinguishes between “Normal” and “Others,” whereas multiclass classification encompasses all error classes. Training accuracy stabilizes about epoch 30, with validation accuracy soon thereafter, indicating that binary classification acquires knowledge rapidly and generalizes effectively. Multiclass classification initially exhibits lower accuracy and a more significant disparity between training and validation accuracy, suggesting generalization challenges associated with task complexity.

Imbalanced vs balanced data

Class imbalance significantly impacts model efficacy. The model rapidly memorizes the majority class in the presence of imbalanced data, resulting in diminished validation accuracy. Conversely, balanced data consistently enhances training and validation accuracy, hence boosting generalization and mitigating the limitations of unbalanced datasets.

Standard vs optimized model

Standard models with default hyperparameters exhibit overfitting and demonstrate sluggish learning, resulting in a disparity between training and validation accuracy. As the validation accuracy aligns with the training accuracy, improved models exhibit accelerated learning and enhanced generalization. This demonstrates how hyperparameter tuning enhances model performance.