Machine learning techniques for imbalanced multiclass malware classification through adaptive feature selection

Introduction

The internet has become increasingly prevalent in daily life due to advancements in information and communication technology. People of all ages rely on various applications operating on various computing devices to meet their daily needs, such as transportation, health, banking, and retail. The increasing use of apps poses security risks for devices and applications. Malicious software, also known as malware, poses the greatest threat to the cyberworld. Malware programs evolve as technology advances, threatening our security and privacy. Malware is increasingly being used to carry out a variety of destructive operations on victims’ devices. Malware causes anomalies in the functioning of a computer system, raising security concerns. As the number of malware variants proliferates, conventional signature-based malware detection has become less effective and time-consuming. Artificial intelligence techniques, such as machine learning and deep learning, are becoming increasingly popular for studying and analyzing malware behavior to detect it more efficiently. The malicious payload is delivered to the affected device via different routes, including email attachments, advertisements, potentially unwanted apps (PUAs), and free utility software. Malcoders typically obfuscate malware families such as Trojan, Backdoor, Spyware, and Worm, posing serious threats to system vulnerabilities (Yan et al., 2023). According to the Quick Heal annual report FY-2022–23, a total of 163.81 million of malware from the families Trojan, Infector, PUA, Worm, Cryptojacking, Ransomware, and Adware were detected during 2021 and 2022. Figure 1 shows the family-wise malware detection statistics for 2021 and 2022. Figure 2 shows the top five malware from July to September 2022. Furthermore, the report shows a substantial number of instances in each class of malware with an exceptional increase in instances of Trojan and Cryptojacking (QuickHeal, 2022).

Protecting devices from malware is possible by using the time-consuming and expensive statistical methods that are really complex in nature. Malware detection is a computationally hard problem, but the use of statistical methods can restrict undetected spreading (Cohen, 1987). A lot of efforts are being made to limit the spread of malware. In this context, classifying malware against benign and malware against malware is an ongoing need. There are two basic approaches, named static and dynamic, to analyze malware. Static analysis is about studying some key features, like opcode sequences, printable strings, etc., of the malware without executing it. It requires some reverse engineering techniques that can disassemble them. However, there may be challenges with packed malware when performing static analysis on it. On the contrary, dynamic analysis is about collecting execution time features like system call graph, application programming interface (API) sequence, registry file contents, etc. by allowing the malware to execute in a controlled environment. However, the malware may not exhibit its original execution pattern when executed under a controlled environment (Damodaran et al., 2017; Shibahara et al., 2016). Many researchers have applied machine learning techniques to mitigate the challenges of analyzing metamorphic and polymorphic malware variants (Wong & Stamp, 2006). Developing an effective malware detection system is tough, especially when dealing with newer threats. Advancements in virus evasion techniques have led to significant consequences. Advanced malware detection technologies, such as machine learning and deep learning, overcome the limitations of traditional methods (Gopinath & Sethuraman, 2023). The authors examine current malware detection algorithms for Android OS, i-OS, IoT systems, Windows OS, advanced persistent threats (APTs), and ransomware. Many dynamic analysis algorithms referred API call sequences to be a crucial behavioral characteristic that distinguishes malicious programs from benign ones utilizing machine learning and deep learning approaches (Gaber, Ahmed & Janicke, 2024). Windows as an operating system provides APIs for a variety of operations pertaining to files, networks, I/O, and processes. Any program, including malicious software, calls APIs sequentially to do the required tasks. An essential feature set for analyzing program activity is the API sequence. In order to avoid detection, malware in particular is designed to call APIs frequently and sporadically. The analysis of benchmark multiclass malware datasets such as VirusSample, VirusShare (Düzgün et al., 2021), and MAL-API-2019 (Ferhat Ozgur et al., 2020) reveals that the length of the API sequences is either excessively long due to repeated calls or insufficient. The dataset is quite challenging because of the wide variation in API sequence length. The goal of this work is to improve performance metrices with regard to multiclass malware datasets that are imbalanced. The objectives of this work are:

• 1.

  Developing a feature selection strategy to address the issue of feature imbalance related to the diversity of API sequence lengths in malware traces.

• 2.

  Developing a unique adaptive feature selector (AFS) that employs a greedy method to improve the classification metrics for imbalanced multiclass malware classification problems.

• 3.

  Designing a framework called Adaptive Multiclass Malware Classifier (AMMC) to train base machine learning models that use fewer resources, ensuring better classification metrics.

“Related Work” will explore related works. “Methodology” elaborates the working principles of AFS and AMMC. “Experimental Evaluation” discusses the dataset’s description and the experimental setup with results. The concluding observation is highlighted in the ‘Conclusion’.

Related work

Recent malware research has revealed that machine learning and deep learning are often used for malware investigation and categorization. API sequences for malware categorization aim to enhance accuracy and efficiency through advanced deep learning and machine learning methods.

Ye et al. (2008) gathered API call data from PE files and created a verifiable and comprehensible feature set. They developed an intelligent malware detection system based on objective-oriented association (OOA) classification. These features were utilized to determine whether the PE file is malicious or not. They have not predicted different types of malware. Vinod et al. (2010) used dynamic analysis of four metamorphic classes of malware to create signature-based malware fingerprints to trace their API sequences. Their experiment used the API sequences of 80 viruses from four families and 20 harmful apps. They used the Chi-square test to determine how closely a piece of malware matched a given class. This method achieved 75%, 75%, 80%, and 80% accuracy rates for the families NGVCK, IL-SMG, G2 and MPCGEN. They stated that increasing the number of samples will enhance precision. The signature matching approach is susceptible to recently identified malware samples. Ding et al. (2013) suggested an API based association mining strategy that eliminated infrequent items from the API sequences. They selected association rules with good classification abilities and implemented them to increase detection accuracy. For classifying malware into 11 families, Kong & Yan (2013) combined several malware attributes like API calls, registers, opcodes, etc. They developed an effective system that was capable of recognizing previously unidentified samples using discriminant distance metric learning, pairwise graph matching, and ensembled classification (Kong & Yan, 2013).

Mehra, Jain & Uppal (2015) obtained control flow and API call graphs from 600 malware and 150 benign samples. They suggested utilizing Gourmand Feature Selection to extract desired attributes from API call graphs. The WEKA classification tool yielded 89%, 91.08%, 92.24%, 94.56%, and 99.1% accuracy rates for KNN, SMO, VP, NB, and J-48 classifiers, respectively. They used portable executables instead of parsing API sequences generated in runtime. A supervised approach, like Random Forests technique, was used by Pirscoveanu et al. (2015) for the classification of malware. The Cuckoo Sandbox application was used to gather a total of 42,000 malware behaviours. Windows API calls were categorised using DNS requests, files that were accessed, mutexes, and registry key information. Additionally, the labels recognised by the Avast applications outperform the outcomes offered by the VirusTotal service for the classes of harmful software. System call sequences were examined by Kolosnjaji et al. (2016) to classify malware. Convolutional and recurrent network layers were used to extract the best characteristics. Using this hybrid neural network architecture, they were able to average precision of 85.6% and recall of 89.4%. Zhang et al. (2016) proposed a simple ensemble learning algorithm for the classification of malware, utilizing data from the Microsoft malware classification challenge on Kaggle. Malware samples from the unbalanced training dataset were accurately classified into the proper family.

According to recent studies on malware analysis, deep neural networks and supervised and unsupervised machine learning techniques are frequently employed to detect malicious activities with higher accuracy, efficiency, and a lower false positive rate. The primary components of machine learning-based malware detection techniques are automatic detection and feature extraction (Ye et al., 2017). Due to the scalability, speed, and flexibility, machine learning algorithms like logistic regression (LR), support vector machine (SVM), Random Forest (RF), K-nearest neighbor (KNN), frequent pattern mining (FPM), etc. are frequently used to discover and categorize unidentified samples for malware family. Han et al. (2019) applied the TF-IDF technique to 807 benign and 3,027 malicious (including packed and unpacked variants) samples. The MalDAE framework uses machine learning techniques such as KNN, decision tree (DT), extreme gradient boosting (XGBoost), and RF to identify and categorize malware based on static, dynamic, and fused API sequences. Dynamic API sequences outperformed static API sequences with accuracy rates of 74.74%, 79.65%, 83.15%, and 85.96%, respectively. However, with the fused API sequence, accuracy improved to 85.26%, 88.42%, 93.33%, and 94.39%, respectively. Given the difficulty posed by the enormous number of malware kinds, deep learning techniques were suggested to boost efficiency.

Huda et al. (2016) suggested an automated malware detection method to calculate statistical information on API calls performed by malicious and benign programs. They developed two novel hybrid API feature selection methods based on hybrid SVM encapsulation heuristic method and maximum associated or minimal redundancy filtering heuristics approach to determine the best API calls to detect malware. Hidden Markov models (HMMs) were utilized by Damodaran et al. (2017) on their own multiclass malware dataset, and it has been concluded that the dynamic approach based on API calls was quite successful. Ucci, Aniello & Baldoni (2019) stated that a malicious program can be obfuscated as a new program that is mistakenly labelled as benign while retaining the original behaviour and its consequences. The detection process for this new software might be readily avoided. Or-Meir et al. (2019) have stated that code obfuscation technique is less effective on dynamic analysis as compared to static analysis. The heuristic-based malware detection algorithms often transform a malware sample into a basic form, such as text, picture, or signal, before extracting essential aspects for further analysis (Hashemi & Hamzeh, 2019; Mohammadi et al., 2019; Fu et al., 2018). The time needed to modify the sample representation as a system call, opcode sequence, etc. and the time required for feature extraction resulted in substandard performance on both the training and testing phases of the algorithms, even though such methods might achieve decent accuracy. Several researchers have tried to use natural language processing (NLP) techniques on malware static properties such as Opcode sequences and file header sections to identify malicious programs. Using feature extraction techniques like n-grams, term frequency and inverse document frequency (TF-IDF), etc., they discovered low detection accuracy—below 87%. To perform static analysis on the visual malware pictures used in their recurrent neural network, Minhash, Visualization, and convolutional neural network (RMVC) approach, Sun & Qian (2021) used recurrent neural network (RNN) and convolutional neural network (CNN). They found accuracy better than 92%, even with a tiny training dataset. Experimental results show that the classification accuracy of malware classes in the dataset with noise factor is 0.96, higher than the accuracy of 0.83 in a dataset without noise factor. They suggested that a potential avenue for future research was to examine the effectiveness of their approach in dynamic analysis. Malware was categorized using images produced from malware binaries by Hammad et al. (2022). KNN, SVM, and the elaboration likelihood model (ELM) are trained and tested using the Malimg dataset. Features are extracted using GoogleNet, a deep learning model, and Tamuar, a texture feature that correlates with human visual perception. They found that ELM performed better than any other model. They have recommended using data augmentation, which could enhance classification outcomes.

System call sequences from 3,536 benign and 3,567 malicious applications for Android-OS were employed by Xiao et al. (2019) on the long short-term memory (LSTM) model. They discovered a low FPR of 9.3% and a high recall of 96.6%. Two LSTM models make up their classifier: one is used to train the malware, while the other is used to train the trusted application. New sequences were classified using the similarity score that was derived from the two models. Mathew & Ajay Kumara (2020) have employed TF-IDF embeddings with N-grams for feature extraction and selection. Using API call sequences, the proposed LSTM model is used to classify applications as benign or malicious. The authors obtained a 92% accuracy score on unidentified test API call sequences. The feature that represents hidden malicious behavior plays a vital role in malware detection. The insignificant feature leads to poor classification performance. To increase classification performance, eliminate insignificant and noisy features from the dataset. Bhat, Behal & Dutta (2023) suggested a very stable and reliable malware binary classification model based on the ensemble technique. To improve intrusion detection systems’ accuracy, Taheri, Ahmadzadeh & Kharazmi (2015) employed the Cuttlefish algorithm for feature reduction. Four distinct datasets, KDD5000, KDD10000, KDD100000, and KDD500000, were used for selecting various feature numbers, such as 3, 5, 10, 13, and 41. They reported promising results using the dataset and an artificial neural network as the evaluation function. Taheri et al. (2024) have investigated two adversarial techniques, Data Poisoning with Noise Injection (DP-NI) and Gradient-based Data Poisoning (GDP), to assess the vulnerability of deep learning-based Android malware classifiers. They have suggested a novel defense technique called Differential Privacy-Based Noise Clipping (DP-NC) to make Android malware classifiers more resilient to these adversarial attacks. By employing adversarial training techniques and deep neural networks, DP-NC demonstrates remarkable efficacy in mitigating the impact of GDP and DP-NI attacks. Panda & Tripathy (2020) have created a host-based anomaly detection system by using the TF-IDF word embedding approach on the DLL sequence of all in-memory processes to identify abnormal processes. Panda, Bisoyi & Panigrahy (2023) have trained multiple 1D-CNN models using the one-vs.-rest strategy and generated API embeddings using the Word2Vec word embedding technique. They proposed ModifiedSoftVoting to integrate classification capabilities in multiclass malware classification challenges to improve classification metrics.

Ferhat Ozgur et al. (2020) published a dynamic data set named MAL-API-2019 using the Cuckoo Sandbox, comprising 7,107 API sequences for eight different forms of malware. The data set has also been trained to perform multiclass classification using single-layer LSTM, two-layer LSTM, SVM, KNN, RF, and DT. Compared to all other models, the single-layer LSTM achieved macro F1-score of 0.47 as the maximum. Li & Zheng (2021) used LSTM and gated recurrent unit (GRU) models on the benchmark dataset MAL-API-2019 to classify malware classes using long-sequence API calls. They have achieved a precision of 0.56 in both approaches and a recall of 0.58 and 0.59, respectively, in LSTM and GRU. The broad applicability of LSTM and the best architecture for malware classification were examined by Avci, Tekinerdogan & Catal (2023). Using the MAL-API-2019 dataset, they evaluated and compared the performance of several LSTM architectures, such as CNN-LSTM, stacked-LSTM, bi-directional LSTM, and vanilla-LSTM. In order to produce comprehensible results and classification metrics, Galli et al. (2024) integrated the eXplainable artificial intelligence (XAI) technique with an AI-based malware detection process. They evaluated various deep learning models on the dataset MAL-API-2019, BiLSTM and achieved the best experimental classification results, with accuracy, precision, recall, and F1-scores of 52.88%, 54.89%, 53.99%, and 54.31%, respectively. CAFTrans, a framework that processes API sequences using CNN and LSTM networks, was proposed by Qian & Cong (2024). The MAL-API-2019 dataset was used to test the framework, and the results showed an F1-score of 0.65 and an area under the curve (AUC) of 0.89. They asserted that by identifying malware in their families more accurately than in others on the same dataset, CAFTrans improves accuracy.

Düzgün et al. (2021) published two datasets VirusShare and VirusSample. These are created using MD5 malware hashes, with 14,616 samples taken from VirusShare and 9,795 from VirusSample sites, respectively. The malware samples’ API calls are retrieved using the Python module PEfile, which extracts API calls from a program’s Portable Executable (PE) file header. These static analysis techniques result in static API call sequences. They have used the balanced version of these two datasets on several machine learning and deep learning models. On VirusShare with SVC, they achieved the highest macro F1-score of 0.76, whereas on VirusSample, the highest macro F1 the CANINE model is 0.91. Demirkiran et al. (2022) classified malware into multiple classes using the benchmark datasets VirusShare, VirusSamle, and MAL-API-2019. Their suggested RTF model outperforms the Transformer, CANINE-S, and BERT deep learning models. With the RTF model, they have achieved the highest macro F1-scores on the benchmark dataset, which are 0.72, 0.80, and 0.61, respectively. Miao et al. (2024) proposed a lightweight malware detection model based on knowledge distillation (DistillMal) that exhibited superior classification performance in several evaluation metrics when applied to VirusShare and VirusSample datasets. On the VirusSample dataset, DistillMal’s experimental performance metrics showed an accuracy of 0.94, a macro precision of 0.50, a macro recall of 0.73, and a macro F1-score of 0.60. In contrast, VirusShare’s accuracy was 0.89, accompanied by a macro F1-score of 0.69, macro recall of 0.63, and macro precision of 0.88.

The challenge of the imbalanced multiclass malware classification problem is addressed using various strategies, with the API sequence being the most crucial feature sequence. Table 1 summarises works by several other researchers considering API sequence as a critical feature. This study solves the issue of feature imbalance concerning diversity in API sequence length of the multiclass malware classification problem using the unique feature selection technique AFS. As machine learning models use fewer resources to train than deep learning models, the proposed AMMC framework trains base machine learning models using Skip-gram API embeddings to ensure superior classification reports with fewer resources. Using base machine learning models instead of deep learning or other complicated models reduces the higher resource requirements for malware detection.

Methodology

The AMMC workflow is depicted in Fig. 3 as a sequence of stages. This framework takes a labeled imbalanced Multiclass Malware API Sequence Dataset as input. It ensures a better classification result through a novel AFS process that works iteratively. In the Nonconventional API Name Removal stage, each API sequence in the dataset is scanned to drop the nonconventional API names. A nonconventional API name may contain symbols beyond the allowed symbols $[A..Z],[a..z],[\mathrm{0..9}]$, and $[\mathrm{\_}]$. Such API names are found in the case of static API sequence datasets.

API call sequences, in both static and dynamic datasets, tend to be lengthy due to repeated occurrences of identical API calls. A notable characteristic of malware behaviour is the repetition of specific API calls within these sequences to evade detection mechanisms. Each API call corresponds to a machine-level task, and analyzing the sequence of unique calls provides valuable insights into malware activity. The proposed AMMC framework introduces a Duplicate API Removal step to address redundancy in API sequences before AFS. This step ensures the preservation of the first occurrence of each distinct API call while eliminating subsequent duplicates. By focusing on unique API calls, the sequence reflects the underlying machine-level tasks performed by the malware more accurately. For instance, consider an encoded API sequence: [2, 2, 2, 5, 5, 2, 2, 5, 1, 1, 2, 2, 5, 2, 5, 1, 1, 4, 4, 2, 2, 2, 5, 1, 1, 1, 1, 4, 4, 4, 4, 7, 7, 1, 1, 4, 4, 2, 2, 5, 2, 2, 5, 1, 1, 1, 4, 4, 4, 2, 2, 2, 5, 5, 2, 2, 5, 5, 2, 2, 5, 1, 1, 1, 4, 4, 4, 7, 7, 1, 1, 4, 1, 1, 4]. After applying duplicate API removal, the processed sequence becomes: [2, 5, 1, 4, 7]. This step reduces the sequence size and ensures that critical, non-redundant information about the malware’s operations is retained, enhancing the efficiency of subsequent analysis stages.

The AFS stage in the framework works iteratively over three substages. The substage Select Feature by Class selects influential APIs in an API sequence considering the TF-IDF feature weight threshold. During influential feature selection, it mitigates the feature sequence imbalance problem concerning API sequence length. At the same time, it preserves the class-wise sample size. The substage Machine Learning Model trains and tests specific classifiers on the finalized API sequences. Considering the classification report of the model, the substage Greedy Strategy for Updating Feature Weight Threshold updates the feature weights for the following selection of influential features by Select Feature by Class. A new goal is set to obtain better classification results once a desired macro F1-score is reached. The best-found classification report of all iterations is produced if the desired macro F1 is not reached after a predetermined number of iterations.

The AFS adapts changes in feature selection criteria to achieve better performance. The functioning of the AFS is detailed in Algorithm 1. It necessitates three inputs: $F_1^t$, $W_{cnt}$, and $D_{api}$. $F_1^t$ is to be provided as a target for the macro F1-score and is expected to be achieved by AFS. In the event that the maximum macro F1 ( $F_1^{max}$) achieved does not reach the target, the $W_{cnt}$ constant is supplied to end the search for $F_1^t$ instead of going into an infinite loop. $D_{api}$ is a benchmark multiclass malware dataset with n classes. Listed are the data structures in statement 1 through statement 4 used in algorithm AFS to search for $F_1^t$. $F_1^{max}$ is initialized to $0$ and used to keep track of the maximal macro F1-score achieved over the iterations against the set $F_1^t$. $Wai{t}_{cnt}$ is initialized to $W_{cnt}$ and is used to restrict the search process entering into an infinite loop while the targeted $F_1^t$ is not becoming realizable. CFWT[ ] is an array of size n to define the classwise feature weight threshold, and all elements are initialized to 0. CFWT gets updated after each iteration to tighten classwise feature section criteria. $F_1[]$ is an array of size $n+1$ where the first n elements represent F1-scores of n classes and the last element represents macro F1-score of the classifier. The scores in $\mathrm{F}1[]$ are used to keep track of improvements while searching for the $F_1^t$ of the classifier.

AFS iterates around statement 5 to statement 29 in the while loop to find the $F_1^{max}$ score. By choosing the features from dataset $D_{api}$ that meet the corresponding weights for each class in CFWT, SelectFeatureByClass generates the final dataset $D_{api}^F$. TrainTestModel builds the classifier and produces the classification report (cr) using the dataset $D_{api}^F$. $Collect{F}_{1}Score$ creates the F1-score array. Furthermore, the classifier’s macro F1 is compared against $F_1^{max}$ to record any improvement. The classifier’s macro F1-score is compared to the $F_1^t$ in each iteration to determine additional weight adjustments in CFWT. When a class’s F1-score is less than the classifier’s macro F1-score, the feature weight for the classes is incremented by 0.01. $F_1^t$ is incremented by 0.05 to look for a better macro F1-score if the $F_1^{max}$ score reaches the $F_1^t$. Lastly, the $F_1^{max}$ score with the associated cr calculated in addition to the CFWT weights is returned if the $F_1^t$ is not realized for the $W_{cnt}$ number of iterations after achieving a $F_1^{max}$ score.

In order to solve the feature imbalance issue brought on by the variety of API sequence lengths in the $D_{api}$, which poses a challenge to the classifier’s effectiveness, the algorithm AFS calls the subroutine SelectFeatureByClass in statement 6. This subroutine chooses significant APIs from API sequences as outlined in Algorithm 2. The statements of SelectFeatureByClass are explained in detail below.

• Statement 3 selects $AP{I}_{seq}$ from $D_{api}$, where each $d_{c_{id}}$ will hold the API sequences of the $D_{api}$ whose class is $C_{id}$. It divides the $D_{api}$ into several disjoint datasets $d_{c_1},d_{c_2},\dots ,d_{c_{id}}$ where $d_{c_{id}}$ is the API data for class $C_{id}$.

• Statement 4 uses a TF-IDF weighting schema to select APIs in $AP{I}_{Seq}$ that qualify the mentioned weights in CFWT by $C_{id}$. It calculates a weight matrix (i.e., $W_{C_{id}}[][]$) for every $d_{c_{id}}$ in order to address the data imbalance problem associated with the variety of API sequence lengths. The TF-IDF weighting schema is applied to each $d_{c_{id}}$ to decide the weights for each distinct API name concerning the API sequence in $d_{c_{id}}$. TF-IDF is a statistical measure that ensures the importance of a word (i.e., a distinct API) to a document (i.e., $AP{I}_{seq}$) in the collection of documents (i.e., $d_{c_{id}}$). Considering TF-IDF over raw frequencies of occurrences of APIs is to scale down the impact of very frequently occurring APIs in an $AP{I}_{seq}$ in the collection of API sequences, which is empirically less informative than the APIs of less frequency. The weight of ith API to jth API sequence is denoted as $W_{ij}$ in the $W_{C_{id}}[][]$ and defined as given in Eq. (1).

(1) $$W_{ij}=T{F}_{ij}XID{F}_i$$

$T{F}_{ij}$ in Eq. (1) is the term frequency for ith API with respect to the jth API sequence and defined as given in Eq. (2).

(2) $$T{F}_{i}j=\frac{Numberoftimes{\mathbf{i}}^{th}APIappearsin{\mathbf{j}}^{th}APISequence}{TotalnumberofAPIsin{\mathbf{j}}^{th}APIsequence}$$

$ID{F}_i$ in Eq. (1) quantifies i^th API’s infrequency throughout $d_{c_{id}}$. It is determined by taking the logarithm of the ratio between the total number API sequences in $d_{c_{id}}$ and the number of API sequences that contain the i^th API as given in Eq. (3). Penalizing APIs that appear in every API sequence is the aim.

(3) $$ID{F}_i=\log \left(\frac{TotalnumberAPIsequencesin{\mathbf{d}}_{c_{id}}}{NumberofAPIsequencesthatcontainthe{\mathbf{i}}^{th}API}\right)$$

• The finalized $d_{c_{id}}$ is found by iterating over each API sequence and retaining those APIs whose weight as per the $W_{C_{id}}[][]$ qualifies the weight threshold stated in $CFWT[C_{id}]$. This is done using statement 5. If none of the APIs meet the threshold, a preventive measure is also taken to return the sequence to its previous state. This prevention step keeps shorter API sequences from being excluded and preserves the sample size of $d_{c_{id}}$.

• In statement 6, the final dataset $D_{api}^F$ is constructed by combining each of the classwise finalized datasets $d_{c_{id}}$. This final dataset ( $D_{api}^F$) is returned to train and validate a classifier.

AFS employs the TrainTestModel as outlined in Algorithm 3 to ensure several performance metrics by training and evaluating a classifier on $D_{api}^F$. It returns the classification report ( $cr$) to AFS for further improvements in CFWT to meet the $F_1^t$. The steps of TrainTestModel are explained below in detail.

• Statement 1 calculates $f_{wt}[][]$ the feature matrix of API embeddings via the Skip-Gram approach of Word2Vec, a neural network model (Mikolov et al., 2013). Technically, it calculates the likelihood of an API being a context API for a specified target API. This way, it also explores the multicollinearity between APIs, taking their semantic relationship. The output probabilities generated by the network will tell us how likely it is to find each vocabulary API near our input API. Figure 4 displays a shallow network with an input layer, a single hidden layer, and an output layer. The fascinating thing is that we do not employ the trained neural network. Instead, the goal is to learn the hidden layer’s weights while predicting the surrounding APIs. These weights represent API embeddings. The input is a one-hot encoded vector corresponding to the API vocabulary size in $D_{api}^F$, with a specific API being the target API. The output layer is made up of neurons equal to the API vocabulary size, and the Softmax activation function is used to calculate the appropriate probability for each API in the API vocabulary. The hidden layer is linear, and its optimal weights are used to generate the learned API embeddings. For example, API embedding of a target API will be a vector of dimension $[1XN]$, obtained by multiplying $[1XM]$ matrix (the one-hot vector of the target API) with [M X N] matrix (the feature weight matrix $f_{wt}[][]$ at the hidden layer), where ‘M’ is the API vocabulary size and ‘N’ is the number of neurons in the hidden layer.

• In statements 2 and 3, the usual 80:20 split ratio for training and testing data is considered during training and evaluating a model. While training, feature weight vectors are normalized, and hyperparameters are tuned with the grid search, emphasizing regularization in the model.

• Statement 4 calculates the classification report (cr) and returns it to AFS for further processing.

The cr represents several performance metrics such as accuracy, precision, recall, F1-score, and AUC, which ensures the model’s efficacy.

• Accuracy is the measure of a classifier’s ability to make accurate predictions. In balanced classes, accuracy is a valuable metric; nevertheless, it is not a suitable fit in imbalanced classes. Furthermore, the distribution of false positives and negatives is unclear. Accuracy for multiclass classification instances will be calculated using Eq. (4).

(4) $$Accuracy=\frac{\sum _{i=1}^{n}cm[i,j]}{\sum _{i=1}^n\sum _{j=1}^{n}cm[i,j]}$$

• Precision is helpful when false positives pose a greater risk than false negatives. It is helpful for skewed and unbalanced datasets. As more false positives are predicted by the model, the precision falls. The accuracy of a model’s predictions of the target class is guaranteed by its precision. Precision for multiclass classification instances will be calculated using Eq. (5). The classifier’s macro-averaged precision score is calculated using the unweighted mean of each class’s precision score.

(5) $$Precisio{n}_c=\frac{cm[c,c]}{\sum _{i=1}^{n}cm[i,c]}$$

• Recall describes the model’s ability to predict the real positive cases accurately. When a false positive is more concerning than a false negative, recall is a valuable statistic. Recall decreases with the number of false negatives the model predicts. Recall for multiclass classification instances will be calculated using Eq. (6). The classifier’s macro-averaged recall score is calculated using the unweighted mean of each class’s recall score.

(6) $$Recal{l}_c=\frac{cm[c,c]}{\sum _{i=1}^{n}cm[c,i]}$$

• The weighted average of recall and precision is known as the F1-score. Recall and precision must both be strong for the classifier to have a high F1-score. This measure only gives preference to classifiers with comparable recall and precision. F1-score for multiclass classification instances will be calculated using Eq. (7). The classifier’s macro-averaged F1-score is calculated using the unweighted mean of each class’s F1-score.

(7) $$F_{1c}=\frac{2}{\frac{1}{Recal{l}_c}+\frac{1}{Precisio{n}_c}}$$

Table 2 provides an example of AFS demonstrating the modification of the selection criteria and enhancement of the macro F1-score of the SVC model on the VirusShare dataset. The greedy approach in updating the feature selection criteria over iterations is shown in this table. Since the goal is to improve the model’s performance for an imbalanced multiclass problem, the greedy property ensures improvement in the macro F1-score of the model by improving the F1-score of the classes with poor performance. Here, the greedy approach updates CFWT[] while considering the F1[] score array from the model’s cr. The primary step of the technique is only to update a class’s feature selection threshold criteria in CFWT[] when the class’s F1-score is less than or equal to the model’s macro F1-score.

In every iteration, AFS compares $F_1^t$ with the macro F1-score of the model. The initial value of $F_1^t$ for the first iteration in Table 2 is 0.75, which is greater than the model’s macro F1-score of 0.73. The fourth iteration resets $F_1^t$ to 0.80, which is attained in the fifth. Moreover, the model’s $F_1^{max}$ score is determined to be 0.91 in iteration 8, and $F_1^t$ is reset to 0.95. AFS ends after the fourteenth iteration since the model’s macro F1-score has not increased during the previous six iterations. Whenever the model detects a macro F1-score higher than or equal to the most recent known $F_1^{max}$, the parameter $Wai{t}_{count}$ returns to its initial value, and $F_1^{max}$ gets updated. When the macro F1-score does not exceed the most recent $F_1^{max}$, the $Wai{t}_{count}$ decreases to prevent AFS from nonterminating. After the fourteenth iteration, the search for a higher macro F1-score ended since the parameter $Wai{t}_{count}$, which has an initial value of 5, expired. Finally, AFS returned the $F_1^{max}$ of 0.91, CFWT array [0.07,0.08,0.0,0.06,0.0,0.0,0.0,0.08] of eight classes and iteration number 8 representing the classification report of the iteration with the maximum macro F1-score.

Experimental evaluation

This section highlights the datasets used to evaluate the proposed framework in terms of its efficacy with several performance metrics. The experimental work for the described AMMC is carried out using an Intel(R) Core(TM) i5-1035G1 CPU @ 1.00GHz X 8 PC with 16 GB of RAM, Ubuntu-22.04.5 LTS, and Python 3.9.

Result analysis

Three fundamental machine learning models, RF, KNN, and SVC, are employed for training and testing in order to guarantee the goal of validating AMMC with models that use fewer computational resources. KNN is used as one of the supervised non-linear machine learning models on vector distance logic, RF is used as an implicit ensemble model, and SVC is used as a candidate linear and non-linear model that can operate on diverse datasets. Additionally, ECLF is employed as an ensemble classifier, with the above three being estimators based on soft voting.

The hyperparameters (WindowSize=10, VectorSize=200) are set through a random search to estimate API weight vectors using the Skip-gram model. Furthermore, the API weight vectors are normalized using the Standard Scalar after considering multiple normalization techniques such as MinMax, Robust, and Standard Scalar. Through the grid search technique, emphasizing regularization, the hyperparameters for SVC, RF, and KNN are determined to be (gamma=‘auto’, kernel=‘rbf’, C=10, max_iter=10000), (n_estimators=200, criteria = ‘entropy’, bootstrap=True, max_depth=1000), and (n_neighbors=8, weights=‘distance’, algorithm=‘auto’, leaf_size=30, p=2, metric=‘minkowski’), respectively.

An ablation study is carried out on the three benchmark datasets by dropping the stage Select Feature by Class and the Greedy Strategy for updating feature weights in AMMC to ensure the impact of AFS. This experiment on AMMC without AFS uses the API sequences generated by the Duplicate API Removal stage to train and validate classifiers. The mean and standard deviation of 10-fold cross-validation results for macro precision, macro recall, macro F1-score, and accuracy are displayed in Tables 4A, 5A and 6A respectively.

Table 4:

Ten-fold cross validation performance measures of AMMC for VirusShare.

	(A) Without AFS				(B) With AFS ( ${\mathit{F}}_t^1=0.95$ and Wcnt = 5)
	SVC	RF	KNN	ECLF	SVC	RF	KNN	ECLF
Precision	0.88 ± 0.01	0.90 ± 0.01	0.84 ± 0.02	0.89 ± 0.03	0.94 ± 0.01	0.97 ± 0.00	0.93 ± 0.01	0.95 ± 0.01
Recall	0.69 ± 0.00	0.68 ± 0.01	0.71 ± 0.02	0.64 ± 0.03	0.87 ± 0.01	0.87 ± 0.01	0.88 ± 0.00	0.81 ± 0.03
F1	0.73 ± 0.00	0.73 ± 0.00	0.75 ± 0.01	0.70 ± 0.02	0.90 ± 0.01	0.92 ± 0.01	0.91 ± 0.00	0.87 ± 0.02
Accuracy	0.90 ± 0.00	0.90 ± 0.01	0.90 ± 0.00	0.90 ± 0.01	0.95 ± 0.00	0.95 ± 0.00	0.94 ± 0.00	0.94 ± 0.01

DOI: 10.7717/peerj-cs.2752/table-4

Table 5:

Ten-fold cross validation performance measures of AMMC for VirusSample.

	(A) Without AFS				(B) With AFS ( ${\mathit{F}}_t^1=0.95$ and Wcnt = 5)
	SVC	RF	KNN	ECLF	SVC	RF	KNN	ECLF
Precision	0.94 ± 0.03	0.93 ± 0.07	0.88 ± 0.04	0.93 ± 0.04	0.95 ± 0.02	0.97 ± 0.01	0.95 ± 0.01	0.96 ± 0.01
Recall	0.71 ± 0.02	0.72 ± 0.03	0.75 ± 0.05	0.72 ± 0.02	0.87 ± 0.03	0.87 ± 0.03	0.86 ± 0.03	0.87 ± 0.03
F1	0.76 ± 0.02	0.78 ± 0.03	0.78 ± 0.02	0.76 ± 0.02	0.90 ± 0.02	0.92 ± 0.02	0.90 ± 0.02	0.91 ± 0.02
Accuracy	0.94 ± 0.01	0.95 ± 0.00	0.94 ± 0.01	0.94 ± 0.01	0.96 ± 0.01	0.96 ± 0.01	0.96 ± 0.01	0.96 ± 0.00

DOI: 10.7717/peerj-cs.2752/table-5

Table 6:

Ten-fold cross validation performance measures of AMMC for MAL-API-2019.

	(A) Without AFS				(B) With AFS ( ${\mathit{F}}_t^1=0.85$ and Wcnt = 15)
	SVC	RF	KNN	ECLF	SVC	RF	KNN	ECLF
Precision	0.65 ± 0.01	0.66 ± 0.01	0.62 ± 0.01	0.66 ± 0.01	0.81 ± 0.02	0.79 ± 0.01	0.73 ± 0.02	0.80 ± 0.01
Recall	0.65 ± 0.01	0.64 ± 0.01	0.62 ± 0.01	0.65 ± 0.01	0.80 ± 0.02	0.77 ± 0.01	0.74 ± 0.01	0.79 ± 0.01
F1	0.65 ± 0.01	0.65 ± 0.01	0.61 ± 0.01	0.65 ± 0.01	0.81 ± 0.02	0.78 ± 0.01	0.73 ± 0.01	0.79 ± 0.01
Accuracy	0.64 ± 0.01	0.64 ± 0.01	0.61 ± 0.01	0.64 ± 0.01	0.81 ± 0.02	0.77 ± 0.01	0.73 ± 0.01	0.79 ± 0.01

DOI: 10.7717/peerj-cs.2752/table-6

To validate AMMC’s efficacy, we reran the experiment on the three benchmark datasets with AFS. The AFS-Pruned dataset is obtained via SelectFeatureByClass. To stress consistency in performance indicators, each model undergoes a 10-fold cross-validation. AFS goes through multiple iterations in each fold to get the maximal macro F1-score ( $F_1^{max}$) by training and evaluating the basic machine learning and ensemble models outlined before. Compared to the macro F1-score without AFS, a higher macro F1 target ( $F_1^t$) and $Wai{t}_{cnt}$ are gradually adjusted to seek a superior macro F1-score with AFS. Tables 4B, 5B, and 6B exhibit the 10-fold cross-validation findings, including the mean and standard deviation of maximal macro F1-scores for each fold. Furthermore, the target macro F1 ( $F_1^t$) and the related $Wai{t}_{cnt}$ are superscribed in the Tables 4B, 5B, and 6B, together with the macro precision, recall, and accuracy. Considering all the classifier’s performance, the impact of AMMC with AFS in contrast to AMMC without AFS is well justified by showing significant improvement with the several performance metrics outlined in Tables 4–6.

Figures 5–7 depicts the reduction in API sequence length achieved by selecting APIs that meet the weight criterion via SelectFeatureByClass for all models on all datasets during the fold, with the highest macro F1-score with the previously indicated $macro{F}_1^t$. In the dataset found after Duplicate API Removal stage, there were 493 API sequences of length greater than 300 for VirusShare, as shown in Fig. 5. However, using SelectFeatureByClass in AFS lowered the number of API sequences with lengths >=300 from 493 to 429, 415, 420, and 415, respectively, for the evaluated models. As a result, the number of API sequences increases in the range >=100&<150, >=60&<80, >=40&<60, >=20&<40, and >0&<20, respectively. It is observed that in the dataset formed after Duplicate API Removal stage, longer API sequences are becoming shorter and shifting toward the lower-length API sequences. Figures 6 and 7 show similar observations for the VirusSample and MAL-API-2019 datasets, respectively. With the reduced API sequences referring to the mean accuracy, precision, recall, and F1-scores for all three datasets highlighted in Tables 4–6 the effect of AFS is well established.

Figure 8 depicts the process of iteratively searching superior macro F1 at the fold with the highest macro F1 for all models using VirusShare. We stopped our experiment at $F_1^t=0.95$ and $W_{cnt}=5$. In the eighth iteration, both SVC and KNN achieved the $F_1^{max}$ score of 0.91 and ended the quest for a higher macro F1-score in iteration 14. However, RF and ECLF achieved the $F_1^{max}$ score of 0.92 and 0.91 at iteration 9 and ended their pursuit for a higher macro F1-score in iteration 15. Similarly, Figs. 9 and 10 depict iteratively searching for superior macro F1 at the fold with the highest macro F1 for all models using VirusSample and MAL-API-2019.

Table 7 highlights the maximum macro F1-score and macro ROC-AUC achieved on VirusShare for all the models with their respective fold and iteration in that fold. One statistical method used to assess a model’s ability to predict multiclass classifications is the Matthews correlation coefficient (MCC). It is employed in machine learning to assess how well predictions are made. On unbalanced datasets, it is more trustworthy than accuracy and F1-score, which might be deceptive. Because it considers the balance of the four confusion matrix entries (TP, TN, FP, and FN), it is more informative than accuracy and F1-score. Since class sizes vary in all the datasets, the MCC scores for each model are also shown in this table to guarantee a fair evaluation. Furthermore, the mean time to detect (MTTD) in milliseconds shows the detection time efficiency in this multiclass problem. During the search for the $F_1^{max}$ score via AFS, the F1-score for classes Ransomware, Trojan, and Virus always stands higher than the macro F1-score of the model. Hence, SelectFeatureByClass method does not make any changes to their API sequences. However, for all other classes, the API selection in a sequence is done by comparing the respective class weight threshold in CFWT via SelectFeatureByClass. The contents of CFWT for the iteration with the maximum macro F1-score are also shown in the table. Tables 8 and 9 represent the CFWT, MCC, and MTTD details of the fold and iteration with maximum macro F1 concerning datasets VirusSample and MAL-API-2019.

Table 7:

Maximal performance metrics of VirusShare.

Bold values indicate the best performance.

Models	Fold	Iteration	Feature weight threshold array CFWT [1..n] for maximal macro F1								Macro F1	Macro ROC-AUC	MCC	MTTD (ms)
			Adware	Agent	Backdoor	Downloader	Ransomware	Trojan	Virus	Worms				Test size 2,770
SVC	5	8	0.06	0.07	0.00	0.06	0.00	0.00	0.00	0.07	0.91	0.98	0.90	0.330
RF	1	9	0.06	0.07	0.02	0.07	0.00	0.00	0.00	0.08	0.92	0.99	0.91	0.030
KNN	2	8	0.06	0.07	0.00	0.05	0.00	0.00	0.00	0.07	0.91	0.97	0.90	0.247
ECLF	8	9	0.07	0.07	0.01	0.06	0.00	0.00	0.00	0.08	0.91	0.99	0.91	0.760

DOI: 10.7717/peerj-cs.2752/table-7

Table 8:

Maximal performance metrics of VirusSample.

Bold values indicate the best performance.

Models	Fold	Iteration	Feature weight threshold array CFWT [1..n] for maximal macro F1						Macro F1	Macro ROC-AUC	MCC	MTTD (ms)
			Adware	Agent	Backdoor	Trojan	Virus	Worms				Test size 1,947
SVC	2	20	0.11	0.06	0.00	0.00	0.00	0.18	0.94	0.98	0.94	0.146
RF	2	14	0.07	0.06	0.00	0.00	0.00	0.13	0.94	0.99	0.94	0.024
KNN	3	10	0.09	0.06	0.00	0.00	0.00	0.09	0.92	0.98	0.92	0.186
ECLF	2	23	0.08	0.06	0.00	0.00	0.00	0.22	0.94	0.99	0.94	0.701

DOI: 10.7717/peerj-cs.2752/table-8

Table 9:

Maximal performance metrics of MAL-API-2019.

Bold values indicate the best performance.

Models	Fold	Iteration	Feature weight threshold array CFWT[1..n] for maximal macro F1								Macro F1	Macro ROC-AUC	MCC	MTTD (ms)
			Adware	Backdoor	Downloader	Dropper	Spyware	Trojan	Virus	Worms				Test size 1,422
SVC	9	24	0.07	0.12	0.03	0.10	0.23	0.22	0.00	0.14	0.84	0.98	0.82	1.143
RF	7	22	0.03	0.14	0.05	0.12	0.21	0.21	0.00	0.08	0.80	0.97	0.77	0.040
KNN	9	24	0.05	0.14	0.07	0.12	0.23	0.23	0.00	0.08	0.75	0.94	0.74	0.361
ECLF	9	21	0.02	0.14	0.04	0.12	0.20	0.20	0.00	0.08	0.82	0.98	0.79	1.872

DOI: 10.7717/peerj-cs.2752/table-9

Figures 11 and 12 show the confusion matrix and ROC curve with AUC scores for all the models on VirusShare of the fold with maximum macro F1-score. Similarly, Figs. 13 and 14 for VirusSample and Figs. 15 and 16 for MAL-API-2019 show the confusion matrix and ROC curve with AUC scores for all the models of the fold with maximum macro F1-score. Considering the principal diagonal of all three classifiers’ confusion matrix, their classification ability is well justified. For all three datasets, the AUC scores of the ROC curve indicate the prominence in performance for multiclass malware classification problems on all the models using AMMC with AFS.

Case study

During the experiment, the case study on different malware classes of the three benchmark datasets revealed notable findings on their possible detection rate. As all the datasets are imbalanced, the precision-recall curves are plotted to emphasize all the classifiers’ positive class prediction performance. The identification of Spyware and Trojans is more complicated than that of the other six malware classes: Adware, Backdoor, Downloader, Dropper, Virus, and Worms in the dynamic API sequence-based unbalanced dataset MAL-API-2019 even though they have a good number of samples. However, even with a smaller sample size for Adware, all the classifiers showed a better detection rate than Spyware and Trojan. The precision-recall curves for the classifiers SVC, RF, KNN, and ECLF are plotted in Fig. 17, and all the classifiers agree with lower precision-recall AUC scores for Trojan and spyware in contrast to other classes.

Identifying the class Worms is more complicated than all other classes in the static API sequence-based imbalanced datasets VirusShare and VirusSample. The detection rates of Adware, Agent, and Backdoor are found to be relatively higher than that of Worms with the VirusSample dataset, even though their sample sizes are nearly equal. In the VirusShare dataset, the detection rate of Worms and Agent is found to be lesser than that of all other classes. For both VirusShare and VirusSample, due to the presence of a higher number of samples for Virus and Trojan, the detection rate is very high compared to others in all the classifiers. The precision-recall curves plotted in Figs. 18 and 19 also show lower precision-recall AUC scores for the class Worms on all the classifiers with VirusShare and VirusSample datasets, respectively.

Limitations and future work

In this work, the iterative feature selection technique AFS goes through many iterations to find a better feature weight. Hence, the additional time of AFS may be an overhead in terms of the total model training time. AFS employs a greedy technique to search weights to select influential APIs in an API sequence. AFS terminates after a fixed number of iterations when there is no evidence of performance improvement. Deciding on the terminating iteration depends on the dataset.

Future work will be devoted to exploring the usefulness of other optimization techniques in contrast to the greedy approach in AFS for a better classification score. The selection of optimal weights through AFS from one of the best-performing base machine learning models can be applied to deep learning techniques. It may highlight better classification abilities in the case of multiclass malware classification problems. A recursive feature selection technique to select influential APIs from API sequences for multiclass malware datasets may be contrasted with the proposed iterative feature selection technique AFS.

Conclusion

This research presents an AMMC framework on variable length API sequences for efficient imbalanced multiclass malware classification. The primary goal has been to develop a technique for adaptively selecting features (i.e., APIs) by class using AFS and demonstrate its applicability and usefulness in multiclass malware classification situations. The greedy search approach used by AFS is fundamental to this work. The described studies on three open multiclass malware datasets, VirusSahre, VirusSample, and MAL-API-2019, demonstrated, as stated in the findings section, the efficacy of AMMC employing AFS. AMMC ensures macro F1 values of 0.92, 0.94, and 0.84 for VirusSahre, VirusSample, and MAL-API-2019, with macro ROC-AUC scores of 0.99, 0.99 and 0.98, respectively. The comparison of AMMC’s performance with the results of other authors on the same datasets reveals considerable improvements. Furthermore, the results of all of the base machine learning models and the ensemble model outlined in Tables 7–9 in this research performed better with AMMC than the results of others shown in Tables 10–12. Exploring other strategies to ensure a better selection of APIs in varying length API sequences on API sequence-based malware datasets will increase detection efficiency even more.

Supplemental Information