A Kullback-Liebler divergence-based representation algorithm for malware detection

Feature extraction

Several studies (Fuyong & Tiezhu, 2017; Zhang et al., 2019; Li et al., 2020b; Yang & Liu, 2020) have suggested the N-gram technique for obtaining N-length or sub-text features from the initial text. From the API sequences of every evasive malicious software sample and benign sample in the database, we extract 2-gram features. Because malware must make several API calls rather than just individual API calls in order to perform malicious activities (Pektaş & Acarman, 2017), we chose the API sequence-based N-gram features. As a result, the extracted API sequence-based N-gram features can be beneficial in understanding and modelling malicious behaviour.

Feature representation

In this subsection, the Kullback-Liebler Divergence-based Term Frequency-Probability Class Distribution (KLD-based TF-PCD) algorithm is proposed to represent the extracted features, including the features that occur in both benign and malware classes, using accurate weight vectors. The drawback of employing the traditional TF-IDF technique is that the distribution of each extracted feature in benign and malware classes is neglected when computing the IDF term. Such neglect leads to calculation of inaccurate IDF values for the sharing features, and thus the overall generated weights are negatively affected by IDF values.

Moreover, mimicking legitimate behavior is a widely used strategy by todays malware (Bulazel & Yener, 2017; Alaeiyan, Parsa & Conti, 2019; Or-Meir et al., 2019; Afianian, Niksefat & Sadeghiyan, 2019; Mills & Legg, 2020; Amer, El-Sappagh & Hu, 2020). In addition, evasion behaviors have been observed in both benign and malicious classes (Galloro et al., 2022). Applying the traditional TF-IDF technique to represent the features related to the above-mentioned behaviors will introduce small IDF values, which cause low TF-IDF weights. Therefore, these features can be ignored in the case of using a feature selection technique or using them with close weights, in which the classification model is confused and provides high false positive and negative rates in case feature selection technique is not utilized. This study supposes that these kinds of features can be meaningful features if their probability distributions in both benign and malware classes are exploited since benign and malware classes exhibited evasion behaviors in different probability distributions (Maffia et al., 2021).

The KLD-based TF-PCD algorithm tackles this issue by calculating the probability distributions of each feature in both benign and malware classes and then identifying the difference between the probability distribution of every feature in the benign class and the probability distribution of every feature in the malware class using the Kullback-Liebler Divergence tool. The general formula applied to calculate KLD-based TF-PCD weights is shown in Eq. (2). (2)${\displaystyle \text{W}\left(n\text{\_}{\text{gram}}_i^j\right)=\text{TF}\left(n\text{\_}{\text{gram}}_i^j\right)\ast \log \left(-1\ast \mathrm{KLD}\left(\text{PMD}\left(n\text{\_}{\text{gram}}_i^j\right),\text{PBD}\left(n\text{\_}{\text{gram}}_i^j\right)\right)\right)}$

where w ($n-{\text{gram}}_i^j$) represents the calculated weight of the ith n-gram in malware instance j, TF ($n-{\text{gram}}_i^j$) represents the term frequency, or how many times the n − gram_i is invoked by instance j. TF is calculated using the formula illustrated in Eq. (3). The probability distribution of $n-{\text{gram}}_i^j$ in malware class and the probability distribution of $n-{\text{gram}}_i^j$in benign class are denoted by $\text{PMD}\left(n-{\text{gram}}_i^j\right)$ and $\text{PBD}\left(n-{\text{gram}}_i^j\right)$, respectively. PMD and PBD are computed using the formulas described in Eqs. (4) and 5, respectively. KLD refers to the Kullback-Liebler Divergence tool that measures the difference between the probability distribution of $n-{\text{gram}}_i^j$ in malware and benign classes. KLD is calculated using the formula shown in Eq. (6). (3)${\displaystyle \text{TF}\left(n\text{\_}{\text{gram}}_i^j\right)=\frac{\text{count}\left(n\text{\_}{\text{gram}}_i^j\right)}{\sum _k\text{count}\left(n\text{\_}{\text{gram}}_k^j\right)}}$ where TF ($n-{\text{gram}}_i^j$) is the frequency of n − gram_i in malware instance j, count ($n-{\text{gram}}_i^j$) refers to the number of times n − gram_i occurs in malware instances j, and ${\sum }_k\text{count}\left(n-{\text{gram}}_k^j\right)$stands for the total number of all n-grams in malware instance j. (4)${\displaystyle \text{PMD}\left(n-{\text{gram}}_i^j\right)=\frac{\text{count}\text{\_}\mathrm{M}\left(i,j\right)}{N\left(m\right)}}$ (5)${\displaystyle \text{PBD}\left(n-{\text{gram}}_i^j\right)=\frac{\text{count}\text{\_}\mathrm{B}\left(i,j\right)}{N\left(b\right)}}$ where $\text{PMD}\left(n-{\text{gram}}_i^j\right)$ and $\text{PBD}\left(n-{\text{gram}}_i^j\right)$ refer to the probability distribution of $n-{\text{gram}}_i^j$ in malware class and benign class, respectively. count_M(i, j), count_B(i, j) represent the number of malware samples that $n-{\text{gram}}_i^j$ is appeared in and the number of benign samples that contain $n-{\text{gram}}_i^j$ , respectively, while N(m) stands for the total number of malware samples, N(b) denotes the total number of benign samples. (6)${\displaystyle \text{KLD}\left(\mathrm{PMD}\left(n-{\text{gram}}_i^j\right),\mathrm{PBD}\left(n-{\text{gram}}_i^j\right)\right)=\text{PMD}\left(n-{\text{gram}}_i^j\right)\ast log2\left(\frac{\text{PMD}\left(n-{\text{gram}}_i^j\right)}{\text{PBD}\left(n-{\text{gram}}_i^j\right)}+1\right)}$ where $\text{KLD}\left(\text{PMD}\left(n-{\text{gram}}_i^j\right),\text{PBD}\left(n-{\text{gram}}_i^j\right)\right)$ refers to the difference in the probability distributions of $n-{\text{gram}}_i^j$ in malware class and benign class, $\text{PMD}\left(n-{\text{gram}}_i^j\right)$ means the probability distribution of $n-{\text{gram}}_i^j$ in malware class, $\text{PBD}\left(n-{\text{gram}}_i^j\right)$ is the probability distribution of $n-{\text{gram}}_i^j$ in the benign class, the constant 1 is added to prevent the difference value to be zero value.

Unlike traditional and developed TF-IDF utilized by existing malware classification solutions, the KLD-based TF-PCF technique used the differences between the probability distribution of the concerned feature in malware class and the probability distribution of that feature in benign class instead of IDF values.

IDF can be more useful in the natural language processing field (NLP) when a text classification task is required. Our assumption is that the text classification task in the NLP field differs from the text classification task in the malware classification field. Since the texts in the NLP field belong to human languages, the words that frequently appear in all the documents, such as the, for, of, and others, are not meaningful words because they frequently occur to give the same meanings for all the documents, while the texts in the malware classification field were not originally human languages. Otherwise, these texts belong to malware and benign behaviors. Therefore, the generalization of the concept of IDF on behavioral-based text classification without considering how differently those behaviors have appeared in malware and benign classes can reduce the discrimination of those behaviors.

Evaluation

This study utilized a variety of machine learning techniques that are extensively employed in the literature, including k-nearest neighbor (KNN), regression trees (CART), Naive Bayes (NB), support vector machine (SVM), artificial neural network (ANN), random forest (RF), logistic regression (LR), and eXtreme Gradient Boosting (XGBoost). Furthermore, 40% of the constructed dataset is randomly reserved for testing purposes as unseen data in order to evaluate these machine learning classifiers, and the remaining 60% has been used to train the chosen machine learning models using the 10-fold-cross validation method. To ensure that all classifiers are evaluated using the exact same data when we compare our suggested model with the relevant work, we used the simple random sampling method to split the dataset into train and unseen test data. Figure 3 shows the training and test stages.

Experiment environment setup

The experiments have been carried out under the environment with specifications as follows: Windows 10 Pro 21H2 is installed on a PC equipped with an Intel(R) Core(TM) i7-4790 CPU running at 3.60 GHz and 16.0 GB of memory size. Python (3.9) libraries using Spyder editor version 5 have been utilized to implement feature extraction, feature representation, and classification techniques that have been developed in this study.

Dataset

We dynamically obtained API call sequences that represent behaviors of 7208 evasive malware collected from Galloro et al. (2022) and Kirat & Vigna (2015) as well as 3848 benign samples collected from Wei et al. (2021) and the freshly installed Windows 7 operating system to evaluate our suggested algorithm. We established our dataset with this number of samples based on the acceptance number we gained from the literature review in the community of malware detection, which mostly ranged between 1000 and 5000 samples (Nunes et al., 2019; Sihwail et al., 2019; Zhang et al., 2019; Kakisim, Nar & Sogukpinar, 2020; Yoo et al., 2021; Nunes et al., 2022; Finder, Sheetrit & Nissim, 2022). The majority of authors either did not make their used datasets accessible or provided broken URL connections (Amer, El-Sappagh & Hu, 2020). Consequently, establishing datasets for evasive malware and benign software was certainly not an easy operation. Our experiments in this work are carried out using the API call sequences dataset that was generated using malware and benign samples mentioned above which have been used in our earlier work (Aboaoja et al., 2023). The contents of the used dataset are displayed in Table 2.

Performance measures

Several performance validation measures, such as the False Positive Rate (FP), False Negative Rate (FN), Accuracy (ACC), Detection Rate (DR), Precision (P), and F-measures (F1), have been applied to assess the effectiveness of the proposed algorithm. Further, FN depicts the proportion of malware classified as legitimate, whereas FP shows the rate of benign classified as malware (Darshan & Jaidhar, 2020; Arslan, 2021). While the detection rate (DR)/Recall calculates the percentage of malicious samples that are accurately classified using Eq. (7), the accuracy (ACC) assesses the fraction of samples that are effectively recognized using Eq. (8) (Rostamy et al., 2015). Moreover, Precision (P) uses Eq. (9) to calculate the ratio of malware samples that are estimated to be a malware across all malicious samples and benign samples that are estimated to be malware. Using Eq. (10), F-measure (F1) sums the mean values of Precision and Detection rate (Rostamy et al., 2015).

(7)${\displaystyle \text{DR}=\frac{\text{TP}}{\text{TP}+\text{FN}}}$ (8)${\displaystyle \text{ACC}=\frac{\text{TP}+\text{TN}}{\text{Tp}+\text{TN}+\text{FP}+\text{FN}}}$ (9)${\displaystyle \text{P}=\frac{\text{TP}}{\text{TP}+\text{FP}}}$ (10)${\displaystyle \text{F}1=\frac{2\text{*Precision*Recall}}{\text{Precision}+\text{Recall}}.}$

Experimental results

To validate the performance of the proposed Kullback-Liebler Divergence-based Term Frequency-Probability Class Distribution (KLD-based TF-PCD) algorithm, it has been applied on a dataset in which the API call sequence-based 2-gram features were stored to represent the features in the form of a weights-based vector for each instance in the dataset. Furthermore, the dataset has been divided into 60% as training data and 40% as unseen test data using a simple random sampling method. 60% of the dataset instead of 80% or 90% is selected to be the percentage of the training data to guarantee the robustness of the developed model because each improvement in the obtained accuracy means more effectiveness of the proposed model against unseen data points, which have further potential to examine the proposed model as long as the test data size is large. Additionally, the training data was utilized to train multiple machine learning techniques, K-Nearest Neighbor (KNN), Regression Trees (CART), Naive Bayes (NB), Support Vector Machine (SVM), Artificial Neural Networks (ANN), Random Forest (RF), Logistic Regression (LR), and eXtreme Gradient Boosting (XGBoost). To evaluate how each classifier was learned from the weight vectors that were generated using the proposed KLD-based (TF-PCD) algorithm, the test data was utilized to measure the classification performance of each classifier.

Even though the imbalanced dataset is considered one of the problem domains in the malware detection community, it is not our focus in this work. Additionally, the related work compared with the proposed algorithm was trained using imbalanced datasets. However, to estimate the negative effect of an imbalanced dataset, we applied the SMOTE technique to generate balanced training data. The results, using unseen test data, show that the model that trained based on the balanced dataset decreases the accuracy obtained by only 0.004%. The interpretation of the decrement in the accuracy may be referred to as the SMOTE has generated more noises and overlapped features leading to a decrease the accuracy. However, an in-depth investigation is required to study the impact of an imbalanced dataset on classification performance. Such investigation has been left for future work.

Table 3 displays the performance results of the developed classifiers which were trained based on weight vectors generated using the proposed KLD-based(TF-PCD) algorithm. The classifiers’ accuracy rates varied from 0.794 for NB to 0.972 for XGBoost. Likewise, the F1 rates ranged from 0.823 for NB to 0.978 for XGBoost. The results illustrate that FPR and FNR values dropped together to be 0.037 and 0.023, respectively, with XGBoost. Further, the highest FPR is performed by KNN with 0.130, while NB achieved the highest FNR of 0.269. Regarding the DR, XGBoost introduced the highest value with 0.977, whereas the lowest DR of 0.914 was provided by KNN. For P, the classifiers achieved P rates that ranged between 0.930 for KNN and 0.978 for XGBoost.

Figures 4, 5, 6, 7, 8 and 9 present the comparison of the proposed KLD-based TF-PCD with the traditional TF-IDF, which was widely used in the state-of-the-art (Belaoued et al., 2019; Ali et al., 2020; Li et al., 2020a) and the DF-IDF that was developed by Xue et al. (2019). In terms of accuracy and F-measure, the proposed KLD-based TF-PCD was higher than other techniques for all the classifiers. Furthermore, most of the classifiers with the proposed KLD-based TF-PCD achieved FPR, FNR, DR, and P higher than the traditional TF-IDF and DF-IDF techniques. The best performance compared to all the classifiers was achieved by the XGBoost classifier, which was trained using the weight vectors generated using the proposed KLD-based TF-PCD algorithm.

Significance test

To evaluate the significance of the proposed KLD-based TF-PCD algorithm statistically against the traditional TF-IDF and DF-IDF techniques based on the classification accuracy, the paired t-test has been carried out with the help 10-fold—cross-validation method using the XGBoost classifier. (α = 0.5) is identified as the standard value to the degree of significance. In the paired t-test, two hypotheses are evaluated against each other. The first hypothesis supposes that the proposed algorithm and the techniques developed in the related work have the same detection accuracy, whereas the second hypothesis supposes that the proposed and related techniques have different detection accuracy. The first hypothesis is discarded if $\left(p\text{-value}<\alpha \right)$. Table 4 displays the t-test outcomes at the 95% level of significance. It can be observed in Table 4 that the classification accuracy of the KLD-based TF-PCD algorithm significantly enhanced the detection accuracy against the traditional TF-IDF and DF-IDF techniques based on the p-values obtained against both TF-IDF and DF-IDF techniques. The rate of accuracy enhancement is indicated in the t-values by which the significance is estimated. The results of the t-test show a significant improvement in the efficiency of the proposed KLD-based TF-PCD feature representation algorithm when compared to the traditional TF-IDF and DF-IDF techniques.