Semantic-aware framework for zero-shot malware classification via attention-based relation network

Our contribution

1. This study presents an image-based ZSL architecture, addressing the challenge of classifying novel and rare malware instances without prior training samples, specifically targeting the Zero-Shot classification problem in malware datasets.

2. We illustrate the effectiveness of incorporating attention mechanisms alongside the Relation Network (RN) to classify malware families.

3. We propose an architecture that features a pairwise attention module to enhance the representation of local patterns and relation-wise attention to capture broader patterns for input into the relation module.

The structure of this article is outlined as follows: ‘Related Concepts’ describes the background that underpins our work. ‘Related Research Work’ provides an overview of the existing work done in this area. Our proposed work is presented in ‘Proposed Methodology’. ‘Implementation of the Proposed Architecture’ presents the implementation specifics of our proposed methodology, detailing each component. The results of our experiments are outlined in ‘Results and Analysis’, while ‘Conclusion’ outlines the main findings and provides a discussion of future work.

Portable executables

A Portable Executable (PE) file, as shown in Fig. S1, is commonly used for executables, dynamic link libraries (DLLs), and other executable components in the Windows operating system. PE files are the standard format for executable files and are necessary for running programs on Windows-based systems. They offer resources, data, and program code in an orderly and structured manner. Because they facilitate program execution and resource management, PE files are essential to the Windows operating system.

Zero-Shot Learning

Few-shot or N-shot learning is a technique that trains a model using a small number of samples from each class, rather than a large amount of data. In this case, ‘N’ represents the number of occurrences per class in the training set. Because FSL does not allow for the use of a huge amount of data for training in real datasets, data scientists may view this as an opportunity. N-shot learning is referred to as ZSL, One-Shot Learning (OSL), or few-shot learning, depending on the value of N. ZSL is a learning approach in which training examples do not occur for all classes, resulting in learning modeling without label availability. To make ZSL practical and ensure successful knowledge transfer from visible to unseen classes, a textual description of each class is required for both seen and unseen categories, which is sometimes referred to as side information. An embedding model, which acquires the embedding weights (linking feature space and semantic space) based on viewed class representations and accompanying attributes, is a common pipeline for ZSL.

Relation network

By learning to compare query images against few-shot labeled example images, RN accomplishes the few-shot job (Sung et al., 2018). The RN is made up of the feature embedding (FE) and relation modules (RM) (Khan et al., 2023c). The training and query images are represented by the embedding module. Subsequently, the relation module joins and analyzes these embeddings to ascertain whether or not they belong to the same category. For a single query image $x_i$ and the training sample set $x_j$ shown in Fig. 1, the relation module generates relation scores $r_{i,j}$, which range from 0 to 1. In the process of meta-learning, the RN acquires distance measurements that allow it to compare images within each episode and calculate a relation score or similarity score. This configuration mirrors the test environment conditions by simulating many few-shot tasks. An award or positive result is noted if the similarity score approaches 1. An overall score or prediction for the query instance is then produced by adding the awards obtained for each pair of query-support instances. Following episodic training, an RN may reliably recognize images of previously unknown classes by determining the relation scores between test images and the few available samples of each new class, all without requiring additional network updates.

Motivation

Novel malware detection addresses the dynamic and intricate nature of malicious software, which is a critical component of cybersecurity (Khan et al., 2024a). The significance of novel malware detection, illustrated in Fig. S2, lies in its ability to address several critical cybersecurity challenges. As cybercriminals continuously develop new forms of malware to bypass conventional defenses, novel detection techniques are essential for identifying and mitigating these emerging threats (Li et al., 2023). They also play a crucial role in safeguarding against zero-day vulnerabilities—software flaws unknown to developers, leaving systems highly susceptible to exploitation (Thapa, Srivastava & Garg, 2023). Furthermore, advanced persistent threats (APTs), often orchestrated by nation-state actors or organized cybercriminal groups, pose long-term risks through stealthy and adaptive attacks aimed at data theft or disruption. Traditional signature-based defenses fall short against such threats, making artificial intelligence (AI)- and machine-learning-driven approaches vital for identifying behavioral patterns indicative of APT activity (Zhang et al., 2023). Ultimately, novel malware detection supports a proactive security posture by enabling early identification and mitigation of threats before they inflict damage (Gu, Xing & Hou, 2024).

Zero-Shot Learning

ZSL has been applied to diverse challenges such as enhancing image quality, learning unfamiliar concepts, and uncovering unseen semantic relationships (Khan et al., 2024b). In the context of malware detection, ZSL enables the classification of previously unseen malware samples. Traditional detection techniques rely on known patterns or signatures, which struggle to keep pace with the rapidly evolving malware landscape (Torres, Álvarez & Cazorla, 2023). ZSL offers a promising alternative by leveraging semantic embeddings or auxiliary information to generalize detection capabilities beyond labeled training data, allowing recognition of unknown malware classes. For instance, Barros et al. (2022) introduced an S-Space representation to compute similarity between object pairs, while (Rahman, Khan & Porikli, 2019) highlighted a new ZSL challenge involving simultaneous recognition and localization of novel objects in complex scenes. Furthermore, Vyas, Venkateswara & Panchanathan (2020) proposed LsrGAN, a generative approach designed to enhance ZSL learning.

Attention based learning

Prior research has extensively examined attention-based mechanisms for malware analysis. A convolutional neural network (CNN)-based approach combined with attention to identify significant byte sequences in binary-derived images was proposed by Yakura et al. (2018). Similarly, the use of Residual Attention for malware detection was explored in Ganesan et al. (2021), with CNNs proving effective in capturing local spatial correlations and enhancing robustness against polymorphic malware (Khan et al., 2023a; Zain et al., 2022), achieving 99.25% accuracy though not addressing novel malware detection. A multi-headed attention mechanism integrated with a CNN achieved 98% accuracy on the Malimg dataset by highlighting infected image regions (Ravi & Alazab, 2023). Additionally, Zhu et al. (2019) proposed a semantic-guided attention localization model that used semantic representations to identify discriminative object parts.

Relation network

Previous efforts have utilized RNs, but before our work, there has been very little exploration into employing ZSL for malware detection. Khan et al. (2023d) have explored RN with one-shot learning. They applied both five-shot learning and one-shot learning, achieving 94% accuracy with one training instance. Sung et al. (2018) introduced the relation network, which is trained extensively to recognize new classes even with minimal examples per class. They conducted experiments using the Omniglot and miniImagenet datasets. Using One Shot scenario, they achieved an accuracy of 50.44%. Hui et al. (2019) expanded on the notion of RN by introducing a Self-Attention Relation Network (SARN) tailored for few-shot learning. They conducted experiments on various benchmark datasets, including Omniglot, miniImageNet, AwA, and CUB. Bishay, Zoumpourlis & Patras (2019), introduced the Temporal Attention-based Relation Network (TARN) tailored for analyzing video segments to establish relation scores between a query video and other sample videos.

Malware visualization

The effective extraction of software features through neural networks necessitates careful consideration of data representation. This ensures the optimal extraction of crucial features and maintains the accuracy of test results. One viable solution involves the utilization of images. Research in the realm of image processing is experiencing a surge in momentum, driven by its promising outcomes across a wide array of applications (Khan et al., 2021). In this approach, the suspected file gets transformed into an image, subsequently processed by the neural network to extract relevant features. The resemblance of file structures is then depicted by the similarity in textures observed in corresponding images. For instance, Nataraj et al. (2011) proposed a malware picture representation scheme wherein the binary code of the malware is transformed into a 2-dimensional matrix. This matrix is represented as a grey-scale graph, given its numerical range [0, 255] as in Fig. S3. The distinctive textures in the data corresponding to different structures become evident through this representation.

Feature embedding module

The model begins with a powerful feature extractor based on ResNet-101, a widely recognized CNN architecture. Residual Networks (ResNets) are known for their exceptional performance in image classification and other computer vision tasks, particularly for very deep networks. ResNet-101, specifically, is a 101-layer deep neural network, making it significantly deeper than earlier models like Visual Geometry Group 16 (VGG-16) or GoogleNet. This feature extractor is pretrained on a large dataset and excels at learning high-level representations from raw data. The standard input size for ResNet-101 is typically 224 × 224 pixels so we provided images of the same dimension. The large number of parameters enables the network to capture complex hierarchical features. Different layers produce feature maps representing the presence of learned features in different spatial locations. These features can be visualized in Figs. S4, S5.

Pairwise attention module

With malware transformed into images, the challenge lies in detecting and pinpointing malware instances that are subtly embedded within the larger image context. Accurate identification and precise localization of these discreetly affected regions are crucial for achieving high malware classification accuracy. To address this, we propose an innovative approach that combines attention mechanisms with CNN. This integrated solution enables us to effectively locate and identify these minuscule infected regions within the broader image context. The implementation of the attention module in our work is illustrated in Fig. 3.

In our approach, we have incorporated two attention-based modules, each serving distinct functions. Within our proposed methodology, the integration of Pairwise Semantic Attention (PwA) into the extracted features plays a crucial role in refining the representation of local patterns and feature interactions within the dataset. This method involves a focused analysis of pairwise relationships between features, utilizing the PwA mechanism to compute attention scores based on measures of semantic similarity or other relevant metrics. By integrating PwA into the feature analysis, the model gains the ability to emphasize specific feature relationships, thereby improving the overall representation of intricate local patterns and interactions within the data.

The focus on pairwise relationships is essential, and the simplicity of Dot-Product Attention proves to be an efficient method for capturing the essence of these relationships between features in both query and support images. This simplicity not only facilitates a clear understanding of basic similarities and differences but also significantly reduces computational complexity, making it suitable for real-time applications and scenarios involving larger datasets.

Furthermore, the decision to utilize dot-product attention is supported by its sufficiency for initial feature comparison. If the primary goal is to identify fundamental similarities and distinctions between image features, the straightforward nature of dot-product attention is adequate without introducing the additional complexity associated with multiple heads. Figure S6 shows the visualization of attended features maps from first channel.

In the context of our model, we defined the matrix Q as having dimensions (batch_size, query_features) encapsulating the feature representation of the query image. Additionally, we introduce matrix S with dimensions (batch_size, num_support_images, support_features), which encompasses the feature representations of all support images in the dataset.

Then the dot product between the query and each support image feature is calculated

$$A=Q.S^{\mathrm{\top }}$$where: ⊤ denotes matrix transpose. We applied softmax to normalize the attention scores:

$$A=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}(A/\sqrt{\mathrm{s}\mathrm{u}\mathrm{p}\mathrm{p}\mathrm{o}\mathrm{r}\mathrm{t}\mathrm{\_}\mathrm{f}\mathrm{e}\mathrm{a}\mathrm{t}\mathrm{u}\mathrm{r}\mathrm{e}\mathrm{s}}).$$Use the attention scores to weigh the support features and sum them:

$$Z=A@S.$$

The output of PwA, Z, is a matrix of dimension (batch_size, support_features) representing the refined feature representation of the query image relative to each support image.

Relation wise attention module

Expanding on the pairwise insights obtained through PwA, Relation-wise Attention (RwA) enhances the analysis by modeling higher-level relationships among multiple features across support images. By considering these intricate connections and dependencies, the model captures broader patterns that go beyond individual pairwise comparisons. Therefore, RwA provides a more comprehensive understanding of the relationships between the query image and different malware classes, resulting in more insightful classification, even for new threats. Multi-head attention unlocks RwA’s full potential, with each head detecting distinct aspects of the relationships between the query image and various malware classes present in the support images. With this deeper understanding, the model achieves more accurate classifications, particularly for novel threats, as it can detect subtle patterns that simpler aggregation methods might overlook. Ultimately, multi-head attention enhances RwA’s ability to represent information, transforming it from a mere data compiler into a sophisticated interpreter of the intricate malware ecosystem.

We take H as the matrix of dimension (batch_size, num_support_images, hidden_size) containing the outputs of PwA for each support image. Each row represents the refined feature representation of the query image relative to a specific support image.

RwA individually prepares the query image and each support image by transforming their features into a format suitable for attention.

$$H_q=W_q\cdot H,H_k=W_k\cdot H$$where:

• $W_q,W_k$ are learnable weight matrices of dimension (hidden_size, head_dim)

• head_dim is the hidden size of each attention head

Then the attention scores are calculated as follows:

(1) $$A=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(\frac{H_q\cdot H_k^T}{\sqrt{\mathrm{h}\mathrm{e}\mathrm{a}\mathrm{d}\mathrm{\_}\mathrm{d}\mathrm{i}\mathrm{m}}}\right)$$where:

• softmax normalizes each row of the attention matrix to sum to 1

Then we applied attention weights

(2) $$H_{\mathrm{o}\mathrm{u}\mathrm{t}}=A\cdot H.$$

The outputs of all attention heads are concatenated

$$H_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}}=[H_{\mathrm{o}\mathrm{u}\mathrm{t}\mathrm{\_}\mathrm{h}\mathrm{e}\mathrm{a}\mathrm{d}1};H_{\mathrm{o}\mathrm{u}\mathrm{t}\mathrm{\_}\mathrm{h}\mathrm{e}\mathrm{a}\mathrm{d}2};...;H_{\mathrm{o}\mathrm{u}\mathrm{t}\mathrm{\_}\mathrm{h}\mathrm{e}\mathrm{a}\mathrm{d}\mathrm{N}}]$$where:

• N is the number of attention heads

• ; denotes concatenation along the hidden size dimension

In the end, a linear transformation is applied

(3) $$H_{\mathrm{f}\mathrm{i}\mathrm{n}\mathrm{a}\mathrm{l}}=W_o\cdot H_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}}.$$where:

• $W_o$ is a learnable weight matrix of dimension (head_dim $\times$ N, hidden_size)

By integrating these two components, we attain a comprehensive representation of the query image, encompassing both local pairwise connections and overarching interactions with various malware categories. This contributes to efficient classification, even for unfamiliar classes. Once RwA has captured global representations from the pairwise relationships between the query and support malware images, the ultimate result, $H_{\mathrm{f}\mathrm{i}\mathrm{n}\mathrm{a}\mathrm{l}}$, is the classification of the query image into the respective class.

Loss

In our model, we used categorical cross entropy loss. It is used because we are dealing with multi-class classification problems where the output can belong to more than one class. Given a true probability distribution y and a predicted probability distribution p, both representing the probability of each class, the categorical cross-entropy loss is calculated as follows:

(4) $$H(y,p)=-\sum _i{y}_i\log (p_i)$$where:

y_i is the true probability of class

p_i is the predicted probability of class.

Dataset and experimental settings

The Malimg dataset comprises 9,339 samples from 25 different malware families, gathered through experiments involving mixtures of network and Windows operating system malware. Notably, the dataset does not include legitimate code samples. Our focus with this dataset is to evaluate the discriminative ability of our model in classifying malware, specifically in identifying the type of malware present. Table 1 provides further information about the Malimg dataset.

MaleVis is designed for vision-based malware recognition research (Jayasudha et al., 2023). It is a corpus involving byte images of 26 (25 + 1) classes. Here, one class represents the cleanware samples while the rest of the 25 classes correspond to different malware types. The MaleVis dataset involves a total of 14,226 images. With this dataset, we can train our model with legitimate code only and evaluate the ZSL classification performance. Table 2 provides detailed information about the MaleVis dataset.

Parameters initialization and network architecture

In the Feature Embedding phase, we leveraged the pre-trained weights from ImageNet for the convolutional layers of ResNet-101 as shown in Fig. 4. Following experimentation, the learning rate was set at a constant value of 0.001. For the PwA component, we employed Kaiming He Initialization for weight initialization. Given that ResNet-101 extensively utilizes ReLU activation functions within its convolutional layers, Kaiming He Initialization was chosen to stabilize gradients through rectified linear unit (ReLU) activation. This choice helped mitigate the risk of vanishing gradients and promotes effective learning. The utilization of Kaiming He Initialization proved instrumental in maintaining a consistent gradient flow, thereby preventing potential training issues.

To mitigate computational expenses, we opted for the utilization of four heads in the RwA mechanism. In the final layers, when addressing multi-class classification, we applied the softmax activation function in the output layer to generate probability distributions across classes. The chosen loss function was categorical cross-entropy. It measures the dissimilarity between the predicted probability distribution and the actual distribution of class labels. Minimizing this loss during training helps the model improve its ability to correctly classify instances into their respective classes.

Performance evaluation metrics

In this work, we used performance metrics such as accuracy, precision, recall, and F1-score which play crucial roles in the evaluation of ZSL-based malware detection systems. Accuracy measures the proportion of correctly classified instances among all instances, providing an overall assessment of the model’s effectiveness.

(5) $$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}=\frac{\mathrm{T}\mathrm{r}\mathrm{u}\mathrm{e}\mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{s}+\mathrm{T}\mathrm{r}\mathrm{u}\mathrm{e}\mathrm{N}\mathrm{e}\mathrm{g}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{s}}{\mathrm{T}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}\mathrm{I}\mathrm{n}\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}\mathrm{s}}.$$Precision measures the proportion of true positive predictions among all positive predictions, highlighting the model’s ability to avoid false positives.

(6) $$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}=\frac{\mathrm{T}\mathrm{r}\mathrm{u}\mathrm{e}\mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{s}}{\mathrm{T}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{c}\mathrm{t}\mathrm{e}\mathrm{d}\mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{s}}.$$

Recall, on the other hand, measures the proportion of true positive predictions among all actual positive instances, emphasizing the model’s capability to capture all relevant instances.

(7) $$\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}=\frac{\mathrm{T}\mathrm{r}\mathrm{u}\mathrm{e}\mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{s}}{\mathrm{T}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}\mathrm{A}\mathrm{c}\mathrm{t}\mathrm{u}\mathrm{a}\mathrm{l}\mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{s}}.$$

The F1-score, which is the harmonic mean of precision and recall, balances between precision and recall, offering a comprehensive evaluation of the model’s performance.

(8) $$\mathrm{F}1\text{–}\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}=2\times \frac{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\times \mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}+\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}.$$In ZSL-based malware detection, where unseen malware families need to be detected, these metrics are particularly important as they reflect the model’s ability to generalize to unseen classes while maintaining high accuracy and minimizing false positives. These metrics collectively provide insights into the robustness, generalization ability, and overall performance of ZSL-based malware detection models.

Baselines

Several baseline articles have been considered for comparison: Hui et al. (2019), Bishay, Zoumpourlis & Patras (2019), Verma, Muttoo & Singh (2020), Bozkir et al. (2021), Barros et al. (2022), Xing et al. (2022), Zhu et al. (2019), Huynh & Elhamifar (2020). We categorized all the baseline methods into two primary groups, depending on the features utilized for training. Hui et al. (2019), Bishay, Zoumpourlis & Patras (2019), Zhu et al. (2019), and Huynh & Elhamifar (2020) employ attention based techniques for feature extraction. While Verma, Muttoo & Singh (2020), Bozkir et al. (2021), Barros et al. (2022), Xing et al. (2022) particularly focus on malware classification. Table 3 summarizes the details of all these articles. Each approach is associated with a specific technique and domain. The techniques range from attention-based feature extraction to deep learning methods, while the domains include both non-malware and malware datasets. While our proposed SMART model is evaluated strictly under a ZSL setting, some of the baseline methods (e.g., Hui et al., 2019) were originally evaluated in a non ZSL setting. We include them here to provide a broader performance context. These results are not directly comparable but help highlight that SMART achieves strong results even under the more constrained ZSL evaluation.

The results presented in Table 4 demonstrate the effectiveness of the proposed SMART model in detecting previously unseen malware families from the Malimg dataset under the ZSL setting. As outlined in the experimental setup, the model was trained on three malware classes and tested on entirely disjoint classes to simulate real-world conditions where novel threats emerge. The model achieves an average accuracy of 94.83%, with consistently high precision, recall, and F1-scores across all three test classes—Worm:AutoIT, Worm, and Backdoor. Specifically, the model performs best on the Worm class with 96.50% accuracy and an F1-score of 0.96, indicating strong generalization ability. These findings suggest that SMART can effectively learn high-level semantic features from the seen classes and accurately apply this knowledge to classify unseen malware variants. This level of performance validates the model’s robustness and highlights its potential for deployment in security systems where timely detection of novel threats is critical.

SMART achieves the highest average accuracy among all other techniques, reaching 95%. Table 5 presents a comparison of performance measures for the zero-shot problem using the MaleVis dataset across existing models and the proposed SMART approach. Overall, SMART demonstrates superior performance across all metrics (Accuracy, Precision, Recall, F1-score). The proposed model exhibits a significant improvement in accuracy compared to existing models, which range from 60.3% to 84%. With an accuracy of 95%, SMART accurately classifies a substantial portion of the samples. Additionally, SMART outperforms all existing models in terms of precision, achieving a score of 0.94. This high precision indicates that a large proportion of samples predicted as positive by SMART are indeed positive. Some models (Verma, Muttoo & Singh, 2020; Huynh & Elhamifar, 2020) show a trade-off between precision and recall, with high precision but lower recall. Surpassing all existing models, SMART achieves a recall of 0.95, effectively identifying a large proportion of actual positive samples in the dataset. Moreover, SMART’s F1-score of 0.947 is the highest among all models, demonstrating an excellent balance between precision and recall. This indicates that SMART achieves both high precision and high recall simultaneously, making it an effective solution for the zero-shot problem in the malware dataset.

Performance across different datasets

We utilized the Malimg and MaleVis datasets to assess the performance of our model. For our testing scenario, we employed three classes from the Malimg dataset, encompassing seven families. The experiments were conducted using all instances per class. However, for visualization purposes, we presented a confusion matrix using a stratified subset of approximately 100 samples per class. This was done to maintain visual clarity and balance across classes, especially considering the significant class imbalance in the original dataset (e.g., Allaple.A with 2,949 samples vs. Autorun.K with only 106). This does not affect the reported performance metrics, which are computed over the full test set.

The findings, as illustrated in Fig. S7, revealed a notably low false positive rate across all families. Specifically, for Allaple.L, the accuracy reached its peak.

The evaluation of SMART for malware classification for the MaleVis dataset gave insightful results across various performance metrics as shown in Fig. S8. The average accuracy of the model was calculated to be 95%, showcasing its ability to classify instances across all classes correctly. Precision, which measures the accuracy of positive predictions, varied across different classes, with some classes exhibiting higher precision due to lower false positive rates, as shown in Table 6. For instance, Adposhel, Hilium.A and Goodware demonstrated a particularly high precision, indicating the model’s proficiency in accurately identifying instances of this class. Additionally, recall, highlighting the model’s ability to identify all positive instances, varied across classes, with certain classes displaying higher recall values than others. Notably, Neoreklami, Elex, Injecto, and Regrun exhibited exceptional recall, underscoring the model’s effectiveness in capturing all instances of this class. F1-scores, which provide a balanced assessment of precision and recall, ranged from 0.83 to 1.00, emphasizing the trade-off between precision and recall in classification performance. The confusion matrix heatmap visually depicts the distribution of true positives, false positives, true negatives, and false negatives, offering further insights into the model’s classification performance for each class. Overall, these results provide a comprehensive understanding of the model’s strengths and weaknesses, facilitating informed decisions for model refinement and optimization.

Table 6 presents a comprehensive overview of performance metrics for all classes within the MaleVis dataset across various malware detection models: Proposed SMART, UMAP, Statistical, DNN, and Malware-SMELL. SMART was evaluated under a ZSL setup, where testing classes were unseen during training. In contrast, the baseline models (UMAP, Statistical, DNN, and Malware-SMELL) were trained and tested using supervised learning setups. This comparison highlights SMART’s strong performance under the more challenging ZSL scenario. We included these baselines to demonstrate that even under the more challenging ZSL scenario, SMART outperforms traditional supervised models on unseen malware classes, underscoring its superior generalization capability. Analyzing the performance of the proposed SMART model, we observed notable variations in detection efficacy across different malware families. For instance, the Adposhel family achieves a commendable accuracy of 0.90 under SMART, indicating strong classification. Conversely, some families, such as Neoreklami, exhibit lower accuracy in baseline methods, underscoring SMART’s improved capability to accurately identify instances from these challenging classes.

A closer examination reveals that certain malware families consistently demonstrate strong performance across multiple metrics within the SMART model. For example, the Vilsel family consistently maintains high accuracy, precision, recall, and F1-score values, suggesting that the SMART model effectively detects instances belonging to this family.

Moreover, the “All family” row provides an aggregate summary of the SMART model’s overall performance across all malware families, offering insights into its collective effectiveness in detecting malware within the MaleVis dataset. This summary facilitates a comprehensive evaluation of the SMART model’s robustness and reliability in real-world malware detection scenarios. Figure 5 illustrates the commendable accuracy achieved by our model, while Fig. 6 demonstrates the favorable precision of our model compared to other models.