Malware homology determination using visualized images and feature fusion

Bytecode image construction

The process of converting a given executable file of malware, i.e., a PE binary file, to an image is shown in Fig. 2. First, the 8-bit length was used as the unit to read in turn and converted it into an integer in the range of 0 to 255, thus forming a one-dimensional vector. Then, according to the fixed row width, we transformed it into a two-dimensional matrix. The value range of each element in the matrix was the same as the image pixel value, which was 0 to 255. This allowed the elements in the matrix to be mapped as pixels, further converted into grayscale image, which we called bytecode images.

We set the width of the bytecode image to a fixed value, while the height depended on the PE file size. Since the sizes of different malware files varied greatly, we adopted different image widths for different file sizes, as shown in Table 1.

Opcode image construction

We obtained the global structure information of the malware PE file after retrieving the bytecode image. We then specified the code segment of PE file and converted the opcode feature information into an opcode image after disassembling the code segment.

The PE file of malware uses flat address space, so the code and data are stored in different blocks in a more fixed format. For example, the code segment is specially applied to store the program code of PE file. Although the information contained in the segment has certain redundancy, it is obviously the most representative feature in the malware information. Therefore, we used the assembly code features of this part to analyze the homology of the malware.

We disassembled the malware sample in the PE format through IDA Pro, which generated the “.asm” format assembly file. In assembly file text, “.text” or “CODE” was typically used to identify each line of code instruction, where the instruction structure contained the opcodes and operands. The operand is actually the memory address of a function or variable. The malware variants with homology were obtained by a deforming code, which may cause operands to change. The part of the opcode in the instruction was unchanged or rarely changed in malware deformation (Bilar, 2007). Therefore, we only used the opcode as the feature and did not consider the operands. We extracted the opcode from the assembly code in the malware assembly file to get the opcode stream (Fig. 3).

It should be noted that malware developers may use techniques such as code obfuscation to disassemble, so that the disassembled program contains invalid interference instructions or control flow graphs. However, the method proposed in this paper mainly focuses on the statistical characteristics of each opcode sequence in malicious code, and it is difficult for code obfuscation techniques to completely change this characteristic.

N-gram is a commonly used semantic feature model in natural language processing, which considers that the occurrence probability of the n^th word is only related to the preceding n−1 word. Program code is essentially a text language, such as the assembly code obtained above also has language structural and semantic features, so the n-gram model was used as the feature analysis and extraction method of malware. We introduced the n-gram model into the semantic information analysis of opcodes, and previous studies have shown that different malware families have different n-gram characteristics (Santos et al., 2009; Li, Chen & Cui, 2017).

However, before extracting the n-gram features, we considered that there are many very rare opcodes in the opcode stream of malware, and the n-gram sequence formed by them only appears in a few malware and may be used by malware makers for code obfuscation. Therefore, these opcodes did not contribute to our analysis because of their capacity for rapid growth of the feature. In order to avoid the influence of rare opcodes on the n-gram feature capacity, we first selected the key $m$ opcodes according to their probability before extracting the opcode sequence. For the set $O$ composed of all the opcodes in the training set X, the probability of the set element $o_i$ is calculated as follows:

(1) $$p\left(o_i\right)={\displaystyle \frac{{\sum }_{x_i\in X}freq\left(o_i|x_i\right)}{{\sum }_{o_i\in O}{\sum }_{x_i\in X}freq\left(o_i|x_i\right)}}$$where $freq\left(o_i|x_i\right)$ refers to the frequency of opcode $o_i$ in the sample $x_i$ that belongs to the training set $X$. For the opcode stream extracted from specific malware samples, we deleted the non-critical rare opcode to get a simplified opcode stream. In this way, we greatly reduced the n-gram feature capacity, so as to improve the calculation efficiency for the subsequent analysis.

For the simplified opcode stream, we use a sliding window with the size of 2 to extract a series of 2-gram opcode sequences, and then counted the frequency of each sequence. In this paper, we set the sequence length of opcodes to 2 instead of using a single opcode, because most operations with malicious intent in malware need to be completed by consecutive multiple opcodes. However, if we extracted a larger length of the opcode sequence, the number of features would be greatly increased, resulting in a rapid increase in computational overhead.

For some opcode stream of malware samples (Fig. 2), if the opcode ‘shl’ does not belong to the m key opcodes screened, the final n-gram opcode sequence generated is $o{s}_1=\left(and,sub\right),$ $o{s}_2=\left(sub,xor\right),o{s}_3=\left(xor,inc\right)$… $o{s}_{12}=\left(cmp,jmp\right)$, and the frequency of each sequence is 1.

We can construct $m^2$ opcode sequences with the length 2 by means of the selected $m$ key opcodes according to permutations and combinations in pairs, and then assign a weight factor $w_i$ to each sequence $o{s}_i$ to get a $m^2$-dimension opcode sequence vector:

(2) $$V=\left\{\left(o{s}_1,w_1\right),\left(o{s}_2,w_2\right),\dots ,\left(o{s}_{m^2},w_{m^2}\right)\right\}$$

The weight factor $w_i$is calculated by a new computing method called TF-GR, and is equal to the product of the word relative frequency $t{f}_i$ of the opcode sequence and the information gain rate:

(3) $$w_i=t{f}_i\times GR\left(X,o{s}_i\right)$$where $GR\left(X,o{s}_i\right)$ represents the information gain rate of the opcode sequence $o{s}_i$ to the malware training set $X$. If there are $l$ opcode sequences in a sample of malware, the frequency of occurrence of each $o{s}_i$ is denoted by $n_i$. The $t{f}_i$and $GR\left(X,o{s}_i\right)$ of each $o{s}_i$ for a malware simple are calculated as follows:

(4) $$t{f}_i={\displaystyle \frac{n_i}{{\sum }_{j=1}^l{n}_j}}$$

(5) $$GR\left(X,o{s}_i\right)={\displaystyle \frac{H\left(X\right)-H\left(X|o{s}_i\right)}{H\left(o{s}_i\right)}}$$where $H\left(X\right)$and $H(o{s}_i)$represent the information entropy of dataset X and opcode sequence $o{s}_i$respectively, $H(X|o{s}_i)$represents the conditional entropy of dataset X under opcode sequence $o{s}_i$. If the data set X has k different families $f_c$, $c=1,2,\dots ,k$, and the number of opcode sequence $o{s}_i$ appears in all malware samples has s different values $nu{m}_j\left(o{s}_i\right)$, $j=1,2,\dots ,s$, then $H\left(X\right),$ $H(o{s}_i)$ and $H(X|o{s}_i)$ can be expressed as follows:

(6) $$H\left(X\right)=-\sum _{c=1}^{k}p\left(f_c\right){\log }_{2}p\left(f_c\right)$$

(7) $$H\left(X|\mathrm{o}\mathrm{s}{}_i\right)=\sum _{j=1}^{s}p\left(\mathrm{n}\mathrm{u}\mathrm{m}{}_j\left(\mathrm{o}\mathrm{s}{}_i\right)\right)H\left(X|\mathrm{n}\mathrm{u}\mathrm{m}\left(o{s}_i\right)=\mathrm{n}\mathrm{u}\mathrm{m}{}_j\left(\mathrm{o}\mathrm{s}{}_i\right)\right)$$

(8) $$H\left(\mathrm{o}\mathrm{s}{}_i\right)=-\sum _{j=1}^{s}p\left(\mathrm{n}\mathrm{u}\mathrm{m}{}_j\left(\mathrm{o}\mathrm{s}{}_i\right)\right){\log }_{2}p\left(\mathrm{n}\mathrm{u}\mathrm{m}{}_j\left(\mathrm{o}\mathrm{s}{}_i\right)\right)$$

It is important to note that if a malware sample does not contain a specific opcode sequence, the weight factor of the opcode sequence is equal to 0, because its frequency is 0.

In this subsection we converted the opcode sequence vector V obtained previously into a grayscale image. We took m selected key opcodes as the horizontal axis and the vertical axis of the image respectively, for images regarded as a pixel matrix and then constructed an $m\times m$ opcode image matrix. The two-dimensional coordinates of each point in the matrix were expressed as an opcode sequence with the length 2, as shown in Table 2. Then, we mapped the opcode sequence vector to the image. The sequence in each element of the vector was mapped to the position of a specific pixel point in the image matrix, and the weight factor corresponding to the sequence was expressed as the pixel value of the point.

Because the pixel value of the image is between the interval [0 255], we normalized the weight factor so that its value fell within the range of pixel value as follows:

(9) $$pi{x}_i={\displaystyle \frac{w_i}{\mathrm{m}\mathrm{a}\mathrm{x}\left(w_i\right)}\times 255}$$

According to the above grayscale image generation method, we mapped the malware opcode sequence vectors of different families to opcode images. The detailed opcode image construction process is given in Algorithm 1. Figure 4 illustrates an example of different families of opcode images (image size $64\times 64$). Malware opcode images of different families are visually distinct, while malware with homology has a high visual similarity in opcode images due to the similar statistical characteristics of opcode.

Image preprocessing

We preprocessed the malware image to meet the requirements of CNN for input data. The CNN model is required to input image data of the same size when it performs a task such as image classification. Moreover, in order to facilitate the subsequent convolution operation, the length and width of image data should be the same. Due to the different sizes of executable PE files of malware, the sizes of various converted bytecode images are also different. Therefore, it is necessary to normalize all grayscale images.

This paper uses four pixel values of the nearest neighbor in the original image to determine a pixel value of the target image by means of bilinear interpolation algorithm, which does not produce the sawtooth effect. The algorithm is more suitable for our scenario than the nearest neighbor interpolation. The normalized size of the grayscale image is a hyper-parameter, reflecting the relationship between classification accuracy and computational expense (Zhang & Wu, 2006). The larger the normalized image size, the more information CNN input data contains, resulting in better classification effect at a cost of higher time consumption. However, the bytecode image and opcode image need to be input to CNN as two channels at the same time. In order to facilitate the subsequent feature fusion operation, we kept the two input data formats unified. The size of the opcode image was $m\times m$, so the bytecode image also needed to get the grayscale image of $m\times m$ through sampling operation. Figure 5 shows a sample of a byte grayscale image with the original size of 512 $\times$ 248 converted into 128 $\times$ 128 and 64 $\times$ 64 input images after sampling.

Dual-branch CNN model design

We propose a dual-branch CNN model suitable for malware image classification as shown in Fig. 6. Bytecode image and opcode image are used respectively as input of dual-tunnel branch networks of the CNN model to extract the deep features of malware. Then, the feature maps respectively obtained by the two branch CNN networks were fused in the last fully-connected layer which served as the final classification basis of the Softmax classifier. Specifically, each branch network consisted of two convolution layers with the ReLU activation function. Their filter size was 3 × 3 and the number is 16 and 32, respectively. Each convolution layer was followed by a max pooling layer, where the pooling size was 2 × 2 and stride was 2. Figure 6 shows a 64 × 64 figure as an example, and 32 feature maps with the size of 16 × 16 were output from each branch network after two rounds of convolution feature extraction. We fused the feature maps to get 64 feature maps. Since the output of the pooling layer was multidimensional, it was necessary to use the flattened layer to convert multidimensional nodes into one-dimensional nodes as the input of the fully-connected layer D1 with 512 nodes. In order to prevent over-fitting, we also added the dropout layer after the fully-connected D1 layer. The final part of the whole network model was the Softmax layer to output the family classification results. The specificity of dual-branch CNN structure and parameters is shown in Table 3.

Convolution maps each position of the image to a new value through a linear transformation to extract a simple feature of the data. Multi-layer convolution uses layer-by-layer mapping to extract more complex abstract features of the image; the feature map may be obtained using the activation function after convolution. The process of convolution operation can be expressed as follows:

(10) $$h\left(i\right)=f\left({\sum }_{x=1}^n{\sum }_{y=1}^n{\sum }_{z=1}^m{a}_{x,y,z}\times w_{x,y,j}^i+b^i\right)$$where $a_{x,y,z}$ represents the value of input data node $\left(x,y,z\right)$ in the filter, and $w_{x,y,j}^i$ represents the weight of the filter for the input node $\left(x,y,z\right)$. $b^i$ represents the bias parameter corresponding to the output node. $f\left(\right)$ is the activation function. $n$ represents the length and width of the filter, and $m$ indicates the depth of the filter.

The pooling layer was used for image feature downsampling and feature dimension reduction. The data processing capacity was compressed while the effective information was retained, which effectively reduced over-fitting and improved the fault tolerance of the model. For the input feature map, new smaller size features were obtained through the pooling operation of Formula (11).

(11) $$Y_j^{i+1}=\mathrm{d}\mathrm{o}\mathrm{w}\mathrm{n}\mathrm{s}\mathrm{a}\mathrm{m}\mathrm{p}\left(Y_j^i\right)+b_j^{i+1}$$where $downsamp(·)$ represents the downsampling mode under image features, and $b_j^{i+1}$ represents the bias parameter. In general, max-pooling or mean-pooling can be selected for pixels in the field determined by the filter. We found that the pooling layer of the CNN adopts the max pooling image feature subsampling method.