Hybrid rule-based botnet detection approach using machine learning for analysing DNS traffic

Honeypots

Honeypots are widely used for identifying and analysing the behaviour of botnet attacks. Honeypots are purposely designed to be vulnerable to botnet attacks to capture and gather as much data as possible on the botnet (Freiling, Holz & Wicherski, 2005). Honeypot also runs specialised software that attempts to match bots’ signatures and discovers the location of the botnet’s C&C server.

There are at least three types or levels of honeypots depending on the required level of bots information, the complexity of the study’s data, and the interaction level permitted to the attacker: low, medium, and high (Koniaris, Papadimitriou & Nicopolitidis, 2013; Nawrocki et al., 2016). A low-level honeypot or Low Interaction Honeypot (LIH) stores unauthorised communication with a limited attacker interaction; therefore, it is safer and easier to maintain than other honeypot types. A Medium Interaction Honeypot (MIH) provides more meaningful interaction with the attacker but not as open as a High Interaction Honeypot (HIH). HIH is a computer with a real OS running vulnerable services to attract intruders to break into to capture their actions for analysis. Table 2 shows the pros and cons of the three types of honeypot.

Honeydns, proposed by Oberheide, Karir & Mao (2007), is a form of LIH that uses some simple statistics over the captured queries and collects DNS queries targeting unused (i.e., darknet) address spaces. This method prevents attackers from avoiding it (Bethencourt, Franklin & Vernon, 2005). However, a honeypot cannot detect all forms of bots, such as bots that are not using scanning to propagate (Dornseif, Holz & Klein, 2004). Furthermore, attackers could utilise honeypots to target other systems or machines outside the honeypots (Liu et al., 2009). Figure 8 shows the standard honeypot configuration.

Anirudh, Arul Thileeban & Nallathambi (2017) built a model using MIH as a sensor to collect attack logs. When coupled with an Intrusion Detection System (IDS) as a verifier, these logs increase 55–60% in IDS efficiency against DDoS attacks compared to using IDS alone. However, their research is limited to DDoS attacks only (Anirudh, Arul Thileeban & Nallathambi, 2017).

Intrusion Detection System (IDS)

Da Luz (2014) and Alomari et al. (2016) categorised IDS into two: anomaly-based and signature-based (Da Luz, 2014; Alomari et al., 2016). The anomaly-based IDS can be further classified into host-based IDS and network-based IDS (Dornseif, Holz & Klein, 2004). The subsequent sections provide more details on the different types of IDS.

Anomaly-based Botnet detection

Anomaly-based detection method relies on different DNS anomalies to identify botnets. Some of the DNS anomalies used for detection include high network latency, Time to Live (TTL) domain, patterns of domain requested per second, high traffic volumes, and irregular device behaviour that may expose bots’ existence. In other words, the term “detection based on anomaly” refers to the act of finding odd habits that differ from the expected ones. The anomaly-based approaches have two detection methods: host-based and network-based (Dornseif, Holz & Klein, 2004; Karim et al., 2014; Da Luz, 2014).

Host-based approaches

Host-based technique scans and protects the computing device locally, or in other terms, ‘host-level. Shin, Xu, and Gu proposed the EFFORT framework that combines several techniques to observe DNS traffic at the host level (Shin, Xu & Gu, 2012). EFFORT has five specific modules that use a controlled machine learning algorithm to report malicious domain names regardless of network topology or communication protocol and performs well with encrypted protocols. However, the EFFORT framework only worked with botnets that rely on the DNS administration to recognise C&C servers’ addresses. Host-based IDS is typically an inadaptable approach. Consequently, the observing agents must be deployed on all devices in the network to be effective against botnet attacks (Da Luz, 2014).

Network-based approaches

Network-based IDS analyses network traffic, either actively or passively (Dornseif, Holz & Klein, 2004; Karim et al., 2014; Da Luz, 2014). The active monitoring approach injects test packets into the network, servers, or applications instead of just monitoring or passively measuring network traffic activities.

• Active Monitoring Approaches

Ma et al. (2015) proposed an active DNS probing approach to extensively determine unique DNS query properties from DNS cache logs (Ma et al., 2015). This technique could be used remotely to identify the infected host. However, injecting packets into the network increased the risk of revealing the existence of the IDS on the network. Furthermore, active analysis of DNS packets could threaten users’ privacy. Besides, the NXDOMAIN requests were absent from the DNS cache entry for domain names. The active monitoring mechanism added additional traffic from test and test packets injected into the network (Alieyan et al., 2016).

FluXOR (Passerini et al., 2008) is one of the earlier systems to detect and monitor fast-flux botnet. The detection technique is based on an interpretation of the measurable characteristics of typical users. It used an active sampling technique to track each suspected domain to detect the fast-flux domain. Not only can FluXORs recognise fast-flux domains, but also the number and identity of related proxy servers to prevent their reuse in a potential fast-flux service network (Monika Wielogorska, 2017). However, FluXOR is restricted to the fast-flux domains advertised by SPAM traffic (Perdisci, Corona & Giacinto, 2012).

• Passive Monitoring Approaches

Passive monitoring utilises specific capturing instruments, known as “sensors,” to track the passing traffic. Subsequently, the traffic on the network under inspection would not increase. Weimer implemented the first passive detection method in 2005 (Weimer, 2005; Zdrnja, Brownlee & Wessels, 2007).

NOTOS (Antonakakis et al., 2010) is a comprehensive domain name reputation system that analyses DNS and secondary data from honeypots and malware detection services. Reputation process inputs are the characteristics derived from the list of domain names, such as the resolved IP address, the domain registration date, identified malware samples accessing a given domain name or IP address, and domain name blacklisted IP addresses. These features allowed NOTOS to change the domain legitimacy model, clarify how malicious domains are run, and calculate the perfect reputation score for new domains. NOTOS has high accuracy and low false-positive rate and can identify newly registered domains before being released on the public blacklist. However, a reputation score algorithm needs a domain registration history (whois), which is not available for all domain names, to award an appropriate reputation score. It is also unusable against frequently shifting C&C domains, such as a hybrid botnet that uses several C&C server hubs to execute commands (Kheir et al., 2014).

Contrary to NOTOS, Mentor (Kheir et al., 2014) proposed a machine learning approach on a statistical set of features. The proposed model sought to exclude all valid domains from the list of blacklisted C&C botnet domains, which helped to minimise both the false-positive rate and domain misclassification during the identification process. To do this, Mentor embedded a crawler to gather data on suspicious domain names, e.g., web content and domain properties, to create a DNS pruning model. The Mentor method’s performance is better when measured against public blacklist domains with meagre false-positive rates.

EXPOSURE is a system proposed by Bilge et al. (2011) that used inactive DNS information to identify domains vulnerable to malicious behaviour. It held a total of 15 features distributed over four classes: time-based, DNS-based, TTL-based, and domain-based. It also used these features to improve the training of PART classifiers.

Kopis introduced a new traffic characteristic by analysing DNS traffic at top-level domain hierarchy root levels (Antonakakis et al., 2011). This method reliably looked at the malware used domains by going through global DNS query resolution patterns. Unlike other DNS reputation strategies such as NOTOS and EXPOSURE, Kopis allowed DNS administrators to freely inspect malware domains without accessing other networks’ data. In addition, Kopis could search malware domains without access to IP reputation info (Xu et al., 2017).

Pleiades (Antonakakis & Perdisci, 2012) helped classify recently controlled DGA domains using non-existent domain responses (NXDOMAIN). However, because its clustering strategy relied on domain names’ structural and lexical features, it was limited to DGA-based C&C only. Also, one of the outstanding issues of NXDOMAIN-based detection was dealing with a compromised host with malware that requested several queries to DGA domains over an extended time. It might be possible to detect the C&C addresses of a domain fluxing botnet in the local network by comparing the accurate domain resolution entropy to the missed one (Yadav et al., 2010). Since the randomness in the domain name alphanumeric characters is measurable by calculating the entropy value, in their implementation, the researchers utilised an offline IPv4 dataset from the Asian region. They achieved a low FP rate of just 0.02%. However, their approach was limited to non-dictionary IPv4 domain names.

There has been extensive discussion on botnet detection approaches that employ machine learning detection in the literature. For example, BOTCAP (Gadelrab et al., 2018) utilises J48 and ‘Support Vector Machine’ (SVM) classifiers for training the extracted DNS features. The authors showed that the J48 classifier, a Java version of the C4.5 classifier, performed better than other classifiers. However, a hybrid detection model that combines the output of the J48 classifier with other classifier models’ output could further improve the performance.

Li et al. (2019) attempted to find the best classifiers from several classifiers, such as Decision Tree-J48, ‘Artificial Neural Network’ (ANN), ‘Support Vector Machine’ (SVM), Logistic Regression, ‘Naive Bayes’ (NB), ‘Gradient Boosting Tree’ (GBT), and ‘Random Forest’ (RF) (Li et al., 2019). As a result, the authors showed that J48 was the best classification algorithm to classify the DGA domain (Li et al., 2019). However, their proposed approach was not using any hybrid rule model.

Haddadi et al. (2014) adopted the C4.5 classifier for botnet classification (Haddadi et al., 2014). However, the selected subset of features did not contribute to any improvement in the classification process. The experimental results achieved an 87% detection rate.

Likewise, deep learning, a subset of machine learning, has received significant attention lately. A deep learning algorithm of recurrent neural networks (RNN), long short-term memory (LSTM), and the combination of RNN and LSTM have been applied as a botnet detection method (Shi & Sun, 2020). The RNN and LSTM combination achieved higher detection results. However, deep learning techniques require massive pre-processing of data, long process time, and resources with high-speed processors. Besides, to discover new bots, re-training the whole model with a new dataset is a must. Re-training is a time-consuming process and not suitable for detecting new botnets.

From the literature above, it is noticeable that there is a lack of significant features and rules that contribute to detecting DNS-based botnet with high accuracy and low false-positive rate.

The summary of some existing botnet detection approaches based on DNS traffic analysis are tabulated in Table 3.

Table 3:

Summary of DNS-based botnet detection techniques.

Author and Year	Detection Approach	Strengths	Weakness
Oberheide, Karir & Mao (2007)	Honeypot	Easy to build; help to discover new botnet within its network	Limited scalability and interaction with malicious activities
Ma et al. (2015)	Active Network-Based	Could identify the infected host in remote management networks	Restricted to domain names in cache entry; cannot detect NXDOMAIN request; high probability to be detected by attackers, and introduce privacy concern
Passerini et al. (2008)	Active Network-Based	Discover fast-flux domains; also detect the number and identity of related proxy servers to prevent future reuse	Limited to fast-flux domains advertised through SPAMs traffic
Shin, Xu & Gu (2012)	Host-Based	Provide real-time protection	Limited to host-level; must be installed in all hosts in networks
Antonakakis et al. (2010)	Passive Network-Based	Has high accuracy and low false-positive rate; recognise recently registered domain names before being published to public blacklist	Cannot classify new domains; inaccurate against frequently changing C&C domains like hybrid botnet architecture that utilises many master C&C hubs to execute a command
Gadelrab et al. (2018)	Passive Network-Based	Able to detect and identify individual bots without collecting massive data from infected machines; based on statistical features of botnet traffic (i.e. independent of traffic content)	Low detection rate
Antonakakis &Perdisci (2012)	Passive Network-Based	Able to analyse DGA-based C&C queries limited to detect C&C addresses for fast-flux botnet in a local network	Limited to high entropy domain names (non-dictionary words) with IPv4 domain resolving
Bilge et al. (2011)	Passive Network-Based	Able to identify new botnet through machine learning classifier	A 15 features detection model consumes a lot of data processing and sensors on RDNS servers for learning model
Kheir et al. (2014)	Passive Network-Based	It has a low false-positive rate	Only identify benign domains; misclassification for the hijacked and high reputation domain name; weak against hybrid botnet
Ramachandran, Feamster & Dagon (2006)	Signature-based	Attempt to recognise botmasters’ address and identify their location	Only detect reconnaissance botmaster; limited to bot advertised through SPAMs traffic using heuristics approach; need to update DNSBL database
Shi & Sun (2020)	Deep learning analysing	Used a hybrid deep learning method to classify DNS-based botnet	it’s required that to train the whole model once more to discover new botnet.
Our approach	Hybrid rule-based	Hybrid machine learning approach using a united of two machine learning classifiers that resulted in high accuracy botnet detection	Not deal with encrypted DNS traffic that uses DNS via the Transport Layer Security protocol (DoT) or DNS via secure hypertext protocol (DoH)

Stage 1: data pre-processing

Data pre-processing stage is critical for the proposed approach. It helps to focus on the required DNS features to provide a more flexible selection analysis. Also, this process reduces the analysis time and false-positive results as well.

It consists of two steps, DNS packet filtering and data cleansing. The packet filtering step ensures that only DNS packets remain in the filtered network traffic. Furthermore, this research assumes that a third-party security mechanism is deployed in the network to prevent or detect DNS fragmentation packets. Therefore, the proposed approach incorporates the third-party mechanism to ensure that the DNS fragmented packet will not bypass the proposed rules.

DNS packet filtering step

The process of resolving DNS queries occurs nearly instantaneously most of the time. Since there is no need for a handshaking technique provided by Transmission Control Protocol (TCP), DNS traffic uses User Datagram Protocol (UDP) at port 53, making the filtering process easier. Furthermore, this study focuses on the analysis of selected features of DNS. The filtering step is responsible for the extraction of the required DNS features from DNS packets. Figure 10 illustrates the process of the data pre-processing stage.

Figure 11 visualises the DNS packet structure. Table 4 tabulates the extracted DNS traffic fields selected for this study. Finally, Table 5 presents the extracted DNS record types with their function in the DNS protocol.

Data cleansing step

Cleansing the data means removing errors and broken DNS sessions from the datasets. Thus, the cleaning process helps achieve more accurate results and reduces the processing time of subsequent stages (Alieyan et al., 2021).

DNS traffic analysis

The DNS traffic analysis stage consists of enriched features calculations (feature engineering) and building training dataset steps. The following subsection provides a more detailed explanation for each step.

Enriched features calculations (feature engineering) step

The feature engineering process employs different machine learning domains to solve various types of problems. Its main task is to select and compute the most significant features or attributes and eliminate irrelevant and redundant features to improve machine learning algorithms’ performance. In this study, the feature engineering process derives enriched DNS features from the basic extracted features in Stage 1.

Based on the review of existing literature and studies, we considered two significant characteristics of DNS-based botnet in its connection phase. Firstly, DNS-based botnet generates a massive number of domain names. Secondly, the generated domain names tend to be random and different from the human-generated ones (Alieyan et al., 2021).

The calculation of randomness of domain names could help to distinguish anomalous traffic and benign traffic. In information theory, the randomness could be calculated by the Shannon entropy equation, first introduced by Claude E. Shannon in his paper titled “A Mathematical Theory of Communication” (1948). Shannon entropy allows estimating “the average minimum number of bits needed to encode a string of symbols based on the alphabet size and the frequency of the symbols.” Moreover, Shannon entropy is also being applied in information and network analysis. Therefore, the proposed approach employs the Shannon entropy algorithm to calculate the resolved domain name’s entropy, using Eq. (1).

(1) $$H\left(x\right)=\sum _{i=1}^{n}p\left(x_i\right)I\left(X_i\right)=\sum _{i=1}^{n}p\left(x_i\right)\log {\displaystyle \frac{1}{p\left(x_i\right)}=-\sum _{i=1}^{n}p\left(x_i\right)\log p\left(x_i\right)}$$

Since bots repeatedly tried to connect with the botmaster’s C&C server, the number of domain resolution requests will be high. The proposed methodology for traffic analysis is to group the requested domain according to source IP. Since the bot or botnet tries to connect with the botmaster in different predefined periods, the average entropy for the source IP is essential to distinguish between benign and malicious traffic. Furthermore, we use the same time value, 5 s, for flow analysis based on a previous study (Alieyan, 2018). Equation (2) calculates the average domain entropy feature (F1).

(2) $$\overline{H\left(x\right)}={\displaystyle \frac{{\sum }_{i=1}^N{H}_{\left(xi\right)}}{N}}$$

where N denotes the number of domain requests in a predefined time (5 s), and $H_{\left(x\right)}$ is as mentioned in Eq. (1). Moreover, as previously mentioned, a botnet in the rallying phase repeatedly tries to connect with its C&C server. Since the C&C server is usually configured with a single or only a few domains from the pool of vast numbers of bot-generated domain names, many failed domain name resolution requests occur before the bot successfully connects with the registered C&C domains. Such actions will increase the NXDOMAIN response ratio from the infected network or host, indicating anomalous behaviour (Wang et al., 2017). Furthermore, regular users usually have different domain request time patterns, whereas the infected host endeavour to connect with their C&C server according to a pre-programmed schema. Consequently, the time for domain request entropy in legitimate hosts diverges from the infected ones (Qi et al., 2018).

Furthermore, the values of legitimate DNS lookup type requests and DNS record types, as stated in Table 5, will differ from the values in an infected host since that user’s behaviour in requesting domain resolution is different from the bot-generated request (Hikaru et al., 2018). Likewise, the attackers exploit fast-flux by combining round-robin IP addresses with a short TTL for the DNS Resource Record (RR) (William & Danford, 2008), leading to different TTL settings for the malicious domains.

Based on the characteristics mentioned above, the equations for the calculation of the enriched feature are as follows:

R is the ratio of the successful DNS response within a predefined time, which is also the definition of the second feature (F2):

(3) $$R={\displaystyle \frac{R_s}{R_n}}$$

where $R_s$ represents the number of successful DNS responses, and $R_n$ represents the number of DNS requests.

H(q) is the randomness number of DNS queries rate within a predefined time interval. It is calculated according to the Shannon entropy stated in Eq. (1). Thus, the definition of the third feature (F3) is calculated by:

(4) $$H\left(q\right)=-\sum _{x=1}^N\left({\displaystyle \frac{q_x}{{\sum }_{x=1}^N{q}_x}log{\displaystyle \frac{q_x}{{\sum }_{x=1}^N{q}_x}}}\right)$$

where $q_x$ represents the number of DNS queries in an $x$ time interval, and N refers to the total number of DNS queries type (Qi et al., 2018).

$\mathrm{D}\mathrm{\Delta }t$ is the number of resolved DNS record types within a predefined time interval. The definition of the fourth feature (F4) is as follows:

(5) $$\mathrm{D}\mathrm{\Delta }t=\sum _{i=1}^N\left(D_i\right)$$

where $\mathrm{\Delta }t$ represents the predefined time, $D_i$ represents the number of the i-th DNS request type as tabulated in Table 5, and N denotes the total number of DNS requested.

The average of the resolved domain name TTL in a predefined time interval, which is the definition of the fifth feature (F5), is measured by:

(6) $$TT{L}_{\mu }={\displaystyle \frac{{\sum }_{i=1}^{N}TT{L}_i}{N}}$$

The total number of various values for TTL within a predefined-time (F6).

The total number of different sizes of DNS packets within a predefined-time (F7).

The number of different DNS destinations within a predefined-time (F8).

The total number of unsuccessful (error) DNS response within a predefined-time (F9).

The ratio of successful DNS response in a predefined-time (F10).

Building training dataset step

The objective of this step is to construct a training dataset to train the machine learning classifiers. The training dataset comprises a set of enriched features computed through a feature engineering process. As mentioned earlier, the features are calculated based on 5 s running time series of the source IP that resulted in a network traffic flow defined as unidirectional traffic with certain packet features that represent a flow tuple (Krmicek, 2011). In this study, the features that describe the flow are the source IP, destination IP (DNS server), and protocol (DNS). Furthermore, the total number of domain requests is one of the features available in the flow but not in the individual packet (Haddadi & Zincir-Heywood, 2015). The use of traffic flow helps to reduce both the training time and the number of process instances. Even though the per-packet analysis is accurate, it requires extensive resources and cannot efficiently deal with encrypted network traffic (Zhao et al., 2013).

Additionally, to avoid being misled while building the rule model, the rule extraction process will remove the source IP address feature used for flow creation since the source IP address in the actual traffic might differ from data collection traffic.

Furthermore, the dataset is presented as a grouped aggregated flow. For a unified grouped aggregated flow time during the calculation of the computed features, the predefined time used for each calculated group is 5 s based on previous studies (Alieyan, 2018; Qi et al., 2018). Additionally, by aggregating the flow in a fixed interval of 5 s, the dataset size and the processing time are reduced. Table 6 tabulates the extracted set of basic features with enriched features.

Stage 3: hybrid rule-based detection model

This stage presents a hybrid rule-based detection model to detect botnet attacks in DNS traffic. The hybrid-rule model is built using the PART and JRip machine learning algorithms. To properly assess the proposed approach’s performance, a ten-fold cross-validation method (Kohavi, 1995) is utilised to select the best model for rule detection.

The PART classification algorithm is a Java-based variation of the C4.5 algorithm (Salzberg, 1994; Thankachan, 2013) and different SVM kernels (Hsu, Chang & Lin, 2003; Chang & Lin, 2011). C4.5 is a popular decision tree supervised classifier widely used in data mining. The C4.5 decision tree is generated based on the provided classes and feature sets (Alazab et al., 2011).

JRip (Repeated Incremental Pruning) is the Weka variant of Repeated Incremental Pruning to Produce Error Reduction (RIPPER), suggested by William W. Cohen as an enhanced version of IREP (Hall et al., 2009). JRip offers a range of capabilities that could improve detection accuracy, such as a technique to revise and replace generated rules, deal with noisy data, and fix over-the-counter issues. In addition, JRip optimises the rule set by the re-learning stage, leading to higher accuracy as the rules are regularly revised. Its classifier performs well even for imbalanced class distribution (Hall & Joshi, 2005; Qazi & Raza, 2012; Napierala & Stefanowski, 2016).

In this study, we selected PART and JRip machine learning classifiers for several reasons. Firstly, JRip and PART are sets of non-complex rules and could be integrated easily with any IDS system. Secondly, even though other classification algorithms are available, JRip and PART classifiers are used by many researchers in their recent work (Faizal et al., 2018; Kumar, Viinikainen & Hamalainen, 2018; Adewole et al., 2019). Thirdly, the proposed approach assumed that the hybridisation of the two classifiers would improve the output result; thereby, the final detection model rule is a hybrid of extracted rules from both PART and JRip output. Both JRip and PART classifiers require a training dataset. The extracted model for each classifier output, including the hybrid set of rules, is evaluated using 10-fold cross-validation. Figure 12 illustrates the process of the proposed hybrid rule-based model for the detection of DNS-based botnets.

Implementation environment

The software used includes Microsoft’s Windows 10 (64-bit) operating system, WEKA version 3.8, Microolap TCPDUMP for Windows® 4.9.2, Wireshark 3.02, and Python 2.8.

We also utilised the WEKA tool to extract the detection rules using the built-in JRip and PART algorithms. It is a set of machine learning algorithms for different data-mining tasks, such as data pre-processing, classification, and clustering.

In addition, Microolap TCPDUMP for Windows®, a network traffic sniffer and analyser software, was used to extract DNS traffic from the benchmark dataset. Wireshark is a network protocol analyser tool used for detailed analysis and basic feature extraction of DNS packets. We used a python script in conjunction with Wireshark to calculate the new enriched features. The results of feature extraction, tabulated in Table 6, were stored in a comma-separated values (CSV) file. Furthermore, having the final training file in CSV file format ensures seamless compatibility since it is fully supported and readable by WEKA.

Finally, The hardware used in this study consists of a CPU with an Intel® Core™ i5-8250u processor, 8 GB of memory, and a 256 GB Solid State Drive (SSD) hard disk.

Benchmark datasets

The experiment of this research is validated using two benchmark datasets: Network Information Management and Security Group (NIMS) dataset (Haddadi & Zincir-Heywood, 2016) and CTU13 dataset (Garcia et al., 2014).

The NIMS dataset by the Network Information Management and Security Group of Dalhousie University in Halifax, Nova Scotia, Canada, contains four distinct traces: a normal traffic trace based on Alexa domain ranks and three different traces of malicious traffic from Citadel, Zeus, and Conficker botnets. Table 7 lists the number of domain names inside the dataset for each trace.

The experiment in this study utilised the regular DNS traffic data within the CTU13 dataset (“Index of/publicDatasets/CTU-Normal-4-only-DNS,” 2016; https://mcfp.felk.cvut.cz/publicDatasets/CTU-Normal-4-only-DNS/). The CTU13 dataset contains 5,966 normal DNS traffic packets. The dataset comprises traffic collected from music streaming service 20songstogo.com, Gmail, Twitter, and regular web surfing via the Google Chrome browser.

Recently, many researchers used the CTU13 dataset in their work (Haddadi, Phan & Zincir-Heywood, 2016; Chen et al., 2017; Pektaş & Acarman, 2019).

The non-malware traffic used in this experiment is from the normal part of CTU13, which is CTU4 and CTU6 (“Malware Capture Facility Project: Normal Captures—Stratosphere IPS”; https://www.stratosphereips.org/datasets-normal). The normal traffic for CTU4 is from a home computer network and includes only regular DNS traffic for privacy reasons. Similar to CTU4, the CTU6 comprises regular DNS traffic generated from a Linux-based notebook in a university network.

Finally, for our static analysis purpose, two enriched datasets were extracted using feature engineering. The first dataset is a mixed dataset that combines both NIMS and CTU13 (normal traffic) datasets, and the second dataset is based only on NIMS datasets. The combination of normal traffic is to reduce overfitting resulted from an imbalance class. Figure 13 shows a sample snapshot of training dataset instances.

It can be noticed that the datasets used for evaluating our proposed approach were from 2014 and 2016. However, using these datasets will not impact the presented result for the following reasons: (i) in our approach, we analysed botnet’s DNS communication patterns, which are totally different from human DNS communication. There is no newer dataset publicly available that fulfils our requirement (DNS-based botnet traffic), and (ii) these datasets were also used by other researchers in their works (as recent as 2020) that we are comparing with. Therefore, we also need to benchmark our proposed work using the same dataset for fair evaluation and comparison.

Furthermore, our proposed work relies on the core DNS features that will always exist in the DNS-based botnet lifecycle, which remains the same as long as it uses the conventional DNS protocol. Therefore, the use of these datasets should not render our proposed approach ineffective in detecting novel or future DNS-based botnets.

Design of the proposed technique

The design of the proposed technique, illustrated in Fig. 9, consists of three stages. This section describes the design of each stage.

Design of pre-processing stage

In this stage, first, the TCPDUMP tool selected and filtered DNS traffic from the network traffic, which reduced network traffic by 68%. This process will reduce the time and resources needed to analyse the remaining traffic. Then, several Wireshark DNS packet filters are used to extract several basic features from the DNS traffic.

Table 8 shows the extracted features and the corresponding Wireshark filters used. The basic extracted DNS features are stored in a CSV file as input for the next stage.

Design of DNS traffic analysis stage

In this stage, the enriched features are calculated based on the basic extracted DNS features from the previous stage. The datasets had been prepared and normalised to calculate the features as tabulated in Table 6.

The first feature is the average randomness in queried domain names (F1), calculated using Shannon entropy, and as described in Section “Materials & Methods”, the queried domains are aggregated according to the source IP address (src_IP) every 5 s. Then, a python script is used to compute the enriched features, including the average entropy (avg_domain_ent) as per Eq. (2).

To calculate the second enriched feature (F2), several Wireshark filters are used in the process. The successful response (dns.sec.resp) is extracted using (dns.flags.rcode == 0) filter; the number of DNS requests (dns.req.num) is extracted using the (dns.flags.response == 0) filter; and both (dns.sec.resp) and (dns.req.num) are aggregated for each 5-second period using (src_IP). The ratio of successful response is calculated using Eq. (3) where the aggregated successful response is divided by the aggregated number of requests.

For the third enriched feature (F3), the DNS query packet is extracted using (dns.flags.response == 0) filter every 5 s. The entropy of the DNS query is calculated using Eq. (4). For the fourth enriched feature (F4), the resolved DNS records number is extracted using (dns.qry) filter. The result is calculated every 5 s using Eq. (5).

For the fifth feature (F5), the value of TTL response is extracted using (dns.resp.ttl) filter; then, the average response TTL is calculated using Eq. (6).

The rest of the features from F6 to F10 are calculated by following the same methods of using Wireshark filters, as shown in Table 8. The calculated DNS features are prepared as input for the next stage and stored in a CSV file. It is then considered as a labelled training dataset with only new DNS features. Table 9 shows the final number of dataset records after performing the flow aggregation.

Design of rule-based detection stage

In this stage, the Weka tool is used to extract botnet-based DNS detection ruleset using both PART and JRip classifiers. Initially, the enriched training dataset is the input for both PART and JRip classifiers. Then, to properly assess the predictive performance and overcome any bias in this process, the k-fold cross-validation training technique is used with the value of k set to 10 to build and test the model (Luo et al., 2017). Figure 14 illustrates the rules extraction process in this stage. Appendices A1, A2 and A3, provide in details the extracted rules for each used classifier.

Result comparison

Haddadi et al. (2014) proposed an approach for botnet detection and tested its performance against NIMS dataset (Haddadi et al., 2014). Later research conducted by the same researchers (Haddadi et al., 2014) used two methods during the pre-processing stage: (1) without using hypertext transfer protocol (HTTP) filters; and (2) using HTTP filters. The first method yielded an 87.5% botnet detection accuracy, while the second method obtained 91.5% accuracy. However, since our proposed approach was not using HTTP filters, we only compared our results with the first test case (Haddadi et al., 2014). Table 11 shows the comparison results.

Like our methodology, Deepbot (Shi & Sun, 2020) also used a hybrid model. It utilised RNN and LSTM algorithms to extract hybrid models for botnet traffic classification. However, despite extracting only 11 DNS features compared to 35 network traffic features by Shi & Sun (2020), our study obtained a better result (99.96% vs. 99.36%) with a higher F1 score of 99.97% vs. 98.4%. Table 12 shows the comparison results.

The proposed new enriched DNS features computed with the aid of information theory contributed to a higher accuracy rate. However, as discussed earlier, the low number of normal instances led to an FP rate of 5% for the NIMS dataset. Thus, to reduce the FP rate, the study used a mixed dataset that comprised a higher percentage of normal instances and successfully achieved a lower FP rate (1.6%).