EAD: effortless anomalies detection, a deep learning based approach for detecting outliers in English textual data

Introduction

Anomalies detection in the realm of data mining is the discovery of data points that deviate from the context of a document. The detection of anomalies is essentially used in different domains like business, human resource management and security, where the data comes from a simple textual script, customer review, or from a system log.

Being a classic problem in the domain of computer science (Mohaghegh & Abdurakhmanov, 2021), a number of algorithms have been proposed so far to gauge the deviation of data points from the baseline of a dataset (Chandola, Banerjee & Kumar, 2009). These purposely built models cover a number of areas, including fraudulent transactions (Garg et al., 2019), medical diagnosis (Zhang, Raghunathan & Jha, 2013), price management (Ramakrishnan et al., 2022) and network intrusion (Kwon et al., 2019).

The anomaly detection models can be categorized into supervised and unsupervised models. These models stumble when new anomalies are faced. In the case of supervised models, manual labeling is required, whereas with unsupervised models, outliers are detected based on specific attributes. Based on the encoding unit, the detection approach may either be character-based, word-based, or sentence-based. In character-level encoding (Mohaghegh & Abdurakhmanov, 2021; Rashid, Othman & Zainudin, 2019), outliers are detected based on the given character set. Such anomalies detection (AD) systems are suitable for short documents only. In the current era of artificial intelligence (AI), however, detection from larger datasets is needed. The word-level models are used for simple classification tasks like distinguishing spam and ham messages. For the word level outliers, typically, the bag-of-words (BOW) (Nowzohour et al., 2017) and the inverse-document-frequency (TF-IDF) (Spärck Jones, 2004) methods are used. Such systems fail to cope with the semantic meaning and context of a large corpus. The last category is sentence-level embedding, where neural network (N.N.) models are utilized (Bengio et al., 2003). Such systems are suitable for outlier detection based on the whole context of a script (Mikolov et al., 2013). Though the NN-based AD offers valuable insight, most of the system suffers from considering semantics (Arslan, Javaid & Awan, 2023). Moreover, the crucial flaws of such systems include extensive training and computational costs (Fan, Han & Liu, 2014). Hence, there is a need for an effortless method to deal with high-dimensional data with less computational cost.

This research work proposes an effective method for extracting anomalies in large-volume, high-variety data with promising accuracy. The model detects outliers in an unsupervised fashion-without labelled input and is implemented through a pipeline of two phases: the preparation phase and the application phase. Anomalies are determined based on contextual similarity with the centroid, as well as supporting novelty. In the preliminary phase, initially preprocessing is performed to remove noises from input data. Embedding is performed to have the latent space for computing centroid. The pre-trained model MinLM-l6-v2 (Galli, Donos & Calciolari, 2024) is exploited to have sentence-level embedding. Dimensionality reduction is performed to reduce the high-dimensional space to a lower dimension. In the application phase, anomalies are detected using the centroids of embeddings. To avoid mistakenly treating novelty as an outlier, the Minimum Covariance Determinant (MCD) based approach is followed to assess the novelty of the input script. The two versions of outliers, Near-anomalies and Real-anomalies, are detected for efficient analysis of input data. A schematic of the proposed method, which is implemented and evaluated in the Python project-App. forAnomalies Detection (AAD), is shown in Fig. 1. The interface of the project is designed in tkinter (Moore, 2021), with self-explanatory buttons to select datasets and find outliers. The outliers are displayed both in textual and graphical form. For the assessments, two independent datasets, the 20 newsgroups dataset from scikit-learn (Pedregosa et al., 2011) and the Kaggle English SMS spam collection dataset (Shafi’I et al., 2017), are used in the two separate evaluation sessions. In the first evaluation session, irrelevant texts were added incrementally to the 20 newsgroups dataset to check the outcomes of the model. In the second evaluation, the system was evaluated using messages in English taken from the Kaggle dataset. As a whole, a satisfactory accuracy score of 94% with an average F1 score of 0.95 was obtained. The outcomes of the analysis revealed the applicability of the proposed system in various domains. Moreover, the method is easily extendable to the realm of education and literature for text classification and coherent text generation.

The rest of the article is organized into four sections. In the literature review section, research works related with the proposed method are discussed. Details about the proposed system are covered in the methodology section. Evaluation and analysis are presented in the result section. The last section is the conclusion, which summarizes the entire research with future direction.

Materials and Methods

The proposed method intends to detect outliers using the unsupervised approach. The technique consists of two main phases- the preliminary phase and the application phase. After preprocessing in the first phase, embedding is performed using the pre-trained model, all-MinLM-l6-v2 (Galli, Donos & Calciolari, 2024). Dimensionality reduction of the sentence level embedding is performed to reduce the high-dimensional space to a lower dimension. To avoid mistakenly treating novelty as an outlier, the MCD-based approach is followed to assess the novelty of the input script. In the second phase, the centroids of embeddings are computed. Outliers are determined both on novelty and on the basis of the contextual similarity with the centroid. The two threshold levels are set as Near-anomalies and Real-anomalies, respectively, to detect possible and actual anomalies. Unlike other outlier detection systems (Schmidl, Wenig & Papenbrock, 2022; Boutalbi et al., 2023; Bobur et al., 2020; Tharshini, Ragavinodini & Senthilkumar, 2017), the method utilizes the cutting-edge all-MinLM-l6-v2 model for effective tracing of the anomalies in an unsupervised manner. Moreover, the two datasets containing text of different sizes and scripts are used to make the system applicable for general-purpose outlier detection. Details of each sub-process are given as follows.

Sentence level embedding

Sentence embedding is the encoding of semantic information in a vector of real numbers. It is the numeric representation of text that can be used for semantic similarity and for the detection of outliers. The effective all-MiniLM-l6-v2 model is utilized to get the optimal sentence-level embedding. The all-MiniLM-L6-v2 is a pre-trained sentence embedding model that represents sentences in 384-dimensional dense vector space (Wang et al., 2020). It is comparatively a fast sentence embedding model with a satisfactory accuracy rate (Wilianto & Girsang, 2023). The architecture is much like the structure of the transformer models, i.e., BERT or RoBERTa. However, there are fewer layers and smaller hidden neurons in the model. There are a total of six transformer layers and 12 self-attention heads. With the self-attention heads, the relationship of each token in a sequence is captured. For effective training, the residual connections are used to connect each output component back to its original input. The sub-layer, Feed Forward Neural Network FNN, is applied to each token to learn complex relationships and make the model more robust. Lastly, layer normalization is performed to normalize activation and convergence. The architecture of the all-MiniLM-L6-v2 is given in Fig. 2.

Let $S=\left\{s_1,s_2,..,s_n\right\}$ be the input text with ‘n’ sentences, the embedding $E=\left\{e_1,e_2,\dots ,e_n\right\}$ is computed with the embedding function £ as, E = £(S). The pre-trained model effectively attains important semantic information contained in the text.

Centroid encoding

The centroid represents the crux context of a document (Brokos, Malakasiotis & Androutsopoulos, 2016). Sentences whose embeddings are closer to the centroid mean that the sentences are closely related to the context. In an n-dimensional embedding vector space, the closeness of sentences to the centroid is measured in terms of angles, as shown in Fig. 3.

In the proposed method, centroids for the embedding vectors of the individual classes are computed. The centroid ‘C’ of an embedding vector $E=\left(e_1,e_2,\dots e_n\right)$, is mathematically represented as;

(5) $$C={\displaystyle \frac{{\sum }_{i=1}^{n}ei}{n}.}$$

If $v_j$ represents the embedding vector of j^th class; $v_j=\left[x_{i_1},x_{i_2},\dots x_{i_m}\right]$, then the Centroid List (CL) is computed as,

(6) $$CL=\left[{\displaystyle \frac{\sum x_{i1}}{d},{\displaystyle \frac{\sum x_{i2}}{d},.,.,{\displaystyle \frac{\sum x_{im}}{d}}}}\right]$$where $\sum x_{i_1}$ represents the sum of i^th token of all vectors in a particular set $v_j$ and ‘d’ being the total number of vectors.

Outlier detection

The designed Real and Near Outlier (R&N) algorithm, as shown in Algorithm 1, is followed to detect actual and near outliers. The function $f_{R\mathrm{\&}N}\left(\mathrm{E},\mathrm{C}\mathrm{L},\mathrm{N}\mathrm{S}\right)$ returns outlier by taking Embedding E, Centroid list CL and Novelty score N.S. as input. The formula for real and near outlier is given as follows.

(8) $$\mathrm{N}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{\_}\mathrm{O}\mathrm{u}\mathrm{t}\mathrm{l}\mathrm{i}\mathrm{e}\mathrm{r}=\mathrm{D}\mathrm{i}\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}\mathrm{f}\mathrm{r}\mathrm{o}\mathrm{m}\mathrm{c}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{r}\mathrm{o}\mathrm{i}\mathrm{d}+2\times \mathrm{N}\mathrm{o}\mathrm{v}\mathrm{e}\mathrm{l}\mathrm{t}\mathrm{y}$$

(9) $$\mathrm{R}\mathrm{e}\mathrm{a}\mathrm{l}\mathrm{\_}\mathrm{O}\mathrm{u}\mathrm{t}\mathrm{l}\mathrm{i}\mathrm{e}\mathrm{r}=\mathrm{D}\mathrm{i}\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}\mathrm{f}\mathrm{r}\mathrm{o}\mathrm{m}\mathrm{c}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{r}\mathrm{o}\mathrm{i}\mathrm{d}+3\times \mathrm{N}\mathrm{o}\mathrm{v}\mathrm{e}\mathrm{l}\mathrm{t}\mathrm{y}.$$

Algorithm 1:

Algorithm for computing real and near anomalies.

DOI: 10.7717/peerj-cs.2479/table-6

An embedding data point- $e_i\in E$, is considered to be within the context if $f_{R\mathrm{\&}N}\left(e_i,CL,NS\right)<Nea{r}_{Outlier}$.

Results

The proposed method is implemented in a case-study project-AAD in the VS-code with the Python libraries. For the implementation and evaluation of the project, a Corei7 laptop with 16 GB RAM, 4 GHz processor, 25 M.B. Cache, and RADEON graphics card was used. Details about the AAD project are given in the following subsection. The interface of ADS is designed in tkinter, which contains different buttons of self-explanatory captions, see Fig. 4. The AAD’s friendly interface contains three buttons: ‘Select DataSet’, ‘Find Anomalies’, and ‘Exit’. On the click event of the button ‘Select DataSet’, a file containing textual data is to be selected. Once a dataset is selected, outliers are found by clicking on the ‘Find Anomalies’ button. After preprocessing, embeddings, centroid and novelty are computed. Besides graphical output, outliers in the text are displayed on the left side of the window. The outliers detected in the four classes of the 20 Newspaper dataset are shown in Fig. 5.

Performance of the proposed method was evaluated in two independent evaluation sessions. In the first evaluation session, irrelevant texts were added to the dataset incrementally to count occurrences of the outliers. For this purpose, the dataset was categorized into three sub-datasets: SD1–SD3. The sub-datasets by which the model was trained and the classes from which irrelevant text was borrowed are shown in Table 1.

For each SD, the system was assessed four times by adding different paragraphs with different numbers of words, as shown in Table 2.

It was observed that with the increase in the number of irrelevant texts, the extraction of outliers was increased accordingly. The linear relationship between the average detected outliers and average borrowed text for each of the SD is shown in Fig. 6. Similarly, more significant numbers of anomalies are detected if the value of the Near-outlier is low; see Fig. 7.

Considering the irrelevant data points as anomalous, the F1 score was computed as,

(10) $$F1={\displaystyle \frac{2\times Tp}{2\times TP+FP+FN}}$$where TP, FP, and FN represent True Positive, False Positive and False Negative, respectively. The satisfactory scores for sensitivity, specificity, precision and F1 for each of the S.D. are shown in Table 3.

As stated in Chicco & Jurman (2020), the Matthews correlation coefficient (MCC) is more reliable than the F1 score; for further assessment, the MCC score was computed using the given equation;

(11) $$MCC={\displaystyle \frac{\left(TP\times TN\right)-\left(FP\times FN\right)}{\sqrt{\left(\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}\right)\times \left(\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}\right)\times \left(\mathrm{T}\mathrm{N}+\mathrm{F}\mathrm{P}\right)\times \left(\mathrm{T}\mathrm{N}+\mathrm{F}\mathrm{N}\right)\left(\mathrm{T}\mathrm{P}\times \mathrm{T}\mathrm{N}\right)-\left(\mathrm{F}\mathrm{P}\times \mathrm{F}\mathrm{N}\right)}}}$$

A satisfactory MCC score of 0.6 was obtained, showing better performance of the model.

In the second evaluation session, the system was evaluated by SMS messages from the kaggle dataset (Shafi’I et al., 2017). Though the dataset has 5,574 English messages being tagged as ham (legitimate) or spam, 4K random SMS were selected. It was observed that 94% of the spam messages were correctly traced as outliers, see Table 4. To compare accuracy of the proposed method, relative analysis was conducted with the state-of-the-art systems, see Table 5. In the table, F1 score against the proposed model is mean of the individual F1 scores obtained for the subdatasets; $S{D}_i{|}_{i=1}^3$.

Conclusions

Detection of text anomalies plays a significant role in various domains for estimating customer satisfaction, product quality, anticipating security risks and other rare events. Normally, the outliers exhibit variations in customer likes-dislikes, security risk, or ambiguous financial transactions. Most of the proposed systems are either data-hungry or computationally costlier. With this work, a sentence-level anomalies detection method is introduced based on the cutting-edge model of all-MiniLM-L6-v2. The method makes use of embedding centroids to extract outliers in the textual data. Unlike other models, the MCD-based approach is followed to support novelty. Hence, novel sentences in the context are not treated as outliers. The method is evaluated in two separate sessions by two non-related datasets with mixed-type contents. For better assessment, non-related texts were added incrementally during the evaluation. Satisfactory accuracy, precision and F1 scores (0.95) were obtained, showing the applicability of the method in a wide range of applications, particularly in text classification and summarization. As no separate tools are used for auto-encoding, tokenization, and pooling, the method ensures less computational cost. Besides the detection of outliers, the model is suitable for the analysis of textual data in business and academic domains.

Moreover, the method is easily extendable to the realm of education and literature for text classification and coherent text generation. It was observed that an increase in the number of sentences in individual paragraphs depleted the performance of the method. In our future work, we aim to enhance the system to avoid the said limitation and to support an even larger dataset of the Reuters Corpus Volume I (RCV1)-the archive of over 800,000 stories.

Supplemental Information