BiLSTM-enhanced legal text extraction model using fuzzy logic and metaphor recognition

Holistic process

On top of the Bi-LSTM layers, we integrate a multi-head self-attention mechanism. Self-attention allows the model to focus on the most relevant parts of the input texts by calculating attention weights between each pair of words. The multi-head extension creates multiple sets of attention weights from different subspaces, enriching the model’s attentional capabilities. Specifically, we use eight parallel attention heads in our model.

The complete process of the proposed model is depicted in Fig. 1.

Figure 1 depicts the holistic process of our methodology. Our data comprises three interdependent elements, namely, the case description, applicable laws, and judgment results. The case description delineates the facts, evidence, and pertinent legal provisions, establishing the foundation for the application of laws and rendering judgments. The applicable laws construe and apply legal provisions and regulations intrinsic to the case, consequently determining the legal application and liability of the case. The judgment results embody the legal judgments made by the judge predicated on the facts, evidence, and applicable legal provisions. As can be seen from Fig. 1, the legal text is first preprocessed to obtain the corresponding feature vector, and then the Bi-LSTM network is introduced to obtain the feature vector of relevant information after model processing. After recognition and classification, the category of the legal text is finally output. To ensure a fair and equitable judgment, it is imperative to meticulously consider the case facts and evidence, apply legal provisions, and make sound judgments based on legal principles and precedents.

Bi-LSTM

During the data preprocessing phase, we initially preprocess the textual content of the judgment and encode it into a vector ${\mathit{x}}_i$. Subsequently, we employ an attention mechanism to amalgamate the distinct legal elements’ representations in the judgment result, culminating in the comprehensive vector representation of the judgment result.

(1) $${\nu }_i=\{\begin{array}{c}{\mathit{W}}_x{\mathit{x}}_i+{\mathit{b}}_x=0\\ -inf=1\end{array}$$

(2) $$a_i={\displaystyle \frac{\exp \left(v_i^{\mathrm{T}}{\mathit{u}}_x/\sqrt{d}\right)}{{\sum }_i\mathrm{e}\mathrm{x}\mathrm{p}\left(v_i^{\mathrm{T}}{\mathit{u}}_x/\sqrt{d}\right)}}$$

(3) $$c=\sum _i{a}_i{h}_i.$$

Here, ${\mathit{W}}_x$, ${\mathit{b}}_x$ and ${\mathit{u}}_x$ correspond to parameter matrices. The vector representation of the input legal element, ${\mathit{x}}_i$, is obtained through encoding, where $d$ represents the dimension of the word vector in the input sequence. Additionally, $c$ denotes the comprehensive vector representation of all legal elements $h_i$.

Keeping this in mind, we have utilized the Bi-LSTM model to extract information from the judgment texts present in our dataset. Bi-LSTM, a type of bidirectional recurrent neural network, comprehensively analyzes the contextual information in the text by considering not only the context information before the current time step but also the context information after the current time step. In addition, the shared feature representation is realized through the shared bidirectional LSTM layer to improve the classification performance of the model. Specifically, Bi-LSTM comprises two LSTM networks, one processing the input sequence in a chronological order and the other processing it in a reverse order. For legal text data, information can be extracted from two different reading directions, one is from front to back, one is from back to front, and finally the output of the two networks is connected into a comprehensive output, which is used as the input of the next layer. Please refer to Fig. 2 for a diagrammatic representation of the Bi-LSTM model.

The initial layer in the diagram exemplifies a forward LSTM block, where the resulting output $h_t$ relies on both the current input $x_t$ and the prior output $h_{t-1}$. The subsequent layer in the diagram illustrates a backward LSTM block, in which the output $h_t$ is influenced by the current input $x_t$ and the succeeding output $h_{t+1}$. Hence, the present output $h_t$ is determined by $x_t$, $h_{t-1}$ and $h_{t+1}$. The computation formula for $h_t$ is expressed by Formula (4).

(4) $${\overrightarrow{h}}_t=LSTM\left(h_{t-1},x_t\right)$$

(5) $${\overleftarrow{h}}_t=LSTM\left(h_{t+1},x_t\right)$$

(6) $$h_t=\alpha {\overrightarrow{h}}_t+\beta {\overleftarrow{h}}_t.$$

The symbols $\mathrm{\alpha }$ and $\mathrm{\beta }$ in the equation represent weight coefficients.

Knowledge graph embedding

To enrich the semantic understanding of legal text data, we incorporated domain-specific knowledge using a knowledge graph embedding technique. The knowledge graph, constructed from legal documents and structured legal databases, captures the relationships between legal terminologies, case descriptions, and regulations. For this purpose, we utilized the TransE algorithm to embed the nodes and edges of the knowledge graph into a low-dimensional vector space.

In the TransE algorithm, the goal is to map entities (nodes) and relations (edges) in a knowledge graph into a low-dimensional space such that for a triple ( $h,r,t$), where $h$ is the head entity, $r$ is the relation, and $t$ is the tail entity, the following equation holds:

(7) $$\mathrm{h}+\mathrm{r}\approx \mathrm{t}$$where $\mathrm{h},\mathrm{r},\mathrm{t}$ are the vector representations of the head, relation, and tail entities in the low-dimensional vector space. The vector space is learned by minimizing the distance $d$ between $\mathrm{h}+\mathrm{r}$ and $\mathrm{t}$.

To learn the embeddings, TransE uses a margin-based ranking loss function:

(8) $$\mathcal{L}(h,r,t)=\sum _{(h^{\mathrm{\prime }}r,t^{\mathrm{\prime }})\in \mathrm{D}}\mathrm{m}\mathrm{a}\mathrm{x}(0,\gamma +d(\mathrm{h}+\mathrm{r},\mathrm{t})-\mathrm{d}({\mathrm{h}}^{\mathrm{\prime }}+\mathrm{r},{\mathrm{t}}^{\mathrm{\prime }}))$$where $\gamma$ is a margin to separate positive and negative triples, $d$ is a distance measure (e.g., Euclidean distance) between the head + relation vector and the tail vector.

To align the knowledge graph embeddings with text embeddings, a common approach is to use cosine similarity. If ${\mathrm{e}}_{text}$ is the embedding vector of the text and ${\mathrm{e}}_{kg}$ is the embedding vector of the knowledge graph, the alignment loss can be represented as:

(9) $${\mathcal{L}}_{\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{g}\mathrm{n}}=1-{\displaystyle \frac{{\mathrm{e}}_{text}\cdot {\mathrm{e}}_{kg}}{\parallel {\mathrm{e}}_{text}\parallel \parallel {\mathrm{e}}_{kg}\parallel }}$$where ${\mathrm{e}}_{text}\cdot {\mathrm{e}}_{kg}$ is the dot product of the text and knowledge graph embeddings, and the denominator normalizes the vectors to compute the cosine similarity. The goal is to minimize the alignment loss to ensure the consistency and complementarity between the two types of embeddings.

Self-attention mechanism

After utilizing Bi-LSTM to extract contextual semantic features, we can deduce the legal article category to which the text vector pertains. In order to improve the recognition efficiency of key sentences, we add the self attention module after the Bi-LSTM feature extraction and before the full connection layer. Self-attention is an extensively employed attention mechanism within natural language processing, serving to compute the interdependence amid elements within the input sequence. It facilitates calculating the extent of correlation between each element and other elements within the input sequence, thereby enabling a more comprehensive acquisition of the semantic information embedded within the input sequence.

The self-attention heads equip the model with an awareness of text structure and inter-dependencies between components of the legal text. This facilitates extracting pertinent features and making inferences about the semantic meaning more accurately. For example, the model can relate keywords in the case description to relevant statutes in the applicable laws section to predict appropriate judgment results.

During training, the datasets for case descriptions, applicable laws, and judgment results are fed into the model. The Bi-LSTM layers extract semantic features from the word sequences in each section. The self-attention head then identifies the most relevant interactions between features, which can establish global dependencies and expand the ability to extract text data. Finally, the representation vectors of the three parts are aggregated to make the final judgment prediction. Among them, the Bi-LSTM network model can better process the relevant data for the data set of case description, applicable law and judgment results. The advantage of self-attention block is that it can effectively combine elements at different positions in the input sequence, while considering their relative importance, so as to capture the interrelationship between elements in the sequence (Pan et al., 2022).

Taking into account the dataset’s properties, we established the experimental configuration as follows: num_heads was set to 8, while query_dim, key_dim, and value_dim were set to 64. Additionally, output_dim was assigned a value of 128, and the dropout_prob was set at 0.1.

Experimental setting

The implementation of the BiLSTM-enhanced legal text extraction model was conducted using a Linux-based operating system, specifically Ubuntu 20.04 LTS. The hardware setup included a high-performance Intel Core i7 processor, complemented by an NVIDIA RTX 3080 GPU with CUDA support to accelerate the training and inference processes. The system was equipped with 32 GB of RAM to handle the computational demands of large datasets and a 1 TB SSD for efficient data storage and quick access to model checkpoints. The software environment was built around Python 3.8, utilizing TensorFlow 2.x as the primary deep learning framework, with supporting libraries such as NumPy and Pandas for data manipulation, Scikit-learn for preprocessing and evaluation metrics, and Matplotlib for visualizations. GPU acceleration was facilitated through CUDA and cuDNN, while Jupyter Notebooks provided an interactive platform for experimentation and result analysis.

Dataset

In the experiment, we utilized a dataset (doi: 10.5281/zenodo.11057826) procured from the internet, comprising of 487 judgments concerning the governance of online environments. The dataset was classified into three distinct categories of charges, each of which was further subcategorized based on the severity of the judgment results, totaling to six categories. For the purpose of simplicity in nomenclature and computation, we represented the three categories of charges as a, b, and c, while the less severe judgment results were denoted by 0, and the more severe ones were represented by 1.

Data preprocessing

During the data preprocessing phase, a series of meticulous steps were implemented to prepare the legal text data for effective input into the Bi-LSTM-based extraction model. First, the raw legal text was normalized by removing extraneous spaces, special characters, and unnecessary formatting to ensure uniformity and cleanliness of the data. The text was then tokenized, splitting the continuous text into individual words or phrases to facilitate subsequent feature extraction. After tokenization, stop words (e.g., common function words like “is” or “and”) were removed to reduce noise and allow the model to focus on meaningful content.

Next, Word2Vec was employed to encode the tokenized text into numerical vectors. These word vectors preserved semantic relationships among words and helped capture contextual information. During this process, term frequency filtering was applied to remove overly frequent or rare terms, which could potentially distort the learning process. Subsequently, the data was organized into three core elements: case descriptions (facts and evidence), applicable laws (specific legal provisions), and judgment results (conclusions based on facts and laws). This structured categorization enabled the model to extract features from multiple dimensions.

To capture long-range dependencies and enhance semantic understanding, a self-attention mechanism was integrated prior to feature extraction. This mechanism calculated weighted relationships between each word in the input sequence, allowing the model to focus on significant information and generate a comprehensive vector representation. To further improve the model’s generalization capability, L2 regularization was applied during feature preparation to mitigate overfitting. Additionally, all feature vectors were normalized to ensure uniform data distribution and stable feature representation.

The dataset was segregated into a training set and a test set, maintaining a 7:3 ratio, with 341 and 146 samples, respectively. Table 1 illustrates the distribution of samples across each category.

To address the issue of dataset size, we employed data augmentation techniques to enhance the dataset and improve the robustness of the model. Specifically, we implemented the following augmentation strategies. (1) Synonym replacement: Randomly selected words in the case descriptions were replaced with synonyms based on a predefined legal thesaurus, preserving the semantic meaning while diversifying the dataset. (2) Paraphrase generation tools were used to create alternate sentence structures for case descriptions and applicable laws, ensuring variety while retaining legal and contextual accuracy. (3) Longer legal texts were divided into smaller segments, while shorter case descriptions were concatenated with related supplementary data, providing a wider variety of sentence lengths and structures. These techniques increased the dataset size from 487 samples to 1,461 samples, achieving a threefold expansion of the data. By diversifying the dataset, we aimed to improve the model’s ability to generalize and handle variations in legal language, as shown in Table 2.

Results

We utilized the t-SNE algorithm to diminish the dimensionality of text-encoded vectors in the dataset, in order to visually demonstrate the disparities present within. The resultant visualizations are depicted in Fig. 3. As shown in the figure, there are clear boundaries between the different categories, especially in the upper right and lower left corner, with relatively dense data between points, compared to relatively small data boundaries in the middle and scattered data between points.

The vector resulting from encoding the text and subjecting it to Self-Attention computation is predisposed towards key phrases. To avert overfitting of the text vector’s classification results, we incorporated the L2 regularization method into the loss function. The resulting modified cross-entropy loss function is presented in Formula (10).

(10) $$Loss=-\sum _{i=0}^n{y}_{i}log\left({\hat{y}}_i\right)+\lambda \sum _{i=0}^n\parallel w_i{\parallel }_2^2.$$

In the given formula, the symbol ${\hat{y}}_i$ denotes the anticipated classification of the degree of unlawfulness pertaining to the case description, $y_i$ signifies the actual value present in the dataset, $\lambda$ stands for the regularization coefficient, and $w_i$ corresponds to the weight parameter of the model. Upon conducting several iterations, we ascertained that establishing the regularization coefficient $\lambda$ as 0.01 facilitates the attainment of superior outcomes.

After preprocessing the data, we can obtain the feature vectors for legal texts. The training dataset comprises the feature vectors for case descriptions and their corresponding judgment categories. These vectors can be utilized to train the Bi-LSTM model. The variation of the loss function during the training process of the Bi-LSTM model with increasing training epochs is demonstrated in Fig. 4.

From the depicted graph, it can be observed that the loss function value diminishes with the increase of training epochs during the model’s training phase. Initially, owing to the random parameter initialization, the loss function value is relatively substantial, approximately 3.5. Nevertheless, the rate of decline of the loss function is swift, and it stabilizes after approximately 60 epochs. Subsequently, it gradually diminishes and attains the minimum value at 152 epochs. At this juncture, the model exhibits commendable proficiency in category prediction.

Subsequent to the training, the model’s efficiency is evaluated using the test set data, and the corresponding confusion matrix is presented in Fig. 5.

The figure illustrates that the case classification problem tackled in this article pertains to a multi-classification predicament, with a confusion matrix size of 6 * 6. Notably, the article utilizes macro-F1 as the evaluation metric to assess the model’s clustering efficacy. Macro-F1 is derived from the F1-score of binary classification. The approach entails dividing the multi-classification confusion matrix into multiple binary classification matrices, based on the binary classification method’s “yes” and “no” criteria, computing several F1-scores, and subsequently averaging them to obtain the Macro-F1 value (Sun et al., 2020). The formula for macro-F1 calculation, as shown in Formula (11), is utilized.

(11) $$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}={\displaystyle \frac{TP}{TP+FP}}$$

(12) $$Recall={\displaystyle \frac{TP}{TP+FN}}$$

(13) $$F_1-score={\displaystyle \frac{2\times Recall\times Precision}{Recall+Precision}}$$

(14) $$Macro-F_1=mean\left(F_{1\mathrm{\_}i}\right).$$

In this context, $mean$ refers to computing the arithmetic mean, while $F_{1\mathrm{\_}i}$ denotes the F1-score of the $i$-th binary classification matrix. Utilizing the confusion matrix illustrated in Fig. 5, it can be converted into several binary classification matrices.

Precision, recall, and F1-score are computed for each of the binary classification matrices, and subsequently, the evaluation values of the multiple binary classification matrices are averaged to derive the evaluation standard for the multiple problems.

Table 3 exhibits the classification efficacy of the model on the test set, predicated on diverse binary confusion matrices. The test set encompasses 146 samples, and the evaluation metrics for distinct categories are delineated in the table. The outcomes evince that the model’s performance is optimal in categorizing c1, with an accuracy of 0.9091. This is owing to the fact that c1 corresponds to the most severe law and encompasses the most potent linguistic depiction in the judgment text.

To compare the performance of different methods, we also deployed other information extraction and classification methods and conducted ablation experiments specifically for the proposed model in this article. Table 3 and Fig. 6 show their respective evaluation results.

VGG (Visual Geometry Group) (Koonce & Koonce, 2021): Suitable for basic visual tasks such as image classification, using a convolution kernel of small size (3 × 3) and deep network structure.

Inception V3 (Wang et al., 2019): Convolution operations using multiple convolution nuclei of different sizes help capture features at different scales.

ResNet (Residual Network) (Targ, Almeida & Lyman, 2016): Uses residual blocks to allow information to flow directly to deeper networks by skipping connections, helping to train very deep networks.

BiLSTM (Bidirectional Long Short-Term Memory) (Siami-Namini, Tavakoli & Namin, 2019): A variant of recurrent neural networks (RNN) that helps capture contextual information in timing data by introducing hidden layers in both forward and backward directions.

Table 4 and Fig. 6 present the results of several methods evaluated by Macro-F1 as a performance metric. The outcomes signify that the Bi-LSTM model with Self-Attention, propounded in this article, surpasses the performance of other methodologies in predicting the classification of illicit content on online platforms. Compared to the VGG, Inception V3, and ResNet models, the Bi-LSTM-SA method proposed in this article discloses an improvement of 11.0%, 6.4%, and 4.0% in precision, 5.0%, 5.5%, and 5.2% in recall, and 4.8%, 2.0%, and 3.8% in macro-F1, correspondingly. In the ablation experiments, we employed the raw Bi-LSTM model for evaluation to demonstrate the effectiveness of the SA module. The experimental results showed that the evaluation performance of Bi-LSTM-SA was superior to that of Bi-LSTM. In summary, the experimental results confirm the effectiveness and robustness of the proposed model in this task. The Bi-LSTM-SA model proposed in this article can effectively learn the global dependency relationship of the text by adding self-attention mechanism on the basis of Bi-LSTM, focusing on the semantic information that is more important for the classification task, and thus achieving significant improvement in illegal content classification task compared with VGG, Inception V3, ResNet and the original Bi-LSTM model. This fully demonstrates the effect of self-attention mechanism on sequence modeling, as well as the innovation and effectiveness of the proposed model. This research provides valuable experience and further optimization directions for using deep learning technology to detect and filter illegal content on the Internet.

To further evaluate the interpretability of the proposed Bi-LSTM model with self-attention, we generated attention heatmaps for sample inputs, as shown in Fig. 7. These visualizations reveal how the model dynamically allocates attention weights to critical components of legal text, such as key terms, legal articles, and contextual phrases. For instance, in Heatmap 1, the model assigns high attention to terms like “violated” and “Article,” highlighting its focus on capturing the explicit relationship between case descriptions and applicable laws. In contrast, Heatmap 2 shifts its emphasis to “defendant” and “21,” demonstrating its ability to adapt attention based on the contextual significance of the legal entity and specific law sections. Heatmap 3 presents a more balanced distribution of attention across “Article” and “Penal Code,” reflecting the model’s capacity to process broader contexts for holistic legal analysis. Finally, Heatmap 4 shows concentrated attention on “21” and “Penal Code,” underscoring the model’s precision in referencing specific legal statutes. These results underscore the effectiveness of the self-attention mechanism in capturing nuanced semantic relationships in legal text, enhancing both classification accuracy and interpretability.