Effectiveness and optimization of bidirectional long short-term memory (BiLSTM) based fast detection of deep fake face videos for real-time applications

Bidirectional long and short-term memory networks and deep falsification techniques

In the network structure of BiLSTM, the forward propagation layer and the backpropagation layer are its core components, as shown in Fig. 1. These two layers work together to produce the final output by sharing weights (Song et al., 2024). Specifically, the network performs forward operations through the forward propagation layer and records the state of the hidden layer at each time point. Then, the backward propagation layer oppositely performs operations, capturing information from backwards to forward. The outputs from these two directions are finally fused in the output layer to form a comprehensive prediction for each point in time. Two LSTM layers with forward and backward orientation form the model structure, which analyzes video sequences in different directions. Multiple layers produce their output, which gets combined to deliver an entire video understanding. The training process included multiple essential strategies that used advanced Xception features to boost BiLSTM performance at detecting fine details in video frames (Hu et al., 2023). Temporal dependencies of the BiLSTM network received optimization through the implementation of CRFs. Multi-task convolutional neural network (MTCNN), together with OpenCV, allowed precise face detection and frame extraction to ensure the model received only important data. The training incorporated both cross-entropy loss as well as auxiliary classifiers to enhance its learning speed and generalization capabilities. Strategies implemented in this approach have enhanced the deepfake detection capabilities of the model while keeping operational expenses minimal.

The network contains two conduction layers, forward and reverse, connected to the output layer and share six parameters (W1 to W6) (Agarwal & Verma, 2020). The network first performs forward computation through the forward layer to record the output at each step. Then, the reverse layer processes it to capture the reverse information. The final output is the sum of forward and reversed outputs. Its mathematical representation is shown below:

(1) $$h_t=f\left(w_1{x}_t+w_2{h}_{t-1}\right)$$ (2) $$h_t^{\mathrm{\prime }}=f\left(w_3{x}_t+w_5{h}_{t+1}^{\mathrm{\prime }}\right)$$ (3) $$o_t=g\left(w_4{h}_t+w_6{h}_t^{\mathrm{\prime }}\right).$$

It contains the output $h_t$ of the forward propagation layer at time point $t$, the output $h_t^{\mathrm{\prime }}$ of the backward propagation layer at time point $t$, and the output $o_t$ of the output layer at time point $t$.

The core of deep forgery techniques is the generative adversarial network (GAN), which is constructed by two competing neural networks: a generator network, whose duty is to create realistic forgery data, and a discriminator network, whose core is to recognize these forgery data (Elaskily et al., 2021). In this process, the generator and the discriminator compete with each other through continuous iteration and optimization, and the generator network gradually improves the quality of the data it generates until it is difficult for the discriminator network to distinguish the authenticity. With the continuous advancement of technology, GAN can generate more realistic images and even deceive some advanced detectors (Samir, Youssef & Nagy, 2021). The branching structure of the GAN network is shown in Fig. 2.

Combining the sequence processing capability of BiLSTM and the generative capability of GAN (Wang et al., 2023), the research in this article aims to develop a fast detection method for deeply forged face videos for real-time applications. The method improves the speed and accuracy of detection by optimizing the model structure and training strategy while keeping the computational cost low.

BiLSTM model architecture

BiLSTM fundamentals

The BiLSTM is an advanced sequence analysis model that combines the strengths of both forward and backward LSTM networks, resulting in bi-directional enhancements in data parsing. This design allows BiLSTM to capture and analyze sequence information from both forward and backward directions simultaneously, significantly improving the model’s ability to predict and understand information (Wang et al., 2022). LSTM functions as a specialized version of RNN because it serves as a sequential data processing model that enables the retention of extended temporal connections. The network maintains a cell state while its gating mechanisms control information flow through forgetting and input and output gates. LSTM features the ability to memorize important information across extended sequences, which proves valuable, particularly when analyzing videos where temporal frame connections matter (Hu et al., 2022). The capability of BiLSTM to examine a sequence from both forward and backward directions improves its video content understanding because it secures contextual information from previous and subsequent frames.

The efficient functioning of the LSTM network relies on the sophisticated structure within it, including the input $X_t$ at time step $t$, the cell state $C_t$, the hidden layer state $h_t$, and the key forgetting gates $f_t$, input gates $i_t$, and output gates $o_t$ (Kumar et al., 2020).

The LSTM network computation process is shown in the following equation:

Calculate the oblivion gate:

(4) $$f_t=\sigma \left(W_f\times \left[h_{t-1},X_t\right]+b_f\right)$$

Compute the memory gate:

(5) $$i_t=\sigma \left(W_i\times \left[h_{t-1},X_t\right]+b_i\right)$$ (6) $$\widetilde{C_t}=\tanh \left(W_C\times \left[h_{t-1},X_t\right]+b_C\right)$$

Calculate the current moment cell state:

(7) $$C_t=f_t\times C_{t-1}+i_t\times {\tilde{C}}_t$$

Calculate the output gate and the current moment hidden layer state:

(8) $$O_t=\sigma \left(W_O\times \left[h_{t-1},X_t\right]+b_O\right)$$ (9) $$O_t=\sigma \left(W_o\times \tanh \left(C_t\right)\right)$$where $W_f,W_i,W_c,W_o$ are the respective weights, and $b_f,b_i,b_c,b_o$ are the offsets. The BiLSTM model architecture is shown in Fig. 3.

Model design

This article uses a BiLSTM to capture the subtle differences of the forged video between the front and back frames. By starting from the video’s temporal characteristics, BiLSTM can deeply analyze and understand the dynamic changes in the sequence, thus revealing the timing anomalies of the forged video. Meanwhile, this study will also use the CRF technique to optimize the output of BiLSTM to improve the accuracy and reliability of the discrimination. A conditional random field is a new type of timing model that can achieve the precise adjustment of bilinear short-term memory by capturing the correlation in timing data and then improving the accuracy of the discrimination. The selection of BiLSTM for this work is due to its exceptional performance when working with sequential data, particularly video frames, in deepfake face video detection tasks. BiLSTM can spot interesting movement directions within a video series better than single-frame systems thanks to its ability to analyze both past and future frame connections. The technique shows its strength by analyzing two-directional data patterns to identify small frame differences that can reveal deepfakes. Besides having both high detection accuracy and productive speeds, BiLSTM supports shorter processing time for real-time deepfake checks on devices with limited bandwidth. The long-term memory capabilities of BiLSTM allowed us to choose it because deepfake video manipulation often occurs across multiple video frames.

Figure 4 shows the overall architecture of the deep forgery video recognition proposed in this study, which integrates the timing analysis capability of BiLSTM and the sequence optimization technique of CRF to build a comprehensive set of detection strategies. The BiLSTM system processes video frames using its backward and forward dependency mechanism, yet Xception extracts the specific features. The CRF layer recognizes optimal temporal connections in BiLSTM outputs to maintain reliable predictions between successive video frames. This framework not only improves the detection accuracy of forged videos but also adapts to continuously updating forging techniques. It is versatile and adaptable.

Feature extraction

In the video preprocessing phase, the research selected a series of image sequences and used them as inputs to the research’s video frame feature extraction module. During this period, the study utilized an advanced network structure based on Xception (Sharma, Sharma & Kalia, 2023), which is a modified model of Inceptionv3. Xception successfully captured the fine-grained information in the video frames by independently processing the channel and spatial dimensions while incorporating techniques from deeply separable convolutional and residual networks (Jaleed et al., 2018).

In order to optimize and maintain the appropriate number of covariates, deep segmented convolution was employed to enhance the modelling efficiency and quality of the study and based on the results of a series of comparative experiments done with the current major recognizers (e.g., MesoNet, MISLne, and Inception-v3), it can be ascertained that the Xception architecture has the best expressive power. It achieves a high level of recognition and effectively captures rich elements of the original image characteristics.

Based on this observation, the Xception network was selected as the basic framework for the study to obtain feature information, and it was set up and optimized in detail. After several rigorous training sessions, the study successfully obtained a feature vector of 2,048 dimensions after global mean pooling. The exquisite design of the Xception network, which consists of three main parts: the input stream, the mid-level stream and the output stream, allows the study to obtain all-round and in-depth feature information.

Finally, the study uses the extracted video frame feature vectors as inputs to the Bi-LSTM network in the subsequent temporal analysis phase in order to understand and analyze the video content in greater depth.

Timing analysis

In the process of time series analysis, the model architecture adopted is shown in Fig. 5. The model takes the feature vector of the video frame sequence obtained in the feature extraction stage as the input. These feature vectors are then fed into a BiLSTM. In BiLSTM, the forward LSTM and the reverse LSTM operate independently and perform their computation processes separately.

Forward LSTM is responsible for the transmission of information from the beginning to the end of the sequence, capturing the forward dependencies in the time series. The reverse LSTM transmits information from the end of the sequence to the beginning in reverse order to capture backward dependencies. Through this bidirectional processing, the model can fully understand the dynamic changes in the sequence (Wu et al., 2023) and thus achieve more accurate and comprehensive predictions in time series analysis.

Processing data using the LSTM algorithm preserves key information and eliminates useless information, which helps in updating the state of each group of neuronal units (Xu, Park & Zhang, 2019). This allows long-distance correlations between feature vectors to be maintained, which in turn enables the feature vectors in consecutive frames to be combined with the contextual environment in which they are embedded for a deeper understanding of this inter-frame information. This information is then deeply mined using the BiLSTM technique, which finally gives the likelihood of the video being true or false in the form of a softmax function. Next, the study takes the independent probability values of the BiLSTM outputs as inputs into a CRF (Liu, Gao & Huang, 2021) model, and with the help of the model’s correlation characteristic function and transition probabilities, the study applies the Maximum Likelihood method of training, which yields the best possible output probability results.

Training strategies

Detection optimization strategies

The technique of combining BiLSTM with CRF provides a highly efficacious analytical tool in the field of sequence analysis (Chakraborty, Bhattacharyya & Ghosh, 2023). The predictions from BiLSTM benefit from CRF because they model the logical connections between successive video frames. CRF serves as a sequentially structured model because it evaluates each frame output through input information from the present time and neighbouring frame labels. Using CRF allows the production of consistent predictions throughout adjacent frames, thus enhancing video classification accuracy. CRF enables better temporal consistency in predictions through our study, which leads to increased model performance when detecting deepfake videos. The output of BiLSTM after being used for temporal analysis is the probability corresponding to each label. Further accessing the results after BiLSTM is done with a conditional random field, the results are optimized by the conditional random field to get the final identification result. The detection optimization approaches analyze two aspects of deepfake face video detection: accuracy rates alongside operational speed. A vital detection strategy consists of implementing CRF together with BiLSTM networks. By implementing CRF, the model becomes better at identifying real or fake videos through frame context analysis. The Xception network optimizes feature extraction by capturing detailed aspects of video frames to detect faint discrepancies in the content. The training process becomes more stable after implementing the batch normalization technique, which promotes faster convergence and minimizes both speed-related problems. The combination of these techniques enhances model accuracy and computational efficiency, thus enabling it to perform continuous video detection operations efficiently. CRF is a serialized labelling algorithm that finds the corresponding variable under a given condition. Given two random variables $X$, $Y$ and $X$ is the given condition, if the random variable $Y$ constitutes a Markov random field that can be obtained from an undirected graph $G(V,E)$, i.e., for any node $v$ in the undirected graph such that Formula (16) holds:

(16) $$P\left(Y_v|X,Y_w,w\ne v\right)=P\left(Y_v|X,Y_w,w\sim v\right).$$

In an undirected graph $G(V,E)$, $V$ represents the set of vertices and $E$ represents the set of edges. CRF is a statistical modelling approach for working with structured data, where $P(Y|X)$ denotes the probability of output $Y$ given input $X$. In a CRF model, each vertex $v$ is associated with a set of vertices $w$ connected to it that do not include $v$ itself.

The CRF model captures the characteristics of the data by defining a feature function. For example, for input $X$, the feature function $f(X,n,y_n,y_{n-1})$ can describe the relationship between the state $y_n$ at the current position $n$ and the state $y_{n-1}$ at the previous position $n-1$. This feature function allows the model to take into account information from neighbouring data points when classifying.

In CRF implementations, linear conditional random fields (LCM) are a common form. In the training phase, the LCM uses maximum likelihood estimation (MLE) or regularized MLE to learn the model parameters in order to predict the output sequence corresponding to the input sequence. In this way, the CRF model is able to effectively classify and label the input data while taking into account the interrelationships between data points. As shown in Fig. 6, the goal of the training process is to find model parameters that maximize the likelihood of the training data.

LCM (He et al., 2019) feature functions are classified into two categories: one is the nodal feature function $Fa(y_i,x,i)$ that depends only on the current node, where a range from 1 to $A$, where $A$ represents the number of nodal feature functions defined on the node $Y$, and $i$ represents the position of the current node in the sequence. The other class is the local eigenfunctions $Fb(y_{i-1},y_i,x,i)$ that depend on the current node and its predecessor, where b ranges from 1 to B, where B represents the number of all local eigenfunctions defined on the node, and i−1 represents the position of the previous node in the sequence.

In calculating the probability of a conditional random field it is actually a process of calculating the conditional probability and its mathematical expectation given the input sequence and output conditions. This calculation can be expressed as the product of the mathematical expectation function and the normalization factor. Essentially, a conditional random field is a log-linear model defined on time-series data, and its training process involves a log-likelihood function, as shown in Eq. (17).

(17) $$L\left(w\right)=L_{\hat{p}}\left(P_w\right)=\log {\prod }_{x,y}{P}_w{\left(y|x\right)}^{p\left(x,y\right)}$$

After processing by a BiLSTM network, a series of predicted values of video frames can be obtained. These predicted values are fed into a conditional random field, where the constraint relationships between the predicted values are automatically learnt through transfer matrix operations and the model is trained using maximum likelihood methods to reduce the error rate. For example, when a set of video sequences with feature vectors $\{K1,K2,K3,\dots ,Kn\}$ are fed into a BiLSTM network for analysis, each vector $K_i$ produces a set of predicted values $\{g_{i\mathit{1}},g_{i\mathit{2}}\}$. The conditional random field learns the predicted values $\{g{k}_{i\mathit{1}},g{k}_{i\mathit{2}}\}$ of each $K_i$ globally through a transfer matrix. It selects the best set of predicted values $\{g\mathit{1}\mathit{,}\mathit{g}\mathit{2}\mathit{,}\dots \mathit{,}\mathit{g}\mathit{k}\}$ as the prediction result of the current sequence features, thus obtaining the optimal prediction result of the whole video frame sequence.

Dataset and preprocessing

In this experiment, the study used the FaceForensics++ database, covering 1,000 source face videos posted on YouTube, where each segment is in the range of 10 to 15 s in duration. These face videos were faked using four unique methods: Deepfakes, FaceSwap, Fac2Face, and NeuralTextures. Deepfakes and FaceSwap focus on identity transformation, while Fac2Face and NeuralTextures work on facial expression reenactment. The research employed fake video content in greater numbers than real videos because deepfake detection tasks typically deal with unequal data ratios. Creating fake videos requires the production of an extended quantity of artificially generated content as opposed to real video material. The usage of additional fake images created opportunities for the model to discover subtle distinctions between authentic and counterfeit videos. Machine learning tasks often require this approach to benefit from a broad sample of training examples, particularly in cases where multiple manipulation detection situations are present in different datasets.

The experimental design of this study includes 1,000 real video samples and 500 videos for each of the four forgery techniques for the training and testing phases. All videos were processed with high-quality standards, as evidenced by a constant rate quantization parameter of 23, which ensures a clear and detailed representation of the videos. Some selected samples of the dataset are shown in Fig. 7, which not only demonstrates the diversity of the dataset but also reflects the rigour and depth of the experiments.

The study adopted a systematic sampling method to ensure the randomness and fairness of the experimental results. From 1,000 real videos and 500 videos generated by each of the four forgery techniques, the study uniformly selected 30 frames from each frame. Such a sampling strategy ensured that the study’s dataset remained representative and diverse in all situations (Yu et al., 2023).

Next, the study divided these frame images into a training set and a test set in a ratio of 7:3. Specifically, the training set contained 21,000 frame images from the real video and 42,000 frame images from the fake video. In comparison, the test set contained 9,000 frame images from the real video and 18,000 frame images from the fake video.

Video preprocessing

In the journey of identifying deeply forged face videos, the study first performs a well-designed video preprocessing step to ensure the accuracy and efficiency of the subsequent analyses:

Video labelling and frame extraction: the study clearly labelled the videos in the dataset, with 0 for deeply faked videos and 1 for real videos. Subsequently, with the power of Python’s opencv library, the study smoothly divided the videos into consecutive frames and saved them properly.

Face detection method selection: before entering the face interception stage, the study conducted a careful comparative analysis of two face detection methods, Dlib and MTCNN. The results show that MTCNN outperforms Dlib in both detection accuracy and speed, especially when dealing with faces with different skin colours; MTCNN demonstrates excellent performance and maintains high accuracy even in the detection of faces with black skin colour (Yao et al., 2018).

Face interception: given the combined advantages of MTCNN in face detection and keypoint localization, the study chose this method for fast and efficient interception of face regions.

Image saving: the study saved the intercepted consecutive face images in the original order of the videos in the dataset. In order to meet the demands of subsequent processing, all saved images were uniformly resized to 256 × 256 pixels to ensure consistency and ease of processing (Elaskily et al., 2020).

Experimental setup

The experiments were performed on a device equipped with an Intel Core i7-6000 processor, a processor frequency of 3.40 GHz, and 8 GB of video memory. In this process, the Python language and NumPy library were used to perform matrix computation tasks, and the TensorFlow framework was used to develop and optimize the convolutional neural network on the back end.

In the parameter configuration part of the experiment, the FaceForensics++ dataset is selected as the research object, and two core parameters are particularly concerned: batch size and convolutional kernel size.

Choosing the right batch size is crucial to ensure the accuracy of the gradient descent algorithm and improve the efficiency of model training. A reasonable batch size can enhance the memory ability of the model for training data and accelerate the learning speed. However, if the batch size is set too low, it may cause the loss of training sample diversity and affect the generalization performance of the model. On the other hand, large batch sizes can lead to unnecessary consumption of computing resources. Therefore, balancing the choice of batch sizes is critical to achieving effective model training. Table 2 provides data on the effects of different batch sizes on model performance, with bolded values representing the best performance. In particular, when the batch size is set to 32, the recognition rate of the model reaches the highest value, so 32 is chosen as the batch size for training.

The size of the convolution kernel directly affects the size of the output feature map. In theory, smaller convolution kernels are preferred, but when dealing with particularly sparse data, smaller convolution kernels may not adequately capture its features. Larger convolution kernels can capture more complex features, but they also increase the complexity of the model. Therefore, it is necessary to select an appropriate convolution kernel size. Table 3 shows the evaluation results of face recognition performance under different convolution kernel sizes, among which the convolution kernel size of 3 × 3 provides the best recognition effect. Hence, the study chooses 3 × 3 as the convolution kernel size.

When constructing a convolutional neural network (CNN), in addition to batch size and convolutional kernel size, there are several key parameters to consider, including learning rate, optimizer selection, pooling layer size, activation function, loss function, and kernel size. Through iteration and optimization of a series of experiments, the optimal configuration of these parameters was set in the study, as shown in Table 4. These carefully selected parameter settings ensure the model can perform best during training.

Performance evaluation

This research is devoted to solving the binary classification problem, aiming to distinguish the authenticity of a video by evaluating the detection probability, that is, to judge whether the video is real or has been deeply falsified. In order to comprehensively evaluate the validity of the proposed identification method, Accuracy is mainly used as the measurement standard. The calculation method of accuracy rate is described in detail by Formula (18).

(18) $$Accuracy={\displaystyle \frac{\left(TP+TN\right)}{TP+TN+FP+FN}}$$

The accuracy rate combines the following four key metrics (Thies, Zollhofer & Nießner, 2019).

In order to deeply analyze the impact of BiLSTM combined with conditional random fields and the convolutional block attention module introduced in the feature extraction phase on the discrimination effect, the study also used several other evaluation metrics, including precision, recall and F1 value. The precision rate, also known as the checking rate, refers to the proportion of correct predictions among all videos predicted to be deep forgeries, which is calculated as shown in Formula (19):

(19) $$Precision={\displaystyle \frac{TP}{TP+FP}.}$$

The recall rate, also known as the check-all rate, indicates the proportion of all actual depth-falsified videos that are correctly predicted and is calculated as shown in Formula (20):

(20) $$Recall={\displaystyle \frac{TP}{TP+FN}}$$

Given that high precision and low recall, or vice versa, may occur in the actual identification process, the study introduces the F1 value to consider precision and recall together. The F1 value is the reconciled average of precision and recall, providing a single metric that balances the two, and is calculated as shown in Formula (21):

(21) $$F1=2\times {\displaystyle \frac{Precision\times Recall}{Precision+Recall}.}$$

Analysis of experimental results

In this study, we develop a novel video discrimination technique that is grounded in the temporal inconsistency between video frames. Using a BiLSTM network, we are able to effectively capture inter-frame variations caused by depth forgery techniques, such as light source and resolution inconsistencies. To verify the effectiveness of our method, we compare its experimental results with the Xception network based on single-frame discrimination and the BiLSTM network without incorporating CRFs, and the experimental results are presented in Fig. 8.

As can be seen from Fig. 8, time series analysis is more effective than the traditional single-frame recognition algorithm in improving the accuracy of recognition of deeply tampered videos. Moreover, when CRF processing is applied to the output of BiLSTM, the accuracy of the model is improved by almost 1%. Although the precision (Precision) is not the highest, it also achieves a 1.6% increase compared to the BiLSTM-only approach. Meanwhile, recall (Recall) and F1-scores performed the best among the compared methods, reaching 90.00% and 87.09%, respectively, and these results further confirm the effectiveness and superiority of the discriminative approach combining BiLSTM and CRF.

In order to show the correctness of the method more clearly, the ROC curve was drawn based on the true positive rate and false positive rate, as shown in Fig. 9. The dotted red line in the chart is used to rate performance. At the same time, the area below (AUC) is set at 0.5, representing the accuracy of random guesses.

Located in the region above this baseline, the AUC value extends from 0.5 to 1.0, and a curve falling within this interval indicates that the identification method exhibits high accuracy and good performance. This is because a high AUC value means that the model can achieve a high rate of true positives at a lower rate of false positives. The region below the baseline, on the other hand, has AUC values ranging from 0.0 to 0.5, and a curve falling in this interval reflects a poor performance of the discriminative method, which has little value for practical application. The closer the AUC value is to 1 and the closer the curve is to the upper left corner, the better the classifier’s performance, which suggests that the model can effectively discriminate between positive and negative cases at all possible classification thresholds.

As shown in Fig. 9, the green solid line clearly presents the identification method proposed in this study that combines a LSTM with a CRF; the yellow line represents a BiLSTM-based identification technique that does not include a CRF; whereas the blue rectangular dashed line demonstrates an identification strategy that does not involve temporal analysis, i.e., it directly utilizes features extracted from Xception networks scheme for authenticity judgement through the fully connected layer. Comparing the data in Fig. 9, our discriminative method leads with an AUC value of 0.912, showing superior performance over the other two methods; followed by the discriminative method using only BiLSTM with an AUC value of 0.890, while the discriminative strategy employing the Xception network has the worst performance in terms of the AUC value, which is only 0.743.

This result is due to the strength of the LSTM network in capturing temporal inconsistencies between video frames and the role of the CRF in optimizing sequence classification accuracy. In contrast, the BiLSTM-only approach, although it also exploits timing information, does not have the global optimization capability of CRF, while the Xception network-only approach cannot analyze timing, resulting in a poorer performance in terms of AUC values than the one combining LSTM and CRF.

Combining the table and ROC graphs, the method proposed in this article achieves an improvement in deepfake video identification. In order to further validate the effectiveness of the method, the study also conducts comparison experiments between it and some of the current popular identification methods. The performance of the method in this article is evaluated against other methods mainly through the accuracy rate. Since the main current identification methods focus on single-frame identification and use datasets with obvious traces of forgery, we conducted comparison experiments on Celeb-DF and FaceForensics++ datasets, respectively.

After a comparative analysis, the method proposed in this study, which has the best performance on the Celeb-DF database, achieves a recognition accuracy of up to 91.72%. Although the method in this study did not achieve the optimal recognition on the FaceForensics++ database, there is only a 0.77% difference compared to the optimal recognition using the Capsule network (i.e., 96.60%). Therefore, the method proposed in this study achieves better detection results on both datasets and improves the detection rate of deeply forged videos. Figure 10 shows the overall comparison of the results. Similarly, Table 5 compares deepfake detection methods.