A review of deep learning in blink detection

Rationale and audience

Blinking, as a typical action of human body, indicates the current physiological and psychological state information of the individual. In recent years, people have gradually found that blink detection can be widely used in many fields of daily life. Therefore, a large number of works related to blink detection have been born in the past decade, and satisfactory results have been achieved. The traditional computer vision method is the most commonly used method, but it suffers from poor generalization ability and robustness. Deep learning is one of the techniques that is gaining popularity and playing an important role in the field. The blink detection method based on deep learning has a powerful feature learning ability. Table 2 summarizes the applications of deep learning-based blink detection methods in various fields. The introduction of deep learning technology can better identify eye features and process eye movement sequences, and the detection accuracy is higher. Despite the significance and rapid progress of blink detection based on deep learning, there is a lack of such an exhaustive literature review. This literature review aims to fill this gap by conducting a comprehensive survey on blink detection in the context of deep learning.

Given that blink detection has applications in several domains, this review caters to a wide audience, including the human-computer interaction (HCI) field, the cybersecurity field, the medical field, and the transportation field. The target audience for this systematic review is human-computer interaction researchers, driver assistance systems engineers, medical and psychological researchers, and computer vision researchers. Computer vision researchers receive a quick overview of current results and methods in the field to advance their research. HCI developers can learn about the performance, advantages and disadvantages of different algorithms in order to choose the most suitable technical solution for their applications. Driver assistance system engineers can learn about new methods for fatigue state through this article. Medical and psychological researchers can learn about the trend of blink detection technology and explore the possibility of its application in clinical and experimental research through the review.

Research questions

The review has summarized and analyzed deep learning-based blink detection algorithms from 2017 to 2024. The following research questions (RQs) are thus formulated:

RQ1: What is the trend in the application of blink detection technology in recent years?

RQ2: What are the applications of blink detection technology in various fields?

RQ3: What are the different models applied in deep learning-based blink detection?

RQ4: What are the advantages and disadvantages of deep learning models applied to blink detection?

RQ5: What are the performance evaluation criteria for blink detection algorithms?

RQ6: What datasets are available for training and testing blink detection algorithms?

RQ7: What are the limitations and challenges of deep learning-based blink detection methods?

RQ8: What are the future research directions for blink detection?

Study search and selection

Articles for this study were collected from a variety of databases, including IEEE Xplore, ACM Digital Library, Web of Sciences, ScienceDirect, and Google Scholar, covering high quality research articles in the field from 2017 to 2024. The keywords are derived from the proposed research questions and include terms such as “blink detection”, “deep learning”, “computer vision” and “eye state detection”. Figure 2 shows the PRISMA analysis chart for articles selection. Table 3 shows the inclusion and exclusion criteria for the selection process. Among the 2,312 records searched, 48 articles were finally selected after screening.

Table 4 summarizes the number of articles published each year in the field of blink detection since 2016, and compares the total number of articles based on deep learning methods with other methods. From Table 4, it is clear that the percentage of the number of articles based on deep learning has gradually increased in recent years, indicating that this field is gradually becoming a focus of research.

Eligibility criteria

Table 3 summarizes the inclusion and exclusion criteria for the studies. These criteria were used to determine the range of studies and to determine which studies should be included and which should be excluded. Inclusion criteria define the points that must be considered for the study. Exclusive criteria define which characteristics would cause the study to be excluded from the analysis. These criteria help to ensure that studies are focused and aligned with the study objectives.

Application of convolutional neural network in blink detection

CNN (Ma et al., 2021) is widely used in the field of image processing. It uses the local characteristics of the data itself through spatial or temporal pooling, local perception and shared weights to achieve the effect of optimizing the network structure (Fukushima, 1980; Krizhevsky, Sutskever & Hinton, 2012; Lecun et al., 1998). Figure 3 shows that the blink detection model based on CNN usually consists of input layer, convolution layer, pooling layer and classification area (Alzubaidi et al., 2021; Hussain, Bird & Faria, 2019). The eye detection picture is fed from the input layer to the convolutional layer. The convolutional layer is an important part of the neural network, in which deeper abstract features are extracted (Li et al., 2022). The pooling layer contains two methods: average pooling and max pooling. The pooling layer can simplify the structure of the neural network.

The classical CNN models are AlexNet, VGGNet, GoogLeNet, ResNet. AlexNet (Krizhevsky, Sutskever & Hinton, 2012), proposed in 2012, is based on the LeNet-5 convolutional neural network structure and deepens the layers of the network. The model can deal with more complex image classification tasks and can also extract higher dimensional image features. A different feature from earlier neural network structures is that the ReLU activation function is used in AlexNet instead of the Sigmoid activation function to solve the problem of gradient dispersion. Dropout random suppression neuron correlation technique is also used to solve the overfitting problem. In 2014, Simonyan & Zisserman (2014) proposed VGGNet. One of the improvements of VGGNet over AlexNet is the use of several consecutive 3 × 3 convolutional kernels instead of the larger convolutional kernels in AlexNet (11 × 11, 7 × 7). For a given receptive field, the use of stacked small convolutional kernels is superior to the use of large convolutional kernels. And adding nonlinear layers allows the network to learn more complex patterns with fewer parameters. In 2015, Szegedy et al. (2015) proposed GoogLeNet, which has 22 layers, a further deepening from the 16 layers of VGGNet. Due to the increased depth and width of GoogLeNet, this network model is able to learn more and more detailed image features than previous deep learning models. GoogLeNet has fewer parameters than AlexNet and VGGNet due to the sparse network structure used in the model design and the proposed Inception module. Traditional neural network models usually deepen the width and depth of the network structure to achieve better feature extraction. However, the network is usually easily degraded after deepening to a certain degree. In order to overcome this situation, He et al. (2016) proposed ResNet, and the shortcut structure of ResNet solves the depth degradation problem well.

During the blink detection, CNN continuously abstracts the data by convolution and pooling the extracted features. Local features are continuously refined into high-level features, and then the purpose of blink detection can be realized. In 2017, methods using deep learning to detect eye blinks gradually emerged. Anas, Henríquez & Matuszewski (2017) proposed two CNN architectures for detecting blinks based on LeNet: one of them is used to identify the eyes as two states (eyes closed and open), and the other for three-level eye state detection (eyes closed, eyes open, and partially eyes open). The experimental results show that the proposed model is superior to the model based on SVM and AdaBoost in the detection accuracy. It illustrates the effectiveness of CNN for blink detection to some extent. In order to further improve the accuracy of blink detection, Sanyal & Chakrabarty (2019) proposed a blink detection model based on two-stream convolutional neural network (TSCNN). The structure of TSCNN model is shown in Fig. 4, which consists of RGB-CNN and MCNN. RGB-CNN takes an RGB image of an eye as input, and MCNN takes a binary mask of the RGB image as input. The RGB-CNN network aims to learn the global features of the eyes. MCNN mainly learns the spatial distribution of eye shape, contour, eye region pixels and non-eye region pixels. Due to the different network structures of RGB-CNN and MCNN, they can extract different discriminative features of the eyes. Therefore, these two networks are combined and jointly trained to achieve the effect of eye state classification. The proposed model is more robust and accurate than its components RGB-CNN and MTCNN for blink detection alone. Experiments on various popular benchmark datasets show that the blink detector of the proposed model achieves 1–2% improvement in precision and recall. A robust, real-time, low-cost blink detection system was proposed by Medeiros et al. (2022). A blink is used as a human-computer interaction signal, and CNN and SVM are used to handle the blink detection task. To start with, single shot multibox detector (SSD) is used for face detection. The eye region and its coordinates are extracted from the facial image by extracting the facial key points. Then CNN is used to classify the eye region to determine whether the eyes are open or closed, and SVM is used to classify the extracted eye coordinates. At last, based on the output of consecutive frames, the blink is detected and its duration is calculated. Chavarro & Karakaya (2024) proposed the use of blinks as an additional feature to address the poor performance of conventional iris recognition systems under non-ideal conditions. By using a pre-trained AlexNet model, the output layer was replaced with a regression layer, allowing the model to accurately output the percentage of blinks.

The blinking process has rich motion information. There are spatiotemporal variations at different scales. To this end, Gao, Wang & Wang (2019) proposed a two-level cascade model of video sequence-level preliminary detection and frame-level accurate detection based on CNN. Video clips with longer eye opening time and possible blinks were first separated. Then the blinking process is discriminated in the video clips where blinks may exist. It solves the problem that CNN is difficult to analyze eye movement sequences. In 2022, Bekhouche et al. (2022) addressed the problem that previous learning methods could only deal with sequences containing only a single blink and ignored the case where multiple blinks were present. This article presents a fast framework for blink detection and blink verification. Multiple blinks are effectively extracted from the image sequence, and the eye signal for blink detection is extracted by constructing a feature-based matrix. The blink verification is then performed using Pyramidal Bottleneck Block Networks (PBBN). The pyramid bottleneck module is shown in the Fig. 5. The total number of modules in the architecture is reduced by using this module. Thus, the number of parameters is reduced and the inference time is shortened. This study provides a potential method for effectively detecting blinking in a variety of practical applications, including disabled communication, fake face detection, and face anti-deception.

In wild environments, blink detection methods are susceptible to adverse lighting conditions. Hong, Kim & Park (2023) designed a CNN based Blink Estimation With Domain Adversarial Training Network (BEAT network), which extracts features that are not affected by the domain. The method of domain adversarial training is used to solve the adaptability of deep neural network to poor lighting conditions and improve the detection system performance. Ibnouf et al. (2023) proposed a fatigue detection method based on blink detection for low light conditions. Firstly, the input image is preprocessed by applying a light enhancement algorithm to improve the brightness and contrast of the image, and then the eye region is extracted from the processed image using a Haar Cascade classifier. Next, the eye region images are input into a CNN model to determine whether the driver is in a fatigue state by calculating the blink frequency. Zhou (2022) adopted Zero-Reference Deep Curve Estimation (Zero-DCE) method based on deep learning to assist blink detection in view of the low accuracy of blink detection in low light environments. The main structure of Zero-DCE is seven convolution layers in symmetrical cascade, which is used to improve the details of dark and fuzzy images. This method can effectively improve the accuracy of blink detection results at night, and expand the application scenarios of detection. Li et al. (2024) proposed a YOLO-based eye state detection algorithm, referred to as ES-YOLO, to address the problems of complexity and susceptibility to light interference in blink detection algorithms. The algorithm optimizes the structure of YOLOv7, integrates multi-scale features using the convolutional block attention mechanism (CBAM), and improves the attention to the important spatial locations in the image. In addition, Focal-EIOU Loss is used instead of CIOU Loss to increase the attention to difficult samples and reduce the effect of sample category imbalance.

Table 5 summarizes the above CNN models. It can be obviously found that CNN has strong feature extraction ability. It extracts the features of the image step by step through the convolution layer and the pooling layer, so as to effectively represent the features of the eye region (Cruz et al., 2022). This feature extraction ability enables the CNN to capture key information such as the shape and texture of the eyes, which contributes to the accuracy of the blink detection task. CNN has a strong data adaptive ability. It can automatically learn the patterns and rules in the blink detection task through large-scale training data, so as to improve the generalization performance of the model. In the case of experimental needs, CNN can be extended and improved by increasing the number of network layers, adjusting the network structure and parameters. To adapt to blink detection tasks of different scales and complexities (Bekhouche et al., 2022). However, CNNs have some drawbacks due to the complexity of CNN and the large number of parameters. The training and inference process for the blink detection task requires high computing resources and time, including GPU and memory (Cortacero, Fischer & Demiris, 2019). In addition, CNN is sensitive to illumination and pose changes when dealing with blink detection tasks. As a result, the model performs unstable under different illumination conditions or large changes in eye pose (Zhao et al., 2018).

In CNN-based blink detection research, a large amount of labeled data is required for training. Especially for blink detection tasks in complex scenes or specific application fields, more diverse training data are needed to improve the robustness of the model. However, in the blink detection task, the eye data in the normal state usually far exceeds the data in the blink state, which will lead to data imbalance (Cortacero, Fischer & Demiris, 2019; Hong, Kim & Park, 2023). The model is too biased towards the normal state to accurately detect blinking. This is a major difficulty in blink detection, and it is necessary to build a larger and more diverse blink data set in subsequent research. In order to provide better training samples and further improve the generalization ability and robustness of the model.

Application of recurrent neural network in blink detection

RNN is a kind of self-connected neural network in the field of deep learning. It learns complex mapping from vector to vector and can be applied to various tasks related to time series (Elman, 1990; Wang, Fan & Wang, 2021). The model structure of RNN recurrent neural network is shown in Fig. 6 (Tan et al., 2018). The input of the hidden layer contains both the input of the current moment and the output of the hidden layer at the previous moment. The neurons in the hidden layer are connected to each other and can remember the previous information (Dhruv & Naskar, 2020; Sherstinsky, 2020). Therefore, it has advantages in dealing with timing problems in blink detection. The RNN-based blink detection models are described below and summarized in Table 6.

In blink detection, RNN plays the role of improving existing neural networks. Human perception of blinking is not just observing images, but viewing blinking as a behavioral process. It includes three stages: start, peak and end (Gao, Wang & Wang, 2019). Fogelton & Benesova (2018) used recurrent neural network as a classifier to detect eye blinks earlier. This is the first deep learning blink detection method capable of evaluating the integrity of blinks. By using RNN to detect the blink of the current frame based on the current and previous features, it provides a new idea for the research of blink detection.

In practical applications, training RNN will encounter the problems of gradient disappearance and gradient explosion, which makes it impossible to process very long eye video sequences (Bengio, Simard & Frasconi, 1994). Hochreiter & Schmidhuber (1997) proposed long short-term memory network (LSTM) in 1997 to achieve the purpose of improving the traditional recurrent neural network model. As shown in Fig. 7, an LSTM unit consists of a cell state $\left(C_t\right)$, an input gate $\left({\sigma }_i\right)$, a forget gate $\left({\sigma }_f\right)$, and an output gate $\left({\sigma }_o\right)$. ${\sigma }_i$ and ${\sigma }_f$ work together to extract the information they need and filter out the noise. Information useful for recognizing the behavior is stored in $C_t$, which in turn updates the cell state. ${\sigma }_o$ is responsible for combining the current input and the valid information in the previous frame recorded in the hidden state $h_t$ to synthetically judge the output. After that, the hidden state is updated according to the current output and the updated cell state, which facilitates the utilization of subsequent LSTM units and enables it to cope with long-term dependencies in time series data (Bahdanau, Cho & Bengio, 2015; Bhaskar & Thasleema, 2023; Houdt, Mosquera & Nápoles, 2020). In 2020, Hu et al. (2020) targeted the multi-temporal nature of blinks. By extracting the multi-temporal scale features of the eyes, the modified LSTM model, Multi-scale LSTM (MS-LSTM), is used to discriminate the blink. It improves the adaptability of LSTM to detect blinks under unconstrained conditions.

The advantage of RNN in blink detection is that it can model the sequence data in a cyclic way and capture the time information of the blink action (Fogelton & Benesova, 2018). This ability allows the RNN to take into account the previous eye state in blink detection to better determine whether a blink is currently occurring. And because the blink action is sequential, RNN can effectively extract temporal features, such as the duration and frequency of blinks. It helps to distinguish more accurately between blinking and other similar actions (Hu et al., 2020; Lu et al., 2024a). However, RNN and LSTM are sequential. Each time step needs to be computed sequentially, resulting in low computational efficiency. In particular, it will bring some delay when dealing with longer sequences or real-time applications (Cruz et al., 2022).

Application of CNN-RNN in blink detection

In engineering practice, there are few cases of applying RNN model alone for blink detection. Most researchers construct a fusion model of CNN and RNN to detect eye blinks. In 2018, Li, Chang & Lyu (2018) targeted CNN-based methods for their inability to consider temporal consistency during blinks (Anas, Henríquez & Matuszewski, 2017). For the first time, the combination model of RNN and CNN is used to discriminate eye blinks, and a long-term recurrent convolutional networks (LRCN) is formed, as shown in Fig. 8 typical CNN and RNN models. The CNN first identifies the discriminative features, and the output is passed to an RNN with long short-term memory for sequential learning of the temporal context of past and future inputs. Finally, the output of the RNN is classified as eye open or eye closed. It solves the problem that the state of the previous eye image cannot be taken into account. Popat et al. (2024) instead segmented the user’s live video stream into a series of image inputs, trained a CNN to classify each frame independently to detect blinks, and used a LSTM to calculate blink frequency. The method can be implemented as a background application on smartphones and computers to help prevent the risk of computer vision syndrome (CVS) or similar disorders. Cruz et al. (2022) and Saealal et al. (2022) improved the LRCN structure. Gonzalo et al. used the Siamese architecture during CNN training to overcome the high-level imbalance present in blink detection and the limited amount of data available to train the blink detection model. Saealal et al. (2022) applied the weight and bias update process to adaptive moment estimation (Kinga & Adam, 2015) to provide a direct way to gradually change the learning rate to adapt to changes in the dataset. Through these improved LRCN models, the performance is improved and the development of blink detection research is promoted.

It can be known from the above research on the blink detection method of CNN and RNN fusion. The biggest advantage of CNN is to extract eye image features, while RNN can effectively obtain the time information of the blink process. Therefore, many researches combine the advantages of the two methods to improve the classification effect of blink detection. When the actual eye area is small, the CNN-based model using only image input is ineffective, while the CNN-RNN model using temporal correlation can correctly predict. However, the model is composed of two deep learning networks, the structure is complex and the training is difficult. In some application scenarios, it is necessary to detect blinking accurately in real time, such as driver fatigue detection. However, RNN models usually require a long inference time (Cruz et al., 2022), which is not suitable for real-time applications. How to improve the inference speed of the model while maintaining the accuracy is a problem to be solved.

Application of transfer learning in blink detection

Transfer learning (TL) can be described as the transfer learning of knowledge learned in the source domain to the target task function (Niu et al., 2020; Zhu et al., 2023; Zhuang et al., 2021). In the field of blink detection, transfer learning can apply the prior knowledge learned from the blink data in the source domain to the target domain to accelerate the training of the model (Kayadibi et al., 2022). Figure 9 shows the transfer learning process for eye blink detection. Transfer learning is used to transform other neural network layers trained by blink-related databases into blink detection models (Liu et al., 2021).

The comparison of blink detection based on transfer learning and traditional machine learning is shown in Fig. 10. The figure shows the difference between the implementation process of traditional machine learning techniques and transfer learning techniques: traditional machine learning methods train different models independently for specific domains, data and tasks (Tan et al., 2018). Transfer learning can effectively use the existing knowledge from one domain to help improve the learning ability of other different target domains. In particular, it is used to overcome the limitations of highly specialized but small datasets (Weiss, Khoshgoftaar & Wang, 2016). Moreover, transfer learning can reuse the knowledge of previous tasks without having to learn a new task from the initial stage.

Blink detection using deep learning methods requires a large number of samples to train. It is difficult to obtain a sufficient number of datasets to meet the training requirements. In response to this problem, Zhao et al. (2018) used transfer learning methods to improve deep ensemble neural networks for blink detection. The base model is pre-trained by learning from the base dataset with enough samples through supervised learning. Then, the learning parameters of the pre-trained base model are transferred to the target network to fine-tune the data set of the target task. The recognition ability of deep learning method is improved. Cruz et al. (2022) also used transfer learning to process a small number of samples in their blink detection network for the problem of relatively small data sets. Making its method robust to class imbalance. In 2021, Liu et al. (2021) pointed out that the facial expression recognition task and the eye state detection task have a common cross feature of the eye region. A transfer learning method is proposed to accelerate the training speed of eye state detection method and improve the performance of the detection model. The introduction of transfer learning improves the parameter training in the network and reduces the limitation caused by the small number of training samples. In 2022, Saurav et al. (2022) addressed the issue that training a deep learning model on a small blink detection dataset may lead to overfitting. The transfer learning method is used to fine-tune the network parameters to overcome overfitting problem. Pahariya, Vats & Suchitra (2024) proposed a migration learning based MobileNetV2. Existing image classification datasets are used to train MobileNetV2, and then the model is adapted to the blink detection task by migration learning. This approach effectively improves the generalization ability and computational efficiency of the model. Table 7 summarizes the above transfer learning-based blink detection methods.

The advantage of the application of transfer learning in blink detection is the ability to transfer the learned weights to other neural network layers. The process of re-training the neural network is avoided and the training efficiency is improved (Liu et al., 2021). Transfer learning uses the learned features and knowledge to optimize the network parameters so as to improve the performance of the model and make up for the shortage of samples, and improve the performance of the model (Kayadibi et al., 2022). However, the application of transfer learning is limited by the similarity between the source and target domains. If the two domains are very different, transfer learning will not bring significant performance improvement (Saurav et al., 2022).

Application of transformer in blink detection

Transformer was first published by the Google research team in 2017 (Vaswani et al., 2017). The model uses multi-head self-attention (MSA) mechanism and Positional Encodings (PE), which can efficiently parallelize computation and capture long-range dependencies. Additionally, it demonstrates strong scalability and generalization capabilities. As a result, Transformer has established itself as a pivotal framework in the field of deep learning.

In the work published by Liu, Xu & Lu (2023), a Transformer-based blink detection algorithm is proposed. Transformer architecture effectively addresses the challenges related to temporal feature extraction and parallelism in RNN-based methods. In addition, the algorithm extends an additional head to detect the blink strength in each frame. This enables more accurate learning of the key features of blinks. Fodor, Fenech & Lőrincz (2023) proposed a fast transformer-based blink detection method to address challenges such as lighting changes and head pose changes. The model fuses multiple input features through a multimodal Transformer, effectively combining low and high level feature sequences to achieve accurate blink detection and eye state recognition. Hong et al. (2024) proposed a Dual Embedding Video Vision Transformer (DE-ViViT) for blink detection. When a facial feature is detected in a frame, both eye regions are cropped according to their coordinates. Every 10 frames are passed to the dual embedding module, which consists of tubelet embedding and residual embedding. After converting the input sequences into embedding vectors, they are processed through the Transformer architecture. Finally, multilayer perceptron (MLP) head detects if a blink occurs based on the output of the encoder.

The blink detection model based on Transformer shows strong multi-task learning capability and feature capture ability, which can accurately recognize complex eye states and improve the robustness of detection. However, Transformer requires substantial computational resources and is not suitable to resource-constrained environments. Furthermore, the self-attention mechanism in the Transformer model is based on global information, which is less effective in detecting small targets (e.g., small changes when eyes are closed).

Application of deep belief network in blink detection

In 2006, Hinton & Salakhutdinov (2006) utilized the stacked dimensionality reduction technique of multi-layer restricted Boltzmann machines (RBM) to achieve an unsupervised feature learning approach, but the problems of unstable gradient in training and sensitivity to raw data were not solved. In the same year, Hinton, Osindero & Teh (2006) added greedy algorithms and linear classifiers to form deep belief networks (DBN), which is widely used nowadays. In 2022, Kir Savaş & Becerkli (2022) used the method of detecting drivers’ blinking behavior as an indicator for assessing driving fatigue by using DBN. RBM automatically extracts features at different abstraction levels through unsupervised learning. After layer-by-layer pre-training, the DBN is subjected to supervised fine-tuning using labeled data to optimize its recognition of blinking. Experiments show that the model can effectively detect the blinking behavior of drivers. Table 8 summarizes the above Transformer and DBN based blink detection methods.

The DBN-based blink detection algorithm maintains good performance despite insufficient data, which is attributed to the capability of extracting hierarchical features through unsupervised learning to provide rich and deep data representations for subsequent blink detection. However, the training process of this algorithm involves the training of multiple RBM layers, which will cost a lot of time.

Based on the analysis presented above, a comparison of the advantages and disadvantages of deep learning methods, including CNN, RNN, transfer learning, Transformer and DBN in the context of blink detection applications, is provided in Table 9. This comparison aims to facilitate a more effective utilization of these methodologies for advancing research in blink detection.

Commonly used datasets

Datasets play a fundamental role in blink detection and are an important factor affecting the development of this research field. Currently, the commonly used datasets in eye blink detection tasks are mainly Closed Eyes in the Wild (CEW) (Song et al., 2014), ZJU (Pan et al., 2007), Eyeblink8 (Drutarovsky & Fogelton, 2015), Constraint-plus (Ghoddoosian, Galib & Athitsos, 2019), Helen (Le et al., 2012), and Real-Time Blink Estimation in Natural Environments (RT-BENE) (Cortacero, Fischer & Demiris, 2019). Table 10 organizes the above commonly used datasets for eye blink detection. The emergence of these datasets has promoted the development of deep learning techniques in eye blink detection research.

CEW: The CEW dataset is extracted from LFW dataset (Huang et al., 2008). The dataset has a total of 2,423 images, of which 1,192 images with closed eyes are collected directly from the Internet and 1,231 images with open eyes are selected from labeled faces in the field database. This dataset involves a wide range of people, and various human eye pictures are comprehensive. But the images provided by each subject lacked variety.

RT-BENE: The RT-BENE dataset has annotations of eye open degree of more than 200,000 eye images. Includes over 10,000 images of eyes closed. This dataset is suitable for gaze estimation in natural environments where the distance between the camera and the subject is large and the subject motion is less constrained. Its large size is an important benchmark in blink detection research.

Constraint-plus: Constraint-plus is established based on the time-series blink behavior dataset RLDD. It combines existing constrained blink datasets (ZJU, Eyeblink8, Talking-faceRLDD, etc.) into a new constrained blink dataset. The dataset contains 4,935 samples (2,435 blink samples and 2,500 non-blink samples). Among them, the training set contains 2,235 blink samples and 2,300 non-blink samples, while the test set contains 200 blink samples and 200 non-blink samples. This dataset is able to illustrate whether the superimposed constrained dataset is a good description of the blinking behavior under unconstrained conditions.

Helen: The Helen dataset contains a large number of images of objects with open and half-open eyes. However, the number of subjects with closed eyes is rather limited. There are 2,330 images in the Helen dataset, consisting of facial images representing different ages, genders, and ethnicities. And the main face locations are accurately and detailed annotated. This dataset contains images with different poses, illuminations, and expressions, and is suitable for training models that extract facial features from the wild.

ZJU: The ZJU dataset consists of 80 videos of 20 people. Clips of several seconds were collected with a 30fps camera with a resolution of 320 × 240. Each person has four segments, which are the frontal view, the upward view, the view with glasses, and the view without glasses. Facial expressions and head movements are absent from the collected segments. Some of these images have low resolution or are occluded by glasses.

Eyeblink8: The Eyeblink8 dataset is a dataset of eye blinking behavior. Eight videos with different temporal lengths are included. Every two videos belong to the same person (one of them is wearing glasses). The entire dataset contains 70,992 images with a resolution of 640 by 480. Videos were recorded under different conditions, with most faces facing directly towards the camera. It contains many natural facial movements and other non-blinking actions, increasing the challenge of blinking detection.

Evaluation metrics

In order to compare different blink detection algorithm models, evaluation metrics are needed. The evaluation metrics include precision, recall, F1-score, accuracy etc. Their numerical values are obtained from the variables TP, TN, FN, FP in the confusion metrix (Ahmed et al., 2023). Where TP is the number of blink samples that were correctly identified. TN represents the number of non-blinking samples correctly identified. FN represents the number of blinking behaviors that were detected as non-blinking behaviors. FP represents the number of non-blink samples detected as blink samples.

The blink detection precision (P) is defined as follows:

(1) $$\mathrm{P}={\displaystyle \frac{TP}{TP+FP}}$$

Recall (R) is defined as follows:

(2) $$\mathrm{R}={\displaystyle \frac{TP}{TP+FN}}$$

Accuracy is defined as follows:

(3) $$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}={\displaystyle \frac{TP+TN}{TP+FP+TN+FN}}$$

The F1-score of blink detection is defined as the harmonic average of accuracy and recall. Its formula is as follows (Cruz et al., 2022):

(4) $$\mathrm{F}1\text{-}\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}=2\times {\displaystyle \frac{P\times R}{P+R}}$$

In addition, ROC-AUC is also an important metric for evaluating the performance of blink detection models. The horizontal coordinate of the receiver operating characteristic (ROC) curve is FPR, i.e., how many samples that are actually non-blinking are predicted to be blinking. The vertical coordinate is TPR, i.e., how many of the actual blinking samples are predicted to be blinking. Area under curve (AUC) is the area under the ROC curve, and its value can evaluate the model. The larger the value, the better the model classification performance.