Classifying the difficulty levels of working memory tasks by using pupillary response

Related work

As the earlier work showed, pupil size is an indicator of mental activity. Hess & Polt (1964) observed that the pupil response is closely correlated with the difficulty of an arithmetical problem. Kahneman & Beatty (1966) was the first work that explores the influence of cognitive load on pupil diameter. They asked participants of an experiment to remember a sequence of digits and nouns. They observed that the pupil dilates as the material is presented and constricts during report.

The cognitive load is usually measured by (i) surveys such as the NASA-TLX (Hart, 2006) or the instructional approach (Eysink et al., 2009); (ii) physiological measures such as electroencephalogram (Lin & Kao, 2018), Galvanic Skin Response (Nourbakhsh et al., 2017), or eye size (Marandi et al., 2018); or (iii) behavioral data as the eye movements (Marandi et al., 2018). However, surveys do not offer real-time information, and physiological skin contact devices can be uncomfortable. An option is to use a remote eye-tracker. This is ideal for long-term use (i.e., continuous use for more than 30 days) because it is usually placed under the screen at 50–70 centimeters from the user.

Common eye-tracker information used to detect cognitive load are the eye-pupil size (Kun et al., 2013; Palinko & Kun, 2012), saccades (Marandi et al., 2018), fixations (Nocera, Camilli & Terenzi, 2006), and blinks (Faure, Lobjois & Benguigui, 2016). Fixations and saccades are made up of multiple gaze points. Examples of gaze features used to describe cognitive load are the length (Mitre-Hernandez et al., 2019), peak velocity (Di Stasi et al., 2010), and movements of saccades (Marandi et al., 2018). Eye blinking is influenced by cognitive processes. In general, the blink frequency drops when the task difficulty increases (Marandi et al., 2018; Faure, Lobjois & Benguigui, 2016). The blink frequency also is affected by changes in auditory tasks; e.g., questions for planned or spontaneous answers (Mitre-Hernandez et al., 2019), interviews (Frosina et al., 2018), or yes/no questions (George et al., 2017). From a statistical analysis of data generated from a driving task, Kun et al. (2013); Palinko & Kun (2012) observe a statistical difference in the pupil size for conversational tasks that increase the cognitive load and those that decrease it. Here, we analyze the statistical significance of different pupil-size indicators and the difficulty of a task. Furthermore, we use these pupil-size indicators as features in several classifiers that predict the difficulty of a memorization task.

Intuitively, the more information we use, the better performance we reach. That is why supervised learning approaches that use eye-tracker data (Table 1) use several features. But, in real applications, researchers face situations where some features cannot be used (e.g., in a language conversation practice, fixations or saccades cannot be available). Besides, some features can be voluntary controlled by the subject. Physiological signals reflect unconscious body changes, and are controlled by the sympathetic nervous system, while visual and audio cues can be adopted voluntarily or involuntarily (Shu & Wang, 2017).

Our ultimate goal is to develop a real-time method that can classify the perceived difficulty of different tasks (as olfactory, verbal, or visual ones) from physiological eye measurements. These considerations reduce the set of possible eye features just to pupil size and blink features. Unfortunately, eye blinking is only noticeable for frequency measurements of at least 100 Hz (Nakamura et al., 2008). That is why this paper focuses on pupil size features for difficulty classification.

Classification is the problem of identifying the category of a given observation. Classification from ocular features has been used in diverse applications such as detecting types of reading (Biedert et al., 2012; Kunze et al., 2013), mental states (Eivazi & Bednarik, 2011), levels of mental fatigue (Li et al., 2020), and driving behaviours (Deng et al., 2019). Another difference between the different approaches shown in Table 1 is the classifier selection. In the following paragraphs, we overview the related work concerning the classifier used.

A Support-Vector Machine (SVM) estimates the hyperplane that separates two classes with the maximal margin (Cortes & Vapnik, 1995). The original SVM only solves binary classification problems. Many techniques extended it to multiclass classification, the general idea is to predict the class of an instance form several binary SVM classifiers. As it is shown in Table 1, the SVM gives acceptable results for predicting mental fatigue (Li et al., 2020); but inconsistent results for predicting driving behaviour (Deng et al., 2019). The SVMs can also manage non-linear classification problems by incorporating a kernel function that maps the inputs into high-dimensional feature spaces. The radial basis function kernel (RBF) is popular to solve many classification problems.

The accuracy of the SVM (RBF) is good for binary classification—e.g., (Biedert et al., 2012); but, as expected, the accuracy is lower for multiclass problems—e.g., (Eivazi & Bednarik, 2011).

Fisher’s linear discriminant (FLD) is in essence a technique for dimension reduction (Fisher, 1936). For two classes, the FLD selects a projection that maximizes the class separation. FLD extension to multiclass is known as linear discriminant analysis (LDA). Its objective is to find a dimension reducing transformation that minimizes the scatter within each class and maximizes the scatter between classes in a reduced dimensional space Kim, Drake & Park (2007). Experiments of Li et al. (2020) to classify mental fatigue show that LDA gives the best results for some features.

The Hidden Markov Model (HMM) considers that a set of observable variables are generated by a sequence of internal (hidden) states. Besides, future hidden states depend only on the present hidden state (Baum & Petrie, 1966). HMM can be used for supervised learning by estimating its parameters from a set of labeled sequences. In this sense, HMM has been used to analyze labeled sequences of saccades, blinks, and other gaze features (e.g., fixation time). For instance, Kollmorgen & Holmqvist (2007) use HMM to detect reading and Li et al. (2020) classify driving behavior.

Decision trees are simple classifiers where a set of decision nodes are arranged in a tree structure. This approach learns rules that best split the learning dataset. Decision threes show average performance for detecting the type of reading (Kunze et al., 2013) and mental fatigue (Li et al., 2020). Boosted trees and Random forests (RF) are classifiers based on decision trees (Ho, 1998). Boosting is a method of combining many weak learners(trees) into a strong classifier. RF uses a feature bagging (bootstrap aggregating) procedure; i.e., it integrates the prediction of several decision trees trained by a random subset of features. Decision and boosting trees obtain inconsistent results for classifying mental fatigue. On the contrary, the RF obtains the best accuracy for discriminating the driving behavior.

In this paper, we study different classifiers to select the best one. Also, we use a feature selection approach to discover the best configuration.

Dataset

Short-term recall of a packed sequence of digits is a common experimental task in cognitive pupillometry. In this task, participants see (or hear) a sequence of numbers; then, they were asked to recall the sequence correctly, with increasingly longer sequences being tested in each trial. We use the dataset proposed in Klingner, Tversky & Hanrahan (2011) which evaluates different effects for cognitive tasks. In particular, we use pupil dilation measurements for the digit sequence recall task. Sections ‘participants’ and ‘materials’ describe the experiment performed by Klingner, Tversky & Hanrahan (2011).

Materials

The visual response of participants was recorded with a remote eye-tracking device Tobii 1750 at 50 Hz. The task was presented on a standard LCD computer display (1, 280 × 1, 024 screen resolution, 17 inches on the diagonal, w:h ratio of 5:4). Experiments were recorded with infrared lights and a high-resolution infrared camera. The eye-tracker supports head movement, but for eye pupil data, the speed movement must be less than 10 cm/s within a head-box of 30 × 15 × 20 cm with an initial distance of 60 cm from the screen. To avoid missing ocular data when participants interact with the screen, the dataset authors placed the eye-tracker on the top of the screen. The experiment room was relatively bright, with 27 cd/m2 of luminance from the surrounding walls at eye level and 32 lx incident at participants’ eyes.

Variables

The following variables were calculated from the pupillary data:

Baseline Pupil Size (BLPS). The prestimulus phase lasts two seconds before the question, Fig. 1(a). The BLPS is used to set a value of the pupil stabilization. In our experiments, this value is obtained as the average pupil diameter in the prestimulus phase.

Mean Pupil Diameter Change (MPDC). (Palinko et al., 2010; Palinko & Kun, 2012; Kun et al., 2013). To estimate the MPDC, the baseline is subtracted from the average of the pupillary data, that is, (1)${\displaystyle \text{MPDC}=\frac{1}{N}\sum _{i=1}^N{\text{PS}}_i-\text{BLPS},}$ where PS_i is the pupillary data collected at time i, and BLPS is the baseline.

Average Percentage Change in Pupil Size (APCPS). (Iqbal, Zheng & Bailey, 2004; Iqbal et al., 2005; Lallé et al., 2015). The Percentage Change in Pupil Size (PCPS) is calculated as the difference between the measured pupil size and the baseline pupil size, divided by the baseline pupil size. (2)${\displaystyle {\text{PCPS}}_i=\frac{{\text{PS}}_i-\text{BLPS}}{\text{BLPS}}}$ where PS_i is the pupil size collected in the i–th time, and BLPS is the baseline of the pupil size. The APCPS is the average in the measurement interval time. (3)${\displaystyle \text{APCPS}=\frac{1}{n}\sum _{i=1}^n{\text{PCPS}}_i}$ where n is the number of measurements in the interval.

Peak Dilation (PD). The Peak Pupil Dilation (PPD) is defined as, (4)${\displaystyle \text{PPD}=max\left\{{\text{PS}}_1,{\text{PS}}_2,\dots ,{\text{PS}}_n\right\}.}$ To reduce the error caused by different sizes of the human eye, we modified the equation proposed in Marandi et al. (2018) by including the BLPS. (5)${\displaystyle \text{PD}=\text{PPD}-\text{BLPS}.}$

Entropy of Pupil (E_pupil). Suppose that the pupil dilation is a random variable S with possible values s₁, s₂, …, s_m such that s₁ = min{PS₁, PS₂, …, PS_n}, s_m = PPD. Consider that p_i = f(i), where f(i) is the relative frequency associated with the i-th value s_i (i.e., how often the value s_i happens divided by the number of observations n). The information entropy is defined as (6)${\displaystyle E_{\text{pupil}}=-\sum _i^{}{p}_{i}log{p}_i}$ Entropy can be described qualitatively as a measure of energy dispersal. The concept is linked to disorder: entropy is a measure of disorder, and nature tends toward maximum entropy for any isolated system.

Time to Peak (TTP). Siegle, Ichikawa & Steinhauer (2008) show that pupil dilation peaked between the completion of the memory encoding interval time and the start of memory storage. The dilation is proportional to the difficulty of the memory task, as higher the difficulty, higher the peak is. Then, the time to peak may reveal the difficulty level. It is defined as, (7)${\displaystyle \text{TTP}=\mathrm{time}\left(\text{PPD}\right).}$

Peak Dilation Speed (PDS) The method of least squares can find a relation between time and peak dilation (Siegle, Ichikawa & Steinhauer, 2008). Consider the PS slope of the line that ends at the pupil dilation peak, estimated as (8)${\displaystyle m=\frac{\sum t_i\cdot {\text{PS}}_i-\frac{1}{n}\sum t_i\sum {\text{PS}}_i}{\sum \underset{i}{\overset{2}{t}}-\frac{1}{n}\sum {\left(t_i\right)}^2}}$ where 0 ≤ i ≤ n, and t_i and PS_i are the time and size of the ith measurement, respectively. Finally, the slope is used to calculate the angle, (9)${\displaystyle \text{PDS}={tan}^{-1}\left(m\right).}$

Procedure

As shown in Fig. 1, the procedure consisted of two phases. During pre-stimuli interval time (Fig. 1A), participants observed an ‘X’ in the center of the screen for 2 s. In the stimuli phase (Fig. 1B), digit-span memory tasks were presented randomly. In each task, one digit appears on the screen each second. Based on the number of digits presented to the participant. The cognitive load evoked by memorizing a given number of digits is not linear (Crannell & Parrish, 1957); here, we consider three difficulty levels: the low-level difficulty was associated with memorizing three-digit numbers, medium-level to five-digit numbers, and high-level to eight-digit numbers.

The number of pupil size measurements depends on the number of digits presented on the screen. For this reason, a different number of measurements were obtained for each subject at each difficulty level. All of them have at least three measurements. And a maximum of 7, 10, and 13 measurements for low, medium, and high difficulty levels, respectively. Figure 2 shows the measurements for different difficulty levels.

The pupil diameter collected in the prestimuli phase was used for BLPS calculation, and the time and pupil diameter in stimuli phase were used to calculate MPDC, APCPS, PCPS, P_pupil, PD, TTP, PDS calculation. Finally, each trial was labeled with one of 3 levels of task difficulty (low, medium, and high).

Data filtering and preprocessing

Data were pre-processed as follows, records with null values or with blinks were eliminated, records with information in both eyes were averaged, and records with data in only one eye were not changed.

A valid trial must not have more than 20% of Invalid records. Finally, the measurements were passed through a Savitzky–Golay filter (Savitzky & Golay, 1964) for the purpose of smoothing the pupil size signal. We choose this filter to preserve characteristics of the initial distribution, which normally disappear with other techniques.

Statistical analysis

To investigate if there were significant differences in the study variables due to the difficulty level we used a linear mixed model (Demidenko, 1987).

Several measurements were taken for each individual at each difficulty level. This process violates the assumption of independence of a linear model. On the other hand, the difficulty results of levels between-subject can present different mean pupil diameter change (Fig. 3).

This characteristic factor affects all the responses of the same subject, which makes these responses interdependent instead of independent. Then, for the analysis, we considered the difficulty level a fixed effect, and both the subject and the subject-level effects random effects.

For the seven variables (BLPS, MPDC, APCPS, E_pupil, PD, TTP, and PDS) a linear mixed model was adjusted. In particular, the model was estimated using the restricted maximum likelihood (REML) approach (Harville, 1977; Patterson & Thompson, 1971). Statistical analysis has been performed utilizing R.

Classification

After the initial statistical analysis, we selected the most precise five classifiers from those shown in Table 1. The tested classifiers were the SVM with four kernels, LDA, Decision Trees, and Random Forest. The code was implemented in python using the scikit-learn library (version 0.14).

We studied the best classifier configuration. For this aim, subsets of the features set {BLPS, MPDC, APCPS, PD, E_pupil, TTP, PDS} were tested. With a set of b features, 2^b − 1 different sets can be realized. In this case, with seven training features, we have a total of 127 possible combinations. Each classifier/subset feature pair was trained with 80% of the data and tested with the remaining 20% data. For this aim, we adopted a k-fold cross-validation scheme (k = 5). In k-fold cross-validation, the original sample is randomly partitioned into k equal sized subsamples. One of these subsamples was used for testing, and the remaining k − 1 subsamples are used as training data.

Statistical analysis

We obtained p-values by likelihood ratio test of the full model with the effect against the model without the effect in question. As shown in Table 2, we observed that all effects were significant at a level of 0.001, except BLPS which was not significant. After fitting a suitable model, post hoc comparisons were made by using the Tukey HSD test. As shown in Table 3 (‘Initial model’ column), the differences were significant at a level of p < 0.05.

A single influential observation (outlier) was found for MPDC, and four influential observations were found for APCPS, TTP, and PDS. Besides, heteroscedasticity of the model residuals for PD and PDS was also observed. Each model was fit to the filtered and normalized data. Table 3 (column ‘final model’) shows the results of the Tukey-HSD post hoc tests of the final models. In general, the conclusions are maintained.

For most variables, the distributional assumptions of the mixed model were not fulfilled. The normality of residuals assumption is the least important one. The linear models are relatively robust against violations of the assumptions of normality. Gelman & Hill (2006), do not even recommend diagnostics of the normality assumption. Indeed, results described in this section show that feature values differ significantly for different difficulty levels.

Classification of cognitive load

Table 4 shows the accuracy results of the ten best-performing classifiers. The model M₁, a Random Forest classifier trained with five features (BLPS, MPDC, APCPS, PD, and TTP), achieved the best average F1 score. This configuration achieved 85% precision for the low difficulty level, 80% for the medium level, and 83% precision for the low level.

Models based on the Support vector machine (SVM) also achieve good results, with F1 scores as high as 76%. Particularly, the M₅ model uses just two features (MPDC, and TTP), and it is able to reproduce the same F1 score (71%) that other SVM models that use more features.