Parameter optimization of histogram-based local descriptors for facial expression recognition

Face registration

Face registration is a procedure that aims to remove image regions that contain no information related to facial expression (such as background and hair) while giving the face a predefined size and pose. Many existing approaches include a normalization or registration step before feature extraction. For instance, Carcagnì et al. (2015) used Viola-Jones-based algorithms and ellipse fitting to detect and align the face image while removing background objects. However, the method depends on facial feature color models, limiting the applicability to gray-scale images. Zhang et al. (2018) used the Viola-Jones algorithm to detect faces and crop the images to 100 $\times$ 100 pixels. A similar approach was followed by Mandal et al. (2019). Hassan & Suandi (2019) used Cascade Linear Regression to detect facial landmarks. The distance between the eyes was used as a reference to align the face and normalize it 110 $\times$ 150. Liu et al. (2021) used a facial landmark detector that marks 68 points on the face, 27 of which permit the face region to be extracted from the background. Note that this approach does not align the face image.

Based on the above works, face pre-processing plays an essential role in FER systems. However, the impact of face registration on the recognition rate has not sufficiently been quantified. Our work investigates the impact of face registration in more detail. We also propose a novel automatic face registration algorithm inspired by the work in Carcagnì et al. (2015) that can also be used with gray-scale images.

Feature extraction

Robustness of feature extraction is dependent on their discriminative power, the ease with which images are captured, their dimension and tolerance to noise, such as illumination changes, and have low intra-class variations (Turan & Lam, 2018). Hence, extensive work has been done to achieve robust feature extraction. Geometric features have been investigated to represent facial expressions (Ghimire & Lee, 2013; Loconsole et al., 2014; Lin, Cheng & Li, 2014). Though geometric features can offer high discriminability, they depend heavily on the accurate detection and alignment of fiducial points in a face. Appearance features can solve this problem because they extract features related to the appearance of the entire face or local patches. Lu et al. (2015) proposed combining Gabor features and sparse representation-based classification to recognize facial expressions. The method is robust to illumination changes, different resolutions, and orientation. Sajjad et al. (2018) fused two appearance and texture features: Histogram of Oriented Gradients and Uniform Local Ternary Patterns, and then classified them using support vector machines. Experiments showed that the method performed well in the presence of occlusions. Deep learning methods have also been applied for facial expression recognition. Some notable works include Zhang et al. (2019), Bougourzi et al. (2020), Dong, Wang & Hang (2021). Though deep learning methods can achieve high recognition rates, they often require complex networks and expensive pre-processing computations to produce these results.

Local descriptors have proven to be a powerful way of characterizing expression-related information while being easy to compute. In particular, the histogram-based local descriptors can represent the occurrences of micro-patterns (such as edges, lines, and bumps) that form facial expressions. However, the aggregation of these micro-patterns into histograms leads to a loss of information about the location of the micro-patterns (Shan, Gong & McOwan, 2009; Moore & Bowden, 2011). To address this issue, weighted histogram representations have been proposed to give higher weights to regions with higher discriminability (e.g., eyes and mouth) and lower weights to regions with lower discriminability (e.g., image borders). Kabir, Jabid & Chae (2012) proposed the Local Directional Patterns with variance (LDPv) descriptor to characterize the texture and contrast information of facial components. The novel descriptor adds weights to an LDP (Local Directional Patterns) code based on its variance across the image. Recently, a similar approach called the Weighted Statistical Binary Pattern descriptor was developed by Truong, Nguyen & Kim (2021). The new descriptor combines the histograms of sign and magnitude components of the mean moments of an image. The histograms are weighted based on a new variance moment that contains distinctive facial features. Though the weighted methods achieve high recognition rates and robustness, they can suffer from high dimensionality and implementation complexity.

A popular technique for enhancing a histogram-based local descriptors is dividing the face into overlapping or non-overlapping sub-regions. Shan, Gong & McOwan (2009) studied facial expression recognition with the Local Binary Patterns (LBP) descriptor by extracting features from sub-regions of a face image. In this study, an LBP histogram was created with 59 bins and extracted from 18 $\times$ 21 pixels sub-regions in a 110 $\times$ 150 image. However, the extraction parameters were not optimized for facial expression recognition. To the best of our knowledge, Carcagnì et al. (2015) were the first to optimize the cell size and histogram bins of the Histogram of Oriented Gradients (HOG) descriptor for facial expression recognition. Results show that the best HOG parameters are independent of the datasets used. The study goes further to optimize three other local descriptors and compare them with the HOG descriptor. However, the optimization of the other descriptors is not given in detail, leaving some questions unanswered, such as whether those local descriptors can also be optimized in a way that is independent of the datasets used.

Benitez-Garcia, Nakamura & Kaneko (2017) proposed a FER system that extracts appearance and geometric features around specific facial regions. This approach segments three facial regions: eye-eyebrows, nose, and mouth, dividing them into sub-regions. The Local Fourier Coefficients and Facial Fourier Descriptors are extracted from each sub-region, and the feature vectors of all the sub-regions are concatenated. It should be noted that the resulting feature vector is a linear difference between the expressive and neutral features. Hence, both the expressive and the neutral face are needed to recognize a given expression, which is not always practical in real-life. Also, the approach could be negatively affected by pose variations. Turan & Lam (2018) studied the performance of several local descriptors when the resolution and number of sub-regions of the face are varied. Five image resolutions were used, and the face was divided into $l\times l$ pixels sub-images where $l$ was varied from 3 to 11 in steps of 1. The study reports that higher resolutions and more sub-regions lead to better classification performance. However, the results do not show a trend in recognition performance as the number of sub-regions varies. Also, there is not enough discussion on how the resolution and sub-region size affects the descriptor’s ability to represent facial features.

Most of the above approaches extract local descriptors by dividing the face images into sub-regions. However, the approaches do not optimize the extraction parameters for FER, except for a few works which attempt to optimize either the sub-block size, image resolution, histogram bins, or a combination of these (Carcagnì et al., 2015; Turan & Lam, 2018). The current work aims to study the importance of optimizing the sub-block size and histogram bins of local descriptors. We believe that these are the most critical parameters for striking a balance between recognition rate and the feature vector length.

Datasets

Two publicly available face datasets are used in this study and organised as described by Badi Mame & Tapamo (2022). The first one is the CK+ dataset (Lucey et al., 2010), which is one of the most used datasets to evaluate FER solutions because it contains subjects from a variety of ethnicities, ages, and genders. This dataset is formed from 123 subjects and is composed of 593 image sequences. These images were captured under controlled and uncontrolled conditions, posed and non-posed statuses. The experiments carried out in this study only used the posed images where the subjects performed six expressions. To obtain a balanced dataset from the available sequences, the following images are selected: the last images in the sequences for anger, disgust, and happiness; the last and fourth images for the first 68 sequences related to surprise; the last and fourth from last images for the images of fear and sadness. The second dataset is the Radboud Faces Database (RFD) compiled by Langner et al. (2010). RFD is a popular dataset among researchers because it contains a variety of expressions, gaze directions, and head orientations. The dataset contains pictures of 67 subjects (both adults and children) displaying eight expressions. A subset was created, selecting 402 images labeled with six expressions, including anger, disgust, fear, happiness, sadness, and surprise. Figures 2 and 3 give some image samples from the CK+ and RFD datasets, respectively. Table 1 summarizes the datasets.

Face detection and registration

An important step in recognizing a facial expression is to locate a face within the input image. Many robust face detectors have been developed (Yang, Kriegman & Ahuja, 2002; Viola & Jones, 2004). In the present study, a general frontal face detector is sufficient since recognition is done on frontal faces only. The face detector is based on the Viola-Jones detector implemented by the OpenCV library. A face region contains all the information necessary to observe a facial expression. However, much of this information can be discarded to allow the model only to use the parts of the image containing facial components. Face registration aims at giving detected faces a predefined pose, shape, and size. According to Carcagnì et al. (2015) the best features are usually extracted from images where the eyes are in a consistent position relative to the image. To address this issue, we proposed a novel face registration algorithm that can be used on gray-scale images. First, we enhanced the image by applying contrast limited automatic histogram equalization a gray-scale image. Next, we located the face blob by performing binary segmentation followed by two filtration techniques (see details further below). Given that binary segmentation is a process that separates an image into two regions based on an interval, the challenge here was to find the correct interval within which the face pixels lie. We propose an algorithm to automatically select the best thresholds quickly and efficiently. Consider an image with pixel intensity $I(x,y)$, the segmented image B, is defined as

(1) $$\begin{array}{r}B(x,y)=\{\begin{array}{cc}1& \mathrm{i}\mathrm{f}{T}_1<I(x,y)<T_2\\ 0& \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\end{array}\end{array}$$where $T_1$ and $T_2$ are the lower and upper thresholds respectively. These thresholds are defined as

(2) $$T_1=\mu -\sigma$$

(3) $$T_2=\mu +2\sigma$$where $\mu$ and $\sigma$ are the mean and standard deviation of the intensity values respectively. Specifically, the mean and standard deviation are obtained by creating a histogram from the face center pattern. The mean and standard deviation are, respectively, defined by Eqs. (4) and (5).

(4) $$\mu =\frac{1}{N}\sum _{x,y}W(x,y)\times I(x,y)$$

(5) $$\sigma =\sqrt{\frac{1}{N}\sum _{x,y}(I(x,y)-\mu {)}^2}$$where N is the number of pixels in an image, and $W(x,y)$ is the frequency of the bin (or interval) where the intensity $I(x,y)$ lies.

The binary segmentation operation is not perfect hence unwanted regions must be filtered out. The first filter is an ellipse which is computed from the face’s dimensions and location. Elements inside the ellipse are retained while elements outside the ellipse are discarded. The second filter consists of searching for the largest connected components (Bolelli et al., 2019) of the graph-encoded binary image. The blob’s shape is then refined by estimating the polygonal shape around it. Specifically, a polygon is estimated by finding a set of contour points (Lim, Zitnick & Dollár, 2013) enclosing the blob, computing its convex hull (Sklansky, 1982) and approximate a polygonal shape. Polygon estimation is important because it fills any holes in the blob and makes the ellipse-fitting step more reliable.

A vertical rotation of the face is carried out, by fitting an ellipse around the face and using its angle of inclination (i.e., the angle between the minor axis and the x-axis). To refine the process, the image is rotated a second time using the eye locations. Elliptical cropping followed by rectangular cropping are then performed to generate an image in which eyes are in predefined position relative to the sides of the image. Finally, the output image is resized. The approach used to perform the face detection and registration is shown in Fig. 4.

Feature extraction

The goal of feature extraction is to generate a feature vector from a facial image for classification. This study is interested in local descriptors used for facial expression recognition. Four local descriptors were examined: Histogram of Oriented Gradients (HOG), Local Binary Patterns (LBP), Compound Local Binary Patterns (CLBP), and Weber’s Local Descriptor (WLD). As previously introduced in Badi Mame & Tapamo (2022), this study uses a method that divides the face into a number of non-overlapping and equally-sized sub-blocks, and a local descriptor is extracted from each sub-block, producing several histograms using Eq. (6). The resulting feature vector is formed by normalizing the histograms and concatenating them into a 1D feature vector using Eq. (7). Histograms of sub-blocks are calculated separately and normalized such that the frequencies sum to one. This rule is applied to all the types of descriptors except the HOG descriptor, where the blocks are normalized in groups using L2-Hys (Dalal & Triggs, 2005). Suppose $I_j$ is a matrix, of dimension $W_S\times W_S$, of intensities that represents the $j$-th sub-block in the image, and the function $D_j(x,y)$ calculates the descriptor at a position $(x,y)$ in $I_j$, then the histogram associated with this sub-block is defined as

(6) $$H^j=\{h_i^j{\}}_{i=0,1,\dots ,N_b-1}$$

$$h_i^j=\sum _{x=0}^{W_S-1}\sum _{y=0}^{W_S-1}\delta (D_j(x,y),i)$$where $\delta (d,i)$ is defined as

$$\begin{array}{r}\delta (d,i)=\{\begin{array}{cc}1& \mathrm{i}\mathrm{f}L(i)\le d<U(i)\\ 0& \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\end{array}\end{array}$$with

$$L(i)=i\times \frac{n}{N_b}\mathrm{a}\mathrm{n}\mathrm{d}U(i)=(i+1)\times \frac{n}{N_b}$$where $n$ is the number of possible descriptor values and $N_b$ is the number of histogram bins.

The resulting feature vector is defined as

(7) $$H=\left\{\frac{H^0}{{(W_S)}^2},\frac{H^1}{{(W_S)}^2},\dots ,\frac{H^{m-1}}{{(W_S)}^2}\right\}$$where $m$ is the number of sub-block¹ .

Local Binary Pattern

The Local Binary Pattern descriptor was first introduced to describe textural appearance in pattern recognition problems. The LBP histogram contains information relating to the distribution of the local micro-patterns, such as flat areas and edges in an image. The original LBP descriptor is computed by considering each pixel of an input image $g_c$ as the ‘threshold’ of a $3\times 3$ grid and assigning a value of 1 or 0 to each of the eight surrounding pixels. The following rule is assumed: if the neighboring pixel is less than the threshold it is assigned the value 0, otherwise 1 is assigned. A binary code is then read from the sequence of 0s and 1s and converted to a decimal number. The decimal number is then assigned to the center (or threshold) pixel. Once this operation has been performed on the entire image, the LBP image is obtained. The histogram of LBP image is then used as a feature vector (Ojala, Pietikäinen & Mäenpää, 2000). Figure 5 gives an example of computing the LBP code of a pixel in a $3\times 3$ neighborhood block.

LBP can be extended to allow any number of neighbors in a circular neighborhood and bilinear interpolation of pixel values (Ojala, Pietikainen & Maenpaa, 2002). In this study, the LBP descriptor was implemented with three neighbors and a radius of 1. The new LBP function is defined as

Algorithm 2 :

Local binary patterns.

Input Image: I

Output Feature vector: $(H^0{H}^1\cdots H^{N_b-1})$

Divide the input image into blocks $(I_b{)}_{b=0,1,\dots ,N_b-1}$.

for $\mathrm{e}\mathrm{a}\mathrm{c}\mathrm{h}\mathrm{b}\mathrm{l}\mathrm{o}\mathrm{c}\mathrm{k},\mathrm{b}$ do

Compute the LBP codes $f_c^b$ for the block b using Eq. 17.

Compute the histogram $H^b$ using Eq. 18.

end for

Concatenate the histograms, $H=(H^b{)}_{b=0,1,\dots ,N_b-1}$.

DOI: 10.7717/peerj-cs.1388/table-8

(17) $$f_c=\sum _{j=0}^{P-1}S(g_j-g_c)2^j$$where $1\le x\le w,1\le y\le h$ and $S(A)$ is defined as

$$S(A)=\{\begin{array}{l}1\text{\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}if}A\ge 0\\ 0\text{\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}else}\end{array}$$and $g_j$ corresponds to the neighboring intensity values; $w$ and $h$ are the width and height of the input image respectively.

After computing all the LBP codes, $f_c(x,y)$, a histogram, $(H_i{)}_{i=0,1,\dots ,n-1}$, is created that counts the occurrences of a range of LBP codes.

(18) $$H_i=\sum _{x=0}^{w-1}\sum _{y=0}^{h-1}I(f_c(x,y)=i)$$where $n$ is the number of bins in the histogram and $I(z)$ is defined as:

$$I(z)=\left\{\begin{array}{cc}1& \mathrm{i}\mathrm{f}\mathrm{Z}\mathrm{i}\mathrm{s}\mathrm{t}\mathrm{r}\mathrm{u}\mathrm{e}\\ 0& \mathrm{e}\mathrm{l}\mathrm{s}\mathrm{e}\end{array}\right\}$$

Figure 6 shows the steps followed to compute LBP features of an image.

Compound local binary pattern

Compound Local Binary Patterns (CLBP) extends the LBP descriptor. CLBP differs from LBP in that it encodes both the magnitude as well as the sign of the differences between a center (or threshold) pixel and its P neighbors. Each pixel is treated as a center having P neighbors. Each neighbor is given a 2-bit code as follows: the first bit is set to 0 if the neighboring pixel is less than the center pixel and is set to 1 otherwise. The second bit is set to 1 if the magnitude of the difference between the neighboring pixel and the center pixel is greater than the threshold, $M_{avg}$, otherwise, the bit is set to 0. The threshold value is defined as the average magnitude of the differences between the center pixel and its P neighbors. The rest of this discussion considers $P=8$, since the same is used in our implementation. Figure 7 illustrates the computation of CLBP features of an image.

Algorithm 3 :

Compound local binary patterns.

Input Image: I

Output Feature vector: $H=(H^0{H}^1\dots H^{N_b-1})$

Divide the input image into blocks $(I_b{)}_{b=0,1,\cdots ,N_b-1}$.

for $\mathrm{e}\mathrm{a}\mathrm{c}\mathrm{h}\mathrm{b}\mathrm{l}\mathrm{o}\mathrm{c}\mathrm{k},\mathrm{b}$ do

Compute the sub-CLBP codes using Eqs. (20) and (21), respectively.

Create a histogram for the codes, $H_1^b$ and $H_2^b$.

Concatenate the sub-histograms, $H^b=\{H_1^b{H}_2^b\}$.

end for

Concatenate the 1D histograms, $H=\{H^b{\}}_{b=0,1,\dots ,N_b-1}$

DOI: 10.7717/peerj-cs.1388/table-9

Let $f_c(x,y)$ be the function representing the CLBP code of a pixel positioned at $(x,y)$ defined as:

(19) $$f_c(x,y)=\sum _{p=0}^{P-1}s(i_p,i_c)2^{2p}$$where $s(i_p,i_c)$ is defined as

$$s(i_p,i_c)=\{\begin{array}{cc}00& \mathrm{i}\mathrm{f}{i}_p-i_c<0\mathrm{a}\mathrm{n}\mathrm{d}|i_p-i_c|\le M_{avg}\\ 01& \mathrm{i}\mathrm{f}{i}_p-i_c<0\mathrm{a}\mathrm{n}\mathrm{d}|i_p-i_c|>M_{avg}\\ 10& \mathrm{i}\mathrm{f}{i}_p-i_c\ge 0\mathrm{a}\mathrm{n}\mathrm{d}|i_p-i_c|\le M_{avg}\\ 11& \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\end{array}$$

$i_c$ and $i_p$ represent the gray values of the center $c$, and a neighbor $p$, and $M_{avg}$ is the average magnitude of the differences between $i_c,i_p$ pairs in the local neighborhood.

The CLBP binary code is further subdivided into two sub-CLBP patterns $f_{c1}$ and $f_{c2}$. Hence, during implementation, Eq. (19) is not used but instead Eqs. (20) and (21) are used. The sub-CLBP patterns can be defined as follows:

(20) $$f_{c1}=\sum _{p\in O,k\in E}\left(S_{1p}(i_p,i_c)2^{k+1}+S_{2p}(i_p,i_c)2^k\right)$$

(21) $$f_{c2}=\sum _{p\in O,k\in E}\left(S_{1p}(i_p,i_c)2^{k+1}+S_{2p}(i_p,i_c)2^k\right)$$where $O=\{1,3,5,7\},E=\{0,2,4,6\}$ and $S_{1p}(i_p,i_c)$ and $S_{2p}(i_p,i_c)$ are defined as

$$S_{1p}(i_p,i_c)=\{\begin{array}{cc}0& \mathrm{i}\mathrm{f}{i}_p-i_c<0\\ 1& \mathrm{i}\mathrm{f}{i}_p-i_c\ge 0\end{array},S_{2p}(i_p,i_c)=\{\begin{array}{cc}0& \mathrm{i}\mathrm{f}|i_p-i_c|\le M_{avg}\\ 1& \mathrm{i}\mathrm{f}|i_p-i_c|>M_{avg}\end{array}$$with

$$M_{avg}=\frac{1}{8}\sum _{p=0}^7|i_p-i_c|$$

Weber’s Local Descriptor

Weber’s Local Descriptor (WLD) is inspired by the human perception of a pattern, which depends not only on the change of stimulus (such as sound and lighting) but also on the original intensity of the stimulus (Chen et al., 2009). WLD is made of two components: a differential excitation ( $\xi$) and a gradient orientation ( $\theta$). The differential component is expressed as the ratio between two terms: the first is the relative intensity differences of a current pixel against its neighbors; the other is the intensity of the current pixel. The second component of WLD is the gradient orientation. A feature vector is generated by first rearranging the differential excitations into subgroups that are used to form histograms, following a particular rule that is further discussed below. Histograms are ultimately reordered and concatenated to form a single feature vector. Figure 8 is an illustration of how WLD feature vectors are extracted from an input image. Algorithm 4 summarized the extraction of WLD features.

Algorithm 4 :

Weber’s Local Descriptor.

Input Image: I

Output Feature vector: $H=(H^0{H}^1\cdots H^{N_b-1})$

Divide the input image into blocks $(I_b{)}_{b=0,1,\cdots ,N_b-1}$.

for $\mathrm{e}\mathrm{a}\mathrm{c}\mathrm{h}\mathrm{b}\mathrm{l}\mathrm{o}\mathrm{c}\mathrm{k},\mathrm{b}$ do

Compute the differential excitations, ${\xi }_j$ using Eq. (22).

Compute the gradient orientations, ${\theta }_j$ using Eq. (23).

Map the gradient orientation, ${\theta }_j^{{}^{\prime }}$ using Eq. (24).

Quantize the mapped orientations into T dominant orientations, producing ${\mathrm{\Phi }}_t$ using Eq. (25).

Create sub-histograms Hm,t using Eq. (26)

Concatenate the sub-histograms $H^b=\{H_{m,t}{\}}_{t=0,1,\dots ,T-1;m=0,1,\dots ,M-1}$

end for

Concatenate the 1D histograms, $H=\{H^b{\}}_{b=0,1,\dots ,N_b-1}$

DOI: 10.7717/peerj-cs.1388/table-10

Consider a $3\times 3$ pixels block $G_s$ taken from an input image G, the WLD operator can be applied as follows:

• Compute the differential excitation component which is expressed as:

(22) $$\xi (x_c)=\mathrm{t}\mathrm{a}{\mathrm{n}}^{-1}\left(\frac{V_s^{00}}{V_s^{01}}\right)=\mathrm{t}\mathrm{a}{\mathrm{n}}^{-1}\left[\sum _{i=0}^{p-1}\left(\frac{x_i-x_c}{x_c}\right)\right]$$

• The gradient orientation $\theta (x_c)$ which is the second component of WLD is computed as follows:

(23) $$\theta (x_c)=\mathrm{t}\mathrm{a}{\mathrm{n}}^{-1}\left[\frac{V_s^{11}}{V_s^{10}}\right]=\mathrm{t}\mathrm{a}{\mathrm{n}}^{-1}\left[\frac{x_5-x_1}{x_7-x_3}\right]$$

$$G_s=\left[\begin{array}{ccc}{x}_0& x_1& x_2\\ x_7& c& x_3\\ x_6& x_5& x_4\end{array}\right]$$

$$\xi (x_c)\in [-\frac{\pi }{2},\frac{\pi }{2}]\theta (x_c)\in [-\frac{\pi }{2},\frac{\pi }{2}]$$

• Map the gradient orientations from the interval $[-\frac{\pi }{2},\frac{\pi }{2}]$ to the interval $[0,2\pi ]$ by applying the following function:

(24) $${\theta }^{\prime }=\mathrm{a}\mathrm{r}\mathrm{c}\mathrm{t}\mathrm{a}\mathrm{n}2(V_s^{11},V_s^{10})$$

$$\mathrm{a}\mathrm{r}\mathrm{c}\mathrm{t}\mathrm{a}\mathrm{n}2({\mathrm{V}}_{\mathrm{s}}^{11},{\mathrm{V}}_{\mathrm{s}}^{10})=\{\begin{array}{cc}\theta & if{V}_s^{11}>0and{V}_s^{10}>0\\ \pi -\theta & if{V}_s^{11}>0and{V}_s^{10}<0\\ \theta -\pi & if{V}_s^{11}<0and{V}_s^{10}<0\\ 2\pi +\theta & if{V}_s^{11}<0and{V}_s^{10}>0\end{array}$$

$$V_s^{11}=x_5-x_1;V_s^{10}=x_7-x_3$$

• The gradients ${\theta }^{{}^{\prime }}$ are further quantized into T dominant orientations using the function.

(25) $${\mathrm{\Phi }}_t=f_q({\theta }^{{}^{\prime }})=\frac{2t}{T}\pi$$

$$t=\mathrm{m}\mathrm{o}\mathrm{d}(\lfloor \frac{{\theta }^{{}^{\prime }}}{2\pi /\mathrm{T}}\rfloor ,\mathrm{T})$$where $t$ is a linear function that maps a differential excitation with orientation ${\theta }^{{}^{\prime }}$ to a particular dominant orientation ${\mathrm{\Phi }}_t$. Next, a 2D histogram ${\left(\mathrm{W}\mathrm{L}\mathrm{D}({\xi }_j,{\mathrm{\Phi }}_t)\right)}_{j=0,1,\dots ,N-1,t=0,1,\dots ,T-1}$ is created. N is the dimensionality (number of pixels) in an image and T is the number of dominant orientations. The 2D histogram has C rows and T columns, where C is the number of cells in each orientation. The value of C is explained in the subsequent part of this section. Intuitively, each cell of the 2D histogram represents the frequency of a differential excitation interval on a dominant orientation.

The 2D histogram is further encoded into a 1D histogram, H as follows. First, each column of the 2D histogram is projected to form a 1D histogram, ${\left(H(t)\right)}_{t=0,1,\dots ,T-1}$. Each histogram is produced by grouping differential excitations ${\xi }_j$ corresponding to a dominant orientation ${\mathrm{\Phi }}_t$.

Next, each sub-histogram, $H(t)$, is divided into M segments ${\left(H_{m,t}\right)}_{m=0,1,\dots ,M-1}$. The sub-histograms $H_{m,t}$ collectively form a histogram matrix. Each row of this matrix corresponds to a differential excitation segment and each column corresponds to a dominant orientation. The histogram matrix is reorganized as a 1D histogram by first concatenating the rows of the histogram matrix as a histogram, $H_m=\{H_{m,t}{\}}_{t=0,1,\dots ,T-1}$. The resulting M sub-histograms are concatenated as a 1D histogram, $H=\{H_m{\}}_{m=0,1,\dots ,M-1}$.

Note, that each sub-histogram, $H(t)$, is divided into M segments, $H_{m,t}$, by dividing the interval of differential excitations $l=[-\frac{\pi }{2},\frac{\pi }{2}]$ into M segments $l_m$. Furthermore, each sub-histogram $H_{m,t}$ is an S-bin histogram defined as:

(26) $$H_{m,t}=\{h_{m,t,s}\}$$

$$h_{m,t,s}=\sum _j\delta (S_j=s)$$where $S_j$ and $\delta (A)$ are defined as

$$S_j=\mathrm{m}\mathrm{o}\mathrm{d}(\lfloor \frac{\xi -{\eta }_{m,l}}{({\eta }_{m,u}-{\eta }_{m,l})/S}\rfloor ,S)$$

$$\delta (A)=\{\begin{array}{cc}1& \mathrm{i}\mathrm{f}\mathit{A}\mathrm{i}\mathrm{s}\mathrm{t}\mathrm{r}\mathrm{u}\mathrm{e}\\ 0& \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\end{array}$$

$$\mathrm{w}\mathrm{i}\mathrm{t}\mathrm{h}{\eta }_{\mathrm{m},\mathrm{l}}=\left(\frac{\mathrm{m}}{\mathrm{M}}-\frac{1}{2}\right)\pi \mathrm{a}\mathrm{n}\mathrm{d}{\eta }_{\mathrm{m},\mathrm{u}}=\left(\frac{\mathrm{m}+1}{\mathrm{M}}-\frac{1}{2}\right)\pi$$

Support vector machines

The support vector machine (SVM) is a machine learning algorithm whose objective is to learn how to classify input features by defining a separating hyperplane. Considering $l$ training feature vectors $x_1,x_2,\dots ,x_l$ with their corresponding labels $y_1,y_2,\dots ,y_l$, where $y_i\in \{-1,1\}$ for $i=1,2,\dots ,l$; the following primal optimization problem can be defined as:

(27) $$\mathrm{m}\mathrm{i}{\mathrm{n}}_{w,b,\xi }=\left\{\frac{1}{2}{w}^{T}w+C\sum _{i=1}^l{\xi }_i\right\}.$$

Subject to

$$y_i(w^T\mathrm{\Phi }(x_i)+b)\ge 1+{\xi }_i$$

$${\xi }_i>0,i=1,\cdots ,l$$where ${\xi }_i$ and $\xi$ are the classification error for the $i^{th}$ training vector, and the total classification error, respectively; $w$ is the normal vector to the hyperplane; $\frac{b}{||w||}$ represents the offset of the hyperplane from the origin along the normal vector $w$ ( $||\cdot ||$ being the norm operator); $\mathrm{\Phi }(x_i)$ is the kernel function, that maps $x_i$ into a higher dimensional space and C is the regularization parameter. The classical SVM approach requires only two classes. So, a set of SVMs and a one-against-one strategy can make support vector machines suitable for multi-class problems such facial expression recognition (Knerr, Personnaz & Dreyfus, 1990). The experiments discussed here use the multi-class SVM strategy. More precisely, scikit-learn’s implementation of the LIBSVM library (https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html) was used.

To train an SVM classifier, a set of hyper-parameters must be selected: the kernel function, the regularization parameter (C), and the kernel coefficient (gamma). For each classifier, an original parameter optimization procedure is performed. Specifically, a grid-search cross-validation procedure is employed (Hsu, Chang & Lin, 2003). In grid-search cross-validation, all possible combinations of the parameters are used to train a classifier using 10-fold cross-validation. Then, the parameter values which produce the best average recall are selected. Table 2 gives the parameter selection for when each type of feature is used to train an SVM classifier.

Experimental phase 1: the effect of face registration

This section studies the effect of face registration on the FER system. To achieve this, the system is first trained on features extracted from unregistered faces and the performance is recorded. Then, the system is trained on features of registered faces. Note than the non-registered images are resized to 65 $\times$ 65 pixels, while the registered images are resized to 59 $\times$ 65 pixels to strike a balance between performance and feature vector length. Also, the parameters used for feature extraction are those recorded in Table 2.

Table 3 shows each expression’s difference in recognition rate, showing that registration either increases the recognition rate (e.g., fear) or has no effect (e.g., HOG happiness). The most significant increase is observed for the LBP descriptor for the anger expression. The results reveal that face registration leads to a better recognition rate regardless of the descriptor used. These results suggest that face registration has a positive (and non-negligible) impact on the feature extraction phase and, ultimately, on the recognition performance. Figure 9 shows an example of a registered image compared to a non-registered one.

Experimental phase 2: optimizing spatial local descriptors

This section evaluates the performance of four local descriptors: HOG, LBP, CLBP, and WLD. Several classifiers are trained on features from different descriptors, and the average recalls during 10-fold cross-validation are recorded. The classifiers are trained on features extracted from two datasets: CK+ dataset of six expressions and RFD dataset of seven expressions. A parameter selection is performed on each descriptor to find the configuration which gives the best performance. There are two parameters for the extraction of LBP and CLBP features: the histogram bin size and the block size. Bin sizes of 8, 16 and 32, and block sizes within the range 2 to 24 (inclusive) are available for selection (where the block size refers to the side length of a square). For the HOG descriptor, the number of orientation bins and cell size are tuned. The orientations used are 3, 5, 7, 9, 12, 15 and 55 and the cell size are varied between four and 15 (inclusive). Concerning the WLD descriptor, four parameters must be tuned (or optimized): T, M, S, and the block size. Block sizes ranging from 2 and 24 (inclusive) are available for selection while T, M and S can take values of 4, 6, and 8.

Figures 10A–10D give the FER results (expressed as average recall) when each descriptor is optimized on the CK+ dataset of six expressions. The best parameters for extracting HOG descriptor are reported as nine orientations and a cell size of 8, giving a recognition rate of 97.4%. Concerning the LBP features, the recognition rate has a maximum value of 96.1% when the number of bins is 16 and the block size of 7 pixels. The best recognition rate for the CLBP descriptor is 94.1% when the number of bins is eight, and the block size is 11. Finally, the FER results when using WLD features, where the best performance (96.9%) when the parameters are block size of 8, $T=8$, $M=4$, and $S=4$. The same parameter optimization procedure is repeated; the features are extracted from the RFD dataset of six expressions. Figure 11A shows that the best parameters of HOG are 7 orientations and a cell size of 8 (RR = 97.2%). From Fig. 11b, the best LBP parameters are reported as 16 bins and a block size of 8 (RR = 94.6%). The best parameters for CLBP are 8 bins and 6-pixel blocks as shown in Fig. 11c (RR = 95.6%). Finally, Fig. 11d shows the best parameters of WLD as a block size of 7, $T=6$, $M=4$, and $S=4$ (RR = 96.5%). The results show that the recognition rate rises and falls as the block size varies. This suggests that smaller blocks lead to too much fine-grain information, which is less useful in representing Action Units. On the other hand, larger blocks may merge multiple Action Units, which is less useful because it mixes discriminatory and non-discriminatory information. A similar finding was presented in Moore & Bowden (2011).

The above results mean that the choice of parameter values improved local descriptors’ performance. Tables 4 and 5 give the best parameter selection and feature vector length for each descriptor on each dataset. It can be observed that the most frequent value for block size is 8, and other values are close, e.g., 6 and 7. This means that a block size of $8\times 8$ pixels could be the best window for extracting features at the current resolution. On the other hand, the number of histogram bins is different for all descriptors.

Comparison with the state-of-the-art

Comparing different FER systems in the literature is a challenging task because many studies omit or poorly discuss their evaluation strategy. Also, the training-testing data split often varies from one study to another. For this reason, only approaches evaluated using a 10-fold cross-validation method and the recognition rate (or average recall) were compared with this study. Table 6 gives various recent state-of-the-art approaches evaluated on the extended Cohn-Kanade (CK+) dataset (when a value is not available, it is replaced by ‘−’). All the approaches used SVM for classification, showing its superior performance, especially when working with small datasets. The proposed approach using the HOG descriptor achieved the best recognition rate, improving the results in Carcagnì et al. (2015) by a difference of 1.6%. We also note that the proposed approach gives a better recognition rate for LBP, CLBP, and WLD descriptors than previously obtained results. This improvement is probably due to this study’s optimum selection of extraction parameters.

As observed in experimental results, the recognition rates first rise then fall as the block sizes are varied. Hence, the block size must be selected carefully when extracting each descriptor. Unlike previous studies, the current approach empirically determines the extraction parameters for LBP, CLBP, WLD, as opposed to determining them by trial and error. Another important aspect that makes the superiority of the proposed method is the histogram normalization. The HOG descriptor provides different histogram normalization methods which enhance its performance (Dalal & Triggs, 2005). Like previous methods, the current approach used a technique called L2-Hys normalization which has proven to be superior than the others. However, the better performance shown by this approach could be attributed to the choice of the number of cells per block ( $n_c$) used during the normalization process. Different $n_c$ values have shown to produce a measurable impact on the performance of HOG, and in this study $n_c=3$ gave the best results. The other local descriptors were previously not normalized. Hence, we introduced a simple normalization method ² which also enhanced their performances. This is evident because the results obtained with normalization were better than those obtained without normalization.

Factors affecting facial expression recognition performance

The performance of the system is seriously affected by three main factors: the selection of datasets, the face registration, and the feature extraction parameters. Incorrect labeling also tends to introduce ambiguity between expressions and so can bring down the accuracy of predictions. The registration step is a crucial component to increase the classification performance. A correctly registered frontal face image must have all the facial components to guarantee extraction of the most key features and high prediction accuracy. This finding was pointed in Carcagnì et al. (2015) and our results confirm that the recognition performance is higher when using registered images than when the images are not registered.