Face-voice based multimodal biometric authentication system via FaceNet and GMM

Gaussian mixture model (GMM)

A parametric probability density function is represented by a Gaussian mixture model, which is the aggregated sum of Gaussian component densities. The probability distribution of continuous measures or features, such as vocal-tract associated spectral data in a speaker recognition system, are typically modeled using GMMs in biometric systems. GMM parameters are estimated from training data using the iterative expectation-maximization (EM) approach or Maximum a Posteriori (MAP) prediction from a learned prior model. The first step in recognition is to scale audio and turn it into MFCC characteristics in order to simplify the model. Then, from the provided feature vector matrix, extract the 40-dimensional MFCC and the delta MFCC features, combine them, and supply the combined data as input to the GMM model.

Speech acquisition and feature extraction

The features are extracted from the voice signals to be converted to a sequence of acoustic feature vertures which is utilized to identify the speaker. The verification process starts with MFCC parameters.

1. MFCC parameters

   1. Pre-emphasis

      This step enhances high frequencies of the spectrum of the form 1: (1)${\displaystyle x_p\left(t\right)=x\left(t\right)-ax\left(t-1\right)}$ where a is assigned a value of 0.98.

   2. Framing

      Then, the signal is divided into frames. The shift is 10 ms, with a frame length of 20 ms. We utilize framing due to fluctuation in frequencies in a signal over a period of time.

   3. Windowing

      Since framing cuts the signals into frames in most cases the start of the following frame usually does not coincide with the finish of the previous frame. Thus, windowing is required to produce an accurate value for the original signal frequency and educe spectral leakage. The Hamming window as follows, (2)${\displaystyle w\left[n\right]=a_0-\underset{\simeq a_1}{\underbrace{\left(1-a_0\right)}}.cos\left(\frac{2\pi n}{N}\right),0\le n\le N}$ Where a₀ = 0.53836.

   4. Fast Fourier Transform (FFT)

      The FFT is utilized to convert the signal from its original domain to a representation in the frequency domain. The Cooley–Tukey algorithm is utilized where it divide-and-conquer and recursively breaks down a Discrete Fourrier Transform (DFT) of any composite size into many smaller DFTs of sizes.

   5. Modulus

      The magnitude is determined by computing the FFT’s absolute value. The power spectrum that has been sampled over 512 points is obtained. Due to the symmetry of the spectrum, only half of those points are relevant.

   6. Mel Filters

      There are several of oscillations in the spectrum at that moment, and it is not desired. The size of the spectral vectors must be reduced, hence a smoothing technique is required. As a result, the spectrum is divide by a filterbank, which is a collection of bandpass frequency filters. For frequency localisation, the Bark/Mel scale was used, which is analogous to the human ear’s frequency scale. The frequency heard by the human ear is the basis for the measuring unit known as the Mel.

   7. Discrete Cosine Transform (DCT)

      The log of the spectrum is transformed using a discrete cosine transform. The MFCCs are obtained, and because the majority of the information is obtained during the first few coefficients, we choose the first few 12. In this step, the MFCC coefficients describing the input signal window is obtained.

   8. Cepstral Mean Subtraction (CMS)

      Finally, each vector has the cepstral mean vector subtracted from it. Furui (1981), which is especially important in speaker verification assignments. Cepstral mean subtraction (CMS) is a step that removes slowly fluctuating convolutive sounds.

   9. Cepstral mean variance normalization (CMVN)

      By linearly converting the cepstral coefficients to have the same segmental statistics, cepstral mean variance normalization (CMVN) eliminates noise contamination for robust feature extraction. Short utterances, however, are known to decrease the performance of speaker verification tasks.

Face detection

Face detection is the process of automatically detecting and localizing faces in a photograph by drawing a bounding box around their extent. We utilized the Haar Cascade classifier to perform this task. It is an object detection algorithm (ODA), a method for finding faces in pictures or live video. In their research work, Viola & Jones (2001) suggested edge or line detecting features. Finding Haar features, creating integral images, training Adaboost, and classifying the images using a cascading classifier are the four processes of the cascade function for image training (Priambodo et al., 2021), as illustrated in Fig. 6. The most crucial step while utilizing the algorithm is that the classification A significant number of datasets with positive facial photos and negative non-face images are required for the training procedure. The Haar Cascade classifier suites the system since only frontal face images are required for detection. For each identified face, the FaceNet model will be utilized to construct a face embedding, and then a linear support vector machine (SVM) classifier model will be developed to predict the identity of that face. After the face is detected the image’s region of interest (ROI) is resized to 96*96.

Face embeddings

The features retrieved from the face are represented by a face embedding, which is a vector. The vectors created for other faces can then be compared. The detected faces from the previous step is loaded and the FaceNet model generates the embbedings for each detected face. To predict an embedding, the image’s pixel values must first be appropriately prepared to fulfill the FaceNet model’s expectations.

Face classification

In this phase our model classifies face embeddings as one of the users stored in the database. The face embedding vectors are normalized utilizing L2 norm. Since vectors are compared utilizing distance metrics, we normalize the vectors by scaling the values until the length of vector is 1.

Fusion data

Data fusion requires computing the EER of the fusion of those two classifiers.

• Voice recognition gives as an output a log-likelihood (GMM).

• Face recognition gives a distance in a feature space.

First we convert log likelihood to probabilities by transforming the log-likelihood to a discrete probability with: (5)${\displaystyle P\left(x|{\theta }_1\right)=e^{l\left(x|{\theta }_1\right)}.}$

The probability is further normalized with: (6)${\displaystyle P\left(x|{\theta }_1\right)=\frac{e^{l\left(x|{\theta }_1\right)}}{\sum _l{e}^{l\left(x|{\theta }_1\right)}}.}$

The results are too small, so the maximum is factored out by: (7)${\displaystyle P\left(x|{\theta }_1\right)=\frac{e^{lmax}.e^{l{\left(x|{\theta }_1\right)}^{lmax}}}{e^{lmax}.\sum _l{e}^{l{\left(x|{\theta }_1\right)}^{lmax}}}=\frac{e^{l{\left(x|{\theta }_1\right)}^{lmax}}}{\sum _l{e}^{l{\left(x|{\theta }_1\right)}^{lmax}}}.}$

Then converting distance to probabilities utilizing the softmax of the negative distance by the formula: (8)${\displaystyle P\left(x_i\right)=\frac{e^{-d_l}}{\sum _j{e}^{-d_j}}.}$

Finally, the probabilities are combined with a weighted average. The GMM tends to give extreme probabilities thus, more weight is added on face recognition to counteract this effect.