Voice spoofing detection using a neural networks assembly considering spectrograms and mel frequency cepstral coefficients

Related work

Interest in biometric recognition of speech and speakers is not new. In 2007, a liveness verification system, based on lip movement, was proposed in Faraj & Bigun (2007) as a means of protection against identity theft attempts, particularly through the generation of videos for this purpose.

Given the importance of the problem, the establishment of a standardized database became necessary. In 2015, the National Institute of Informatics initiated two challenges, namely the Voice Conversion Challenge and the ASVspoof Challenge, with the aim of providing an evaluation platform and metrics to facilitate a fair comparison among the proposed techniques related to media cloning and detection.

The Voice Conversion Challenge (Toda et al., 2016) is a biennial event that started in 2016, in this challenge participants are provided with a database and tasked with developing voice converters using their own methods. The organizers then evaluate and classify the transformed speech submitted by the participants.

The ASVspoof challenge (Wu et al., 2015), which is also a biennial event, is highly relevant to this work. The challenge provides a database comprising pairs of genuine and false or generated audios, which participant’s models must accurately classify. Since the release of the ASVspoof challenge databases, investigations have yielded remarkably favorable results. The latest version of the databases was published in 2021.

In Zhang, Yu & Hansen (2017), the authors propose an architecture that combines convolutional neural networks (CNN) and recurrent neural networks (RNN) simultaneously. To evaluate the effectiveness of their method, they utilized the ASVspoof 2015 database, where the input of their model consisted of spectrograms extracted from the audios. Due to the widely varying durations in this database, the authors decided to standardize the duration to four seconds for all audio samples. Through their experiments, they achieved an equal error rate (EER) of 1.47%.

It is worth highlighting the efforts made to create new detection models. For instance, in Lavrentyeva et al. (2017), the authors proposed a simplified version of the Light CNN architecture that employs Max Feature Map (MFM) activation. The authors implemented this network to classify audios into two possible outcomes: genuine or false, specifically to prevent spoofing, they reported an equal error rate of 6.73% in the ASVspoof 2017 database.

In addition to proposing novel models, another factor to consider is the level of complexity required for the proposals. In Pang & He (2017), the authors demonstrated that the implementing very deep or complicated neural networks is not necessarily essential for impersonation detection in identity verification. They explained that satisfactory results can be achieved with simple models. Their model consisted of an input layer, two CNN layers, a gated recurrent unit (GRU) layer, and a final layer. Despite its apparent simplicity, this model yielded excellent performance, with an EER of 0.77% on a corpus of 28,000 audios extracted from the APSRD (Authentic and Playback Speaker Recognition Database).

The alternative approach involves employing increasingly deeper neural networks; however, this can give rise to the vanishing gradient problem. To address this difficulty, residual neural networks (ResNet) have emerged. They have proven to be successful in image recognition, as demonstrated in Chen et al. (2017), where their effectiveness for spoofed audio detection was investigated. The evaluation of this research was conducted using the ASVspoof 2017 dataset, revealing that Resnet achieved one of the best performances among the systems employing a single model.

So far, it has not been mentioned whether high-quality audio is truly essential to carrying out an audio attack. However, Lorenzo-Trueba et al. (2018) explored the potential of utilizing solely low-quality data to train models against spoofing. For this purpose, they developed a system based on generative adversarial networks (GAN), to enhance the quality of audio files accessible on the internet.

Understanding the significance of audio quality is crucial to comprehend the nature of the challenges at hand. This is particularly relevant because voice-controlled devices (VCDs) including popular examples like Alexa, Siri, and others are increasingly prevalent. These devices are primarily utilized for automating home appliances and other entertainment tools. Since voice attacks do not necessitate high-quality audio, it can be inferred that these devices might be vulnerable to such attacks.

In an analysis conducted by Malik et al. (2020), multi-hop replay attacks were shown to be a vulnerability since they are carried out using VCDs with the intention of accessing other VCDs connected to the internet. For example, a device could be used to replicate the voice of the speaker, giving an order or command to a second VCD, and the latter would fulfill its function without verifying whether the instruction truly originates from the speaker or is simply a repetition of the voice of the user.

After establishing the vulnerability of VCDs, (Gong, Yang & Poellabauer, 2020) presents another concern regarding the number of channels employed in attacks on these devices. They present a neural network-based model designed for the specific purpose of detecting multichannel audio. The proposed model allows for an arbitrary number of input channels, in addition, what can be highlighted is that their model is fully developed in a neural network framework, enabling potential integration with other neural network-based models in the future.

Combinations of neural networks are frequently employed in recent and advanced studies. In Dua, Jain & Kumar (2021), the authors examined the performance of various architectures, incorporating deep neural networks (DNNs), long short-term memory (LSTM) layers, temporal convolution (TC), and spatial convolution (SC). To evaluate the performance of their proposal, they used ASVspoof 2015 and ASVspoof 2019 databases, achieving good results in both, with particularly impressive results in the latter.

It is worth mentioning that there are two main research approaches for spoof detection. The first approach involves using diverse architectures and classification models, including the Gaussian mixture model (GMM), support-vector machines (SVM), neural networks such as CNN, RNN, LSTM, GAN, ResNet, and even autoencoders.

The second research approach involves using diverse techniques for audio feature extraction, including different types of spectrograms or coefficients such as mel frequency cepstral coefficients (MFCC), inverse mel-frequency cepstral coefficients (IMFCC), complex cepstral coefficients (CCC), linear frequency cepstral coefficients (LFC), constant Q cepstral coefficients (CQCC), teager energy cepstral coefficients (TECC), linear predictive cepstral coefficients (LPCC), as well as various combinations thereof.

Although some of the extraction techniques and proposed architectures are highly efficient, they can be complex to comprehend and replicate. Therefore, we believe it is crucial to propose a technique that is both easily understandable and reproducible, while maintaining high precision in the specific task of audio spoof detection.

Spectrograms

To obtain the spectrograms, samples are taken through a time window to calculate the frequency content of the samples using the short time Fourier transform (STFT). This process involves extracting and analyzing several signal frames at each window displacement over time. Each frame is added to a matrix that represents the variation in the spectrum and energy of the signal. As new frames are obtained, they are consecutively added to the first position in the array. In this way, the variation of the signal's spectrum and energy can be represented as a function of time.

After conducting tests, it became clear that linear spectrograms alone did not capture all the necessary information to distinguish between a spoof audio and a genuine one. Therefore, we decided to use spectrograms with a logarithmic scale to better capture audio information. As expected, after performing additional experiments, we found that both linear and logarithmic spectrograms were necessary to achieve high accuracy. In the following paragraphs, we provide more details on these findings.

Although there are many representations available, for this work, we considered the time on the abscissa axis as consecutive sequences of Fourier transforms, with the frequency expressed in Hz on the ordinate axis, and finally the representation of the energy expressed in dB represented with a color palette, Fig. 1 illustrates a linear spectrogram with time on the horizontal axis and frequency on the vertical axis.

The main modification for logarithmic spectrograms involves changing the scale of the ordinate axis from a linear to a logarithmic scale. For this study, we considered two different areas of interest. The first area includes values between 10² and 10⁴, as shown in Fig. 2A. This interval was selected because it represents the range where the greatest amount of sound wave energy is concentrated for human voice. The second zone was reduced to values between 10³ and 10⁴ to focus on the zone that corresponds to the highest frequencies and exclude the lowest ones, as shown in Fig. 2B.

MFCC

The most commonly reported feature extraction technique for spoof detection in the specialized literature is the mel frequency cepstral coefficients, originally proposed by Davis & Mermelstein (1980). These are the widely used features to represent the human voice and have shown good results in various environments.

Prior to the extraction of MFCCs, an analog signal is converted into a digital signal through sampling at a specific sample rate. The digital signal is subjected to a series of processes, as illustrated in Fig. 3, to extract the MFCC features.

After dividing the analog signal into overlapping frames, the Discrete Fourier Transform (DFT) of the signal is calculated. Subsequently, the signal is filtered using the Mel filter bank, and the output is log compressed. It is then transformed into the cepstral domain using the Discrete Cosine Transform (DCT), preserving the first 13 coefficients while discarding the higher ones. As mentioned in Kuamr, Dua & Choudhary (2014), this is because 13 coefficients are sufficient for representing the speech signal.

In this work, MFCCs are used in two different ways. First, 13 coefficients are extracted for each window, resulting in a 13 × 298 matrix. Subsequently, the matrix is reduced to a vector of 13 coordinates, by calculating the average of all the windows. Both the matrix and the vector are used for audio classification.

Models for spectrograms

To extract information from the spectrograms and properly classify the audios, we propose using two models based in convolutional neural networks.

The first model comprises four 2-D convolutional layers with 32, 64, 96, and 96 filters, respectively. After each convolution, batch normalization and max-pooling are performed. Finally, a flatten layer and two dense layers are included, as shown in Fig. 4. We trained two copies of model 1: one with linear spectrograms and the other with logarithmic spectrograms with values between 10² and 10⁴.

For the linear spectrogram version, we used five epochs, a batch size of 200, a binary cross-entropy loss function, and RMSprop optimizer. In contrast, the logarithmic spectrogram version required 10 epochs, a smaller batch size of 20, the same loss function, and the Adam optimizer.

The second model was trained using logarithmic spectrograms with values between 10³ and 10⁴. The network comprises three time-distributed 2-D convolutional layers with 32, 64, and 96 filters respectively. After each convolution, time-distributed max-pooling is performed. Next, a flatten layer, a dense layer and a dropout layer are included. Finally, a dense layer completes the model, as shown in Fig. 5, 10 epochs were required to achieve good convergence with a batch size of 10, we used a binary cross-entropy loss function and the Adam optimizer.

Models for MFCCs

For the MFCCs coefficients, we used two additional models.

Model 3 takes the vector representation of the MFCCs as input and passes the 13 coefficients through three time distributed 1-D dense layers with 24, 13 and 10 units, respectively, along with a dropout layer. Next, we added three LSTM layers with 10, 15 and 30 units, followed by a second dropout layer. Finally, a dense layer with 30 units was included, as can be seen in Fig. 6.

Model 4 receives the two-dimensional MFCCs as input. The network comprises a time distributed 2-D convolutional layer with 36 filters, followed by a batch normalization layer, a second 2-D convolutional layer with 64 filters, and a max pooling layer. We then included two dense layers with 24 units each, followed by a flatten layer and a dense layer, as can be seen in Fig. 7.

For the models using MFCCs, more epochs could be used due to the reduced feature extraction size. For the model that uses a vector, we employed 60 epochs, a batch size of 200, the binary cross-entropy loss function, and the Adam optimizer. As for the version using MFCCs in matrix form, it required 30 epochs for better convergence, a large batch size of 300, the same loss function, and the RMSprop optimizer, it should be noted that for all these models, whether using spectrograms or MFCCs, a learning rate of 0.001 was applied.

For each audio, the linear spectrogram, the two logarithmic spectrograms described above, and the MFCC coefficients are calculated and introduced into their respective models. The classification results from each model are concatenated in a vector, which is used as input for a new neural network. This provides the final classification prediction, as depicted in Fig. 8.

All models were individually trained and subsequently assembled to form the final architecture, as illustrated in Fig. 9. This design offers a significant advantage because it allows for the seamless addition of new models and feature extractions to the architecture. This flexibility arises from the way they are integrated to produce the final classification for each audio.