The application of fractional Mel cepstral coefficient in deceptive speech detection

The introduction of speech signals based polygraph application

Lying is a process which moves from psychological activity to physiological execution. Firstly, people decide to lie from their conscious mind, and then organize the language to cover the real content. Finally, control vocal organs to form the voice. Tommaso, Fabio & Massimo (2013) studied the deception detection results among the people of different personality by the means of pattern recognition. The conclusion is that the outgoing personality groups are easier to be identified. It is also proved that lie detection has individual differences. Lamb & Skillicorn (2013) reported that the frequency of words with different parts of speech appear in the trial process may also be a way to analyze the possibility of lying. In the field of natural language processing, Christie, David & Dursun (2008) showed that the result is influenced by lexical, syntax, sentence length and motivation. Furthermore, the organization of text and voice can be used in an anti-phishing detection system, preventing people from being cheated in instant messengers (Mohammed & Lakshmi, 2012). These researchers discuss the deceptive speech detection results in psychology and natural language processing fields, including personality difference, pronunciation difference, language expression, sentence organization and so on. Then, Sofia et al. (2013) researched the speaker’s differences under normal state and tense state. Cheryl et al. (2013) summarized that the person under tension will show the following case: increase of adrenaline, higher blood pressure, sympathetic excitement, bronchiectasis, and cricothyroid muscle tension. Some people tended to show increased pitch frequency and voice trembling, but not everyone exhibited these traits. These physiological studies have proved the existence of the difference between normal and deceptive state, and provide some simple basis in the physiological field for speech deception detection.

The above conclusion from psychology, linguistics and physiological aspects are relatively consistent, but the research in the acoustic field have different results. Gopalan & Wenndt (2007) claim that the trace of pitch and first formant, which are processed by AM-FM model and Teager operator, presented the definite difference between normal and deceptive speech. However, Christin & David (2013) gave the opposite conclusion. They executed several experiments, and the results are presented on a range of speech parameters including fundamental frequency, overall amplitude and mean vowel formants F1, F2, F3. They could not establish a significant correlation between deceptiveness and truthfulness; the two results appeared opposite. Pitch and formants are susceptible to the influence of speaker’s pronunciation habit, language content, and coarticulation. The parameters will also be changed due to different extraction algorithm, so it is not a good choice for using them as the speech features for lie detection. There is little research to reveal the process from psychological activity to speech production. Muhammad & Kaliappan (2013) use the Bark spectrum as speech features, and use the neural network as the classification model to identify the truthfulness and deceptiveness. Therefore, using robust acoustic characteristics for deceptive speech identification should be more reliable, and there is still plenty of scope for more progress.

The fractional Fourier transform application foundations

Current research has not investigated the feature difference between normal speech and the lie. Physiological studies also could not provide any explanation for whether there are specific changes of articulators or not when people are lying. The existing information is only obtained from speech features statistical research result or classification conclusion by traditional pattern recognition models. The speech features in the frequency domain are mostly achieved by power spectrum transformation. If the liar’s psychological and physiological changes indeed impact the frequency of speech, the deceptive information can be extracted by short-time frequency analysis for voice signals. But it does not work, if only the speech phase is changed. Therefore, a speech feature which can express both the change of frequency and phase is needed. The fractional Fourier analysis is applicable to the task.

That being the case we take the cosine signal as an example to compare the difference between the traditional Fourier transform and fractional Fourier transform. (Please refer to the next section for the detail description of FrFT.) (1)${\displaystyle x\left(t\right)=cos\left({\varpi }_{0}t\right)\iff \left|X\left(\varpi \right)\right|=\pi \left(\delta \left(\varpi -{\varpi }_0\right)+\delta \left(\varpi +{\varpi }_0\right)\right)}$ (2)${\displaystyle x\left(t\right)=cos\left({\varpi }_{0}t+\theta \right)\iff \left|X\left(\varpi \right)\right|=\pi \left(\delta \left(\varpi -{\varpi }_0\right)+\delta \left(\varpi +{\varpi }_0\right)\right).}$

Through (1) and (2), it is shown that the Fourier amplitude–frequency response can’t reflect the phase difference of the two signals. According to trigonometric formula, Eq. (2) can be expanded and transformed by FrFT as the Eq. (3). (3)${\displaystyle x\left(t\right)=cos\left({\varpi }_{0}t+\theta \right)=cos\left({\varpi }_{0}t\right)cos\left(\theta \right)-sin\left({\varpi }_{0}t\right)sin\left(\theta \right)\dots \stackrel{frft}{\iff }{X}_{\alpha }\left(u\right)=cos\left(\theta \right)X_{\alpha }\left[cos\left({\varpi }_{0}t\right)\right]\left(u\right)-sin\left(\theta \right)X_{\alpha }\left[sin\left({\varpi }_{0}t\right)\right]\left(u\right).}$ Then: (4)${\displaystyle X_{\alpha }\left[cos\left({\varpi }_{0}t\right)\right]\left(u\right)=\sqrt{1+jtan\left(\alpha \right)}cos\left(u{\varpi }_{0}sec\left(\alpha \right)\right)exp\left(-\frac{j}{2}\left({\varpi }_0^2+u^2\right)tan\left(\alpha \right)\right)}$ (5)${\displaystyle X_{\alpha }\left[sin\left({\varpi }_{0}t\right)\right]\left(u\right)=\sqrt{1+jtan\left(\alpha \right)}sin\left(u{\varpi }_{0}sec\left(\alpha \right)\right)exp\left(-\frac{j}{2}\left({\varpi }_0^2+u^2\right)tan\left(\alpha \right)\right)}$ (6)${\displaystyle X_{\alpha }\left(u\right)=cos\left(\theta \right)\sqrt{1+jtan\left(\alpha \right)}cos\left(u{\varpi }_{0}sec\left(\alpha \right)\right)exp\left(-\frac{j}{2}\left({\varpi }_0^2+u^2\right)tan\left(\alpha \right)\right)\dots -sin\left(\theta \right)\sqrt{1+jtan\left(\alpha \right)}sin\left(u{\varpi }_{0}sec\left(\alpha \right)\right)exp\left(-\frac{j}{2}\left({\varpi }_0^2+u^2\right)tan\left(\alpha \right)\right)=\sqrt{1+jtan\left(\alpha \right)}exp\left(-\frac{j}{2}\left({\varpi }_0^2+u^2\right)tan\left(\alpha \right)\right)\left(cos\left(\theta \right)cos\left(u{\varpi }_{0}sec\left(\alpha \right)\right)+sin\left(\theta \right)sin\left(u{\varpi }_{0}sec\left(\alpha \right)\right)\right)=\sqrt{1+jtan\left(\alpha \right)}exp\left(-\frac{j}{2}\left({\varpi }_0^2+u^2\right)tan\left(\alpha \right)\right)cos\left(u{\varpi }_{0}sec\left(\alpha \right)+\theta \right).}$ Equation (6) expressed the FrFT result of cos(ϖ₀t + θ). The phase θ still exists in |X_α(u)|, so Eq. (6) can reserve the phase information.

The use of speech signal for lie detection is only in the preliminary stages. If the difference between truthfulness and deceptiveness is really expressed by the amplitude and phase of speech spectrum, the fractional Fourier transform analysis method should be an effective way to reveal the distinction. So as to suit for the diversity of speakers, variety orders of FrFT should be involved. The difference between normal speech and lie can be enhanced due to some orders of FrFT.

LDA algorithm

The main function of linear discriminant analysis (LDA) is to project the high-dimensional samples onto a low dimensional space. It is aimed to maximize the distance between classes, and minimize the distance in the class. Therefore, LDA is suitable for two group classification task, such as truthfulness and deceptiveness division. The $S=\left\{s_1,s_2,\dots ,s_n\right\}$ refers to the training voice set, and s_i is belong to ω₁ or ω₂, which represents for normal speech or lie respectively. A projection operator w defined as the best vector may map x_i to the one-dimensional y. (12)${\displaystyle y=w^T{s}_i.}$ Then, it is very easy to make a decision by a simple comparison. (13)${\displaystyle \left\{\begin{array}{ll}{s}_i\in {\omega }_1\hfill & y\ge t\hfill \\ s_i\in {\omega }_2\hfill & y<t\hfill \end{array}\right.\left(t\text{ is a threshold}\right).}$ So the main task of LDA is to calculate the optimal mapping vector w.

The mean of each category can be expressed as (14)${\displaystyle {\mu }_i=\frac{1}{N_i}\sum _{s\in \underset{i}{\omega }}{s}_i.}$ The mean value is changed after projection. (15)${\displaystyle {\tilde{\mu }}_i=\frac{1}{N_i}\sum _{y\in \underset{i}{\omega }}y=\frac{1}{N_i}\sum _{s\in \underset{i}{\omega }}{w}^{T}s=w^T{\mu }_i.}$ The distance between two means is (16)${\displaystyle D\left(w\right)=|{\tilde{\mu }}_1-{\tilde{\mu }}_2|=\left|w^T\left({\mu }_1-{\mu }_2\right)\right|.}$ And the variance after projection is (17)${\displaystyle {\tilde{d}}_i^2=\sum _{y\in \underset{i}{\omega }}{\left(y-{\tilde{\mu }}_i\right)}^2.}$ Define the objection function J(w), when reaching the max ratio of the distance between these two categories and the variance within the classes, the w is the best vector. (18)${\displaystyle J\left(w\right)=\frac{|{\tilde{\mu }}_1-{\tilde{\mu }}_2{|}^2}{{\tilde{d}}_1^2+{\tilde{d}}_2^2}.}$ The Eq. (17) can be decomposed into (19). (19)${\displaystyle {\tilde{d}}_i^2=\sum _{y\in \underset{i}{\omega }}{\left(y-{\tilde{\mu }}_i\right)}^2=\sum _{s\in \underset{i}{\omega }}{\left(w^{T}s-w^T{\mu }_i\right)}^2=\sum _{s\in \underset{i}{\omega }}{w}^T\left(x-{\mu }_i\right){\left(x-{\mu }_i\right)}^{T}w.}$ Then define (20)${\displaystyle d_i=\sum _{s\in \underset{i}{\omega }}\left(s-{\mu }_i\right){\left(s-{\mu }_i\right)}^T.}$ And we may have (21)${\displaystyle d_w=d_1+d_2.}$ So (22)${\displaystyle {\tilde{d}}_1^2+{\tilde{d}}_2^2=w^T{d}_{w}w.}$ And (23)${\displaystyle {\left({\tilde{\mu }}_1-{\tilde{\mu }}_2\right)}^2={\left(w^T{\mu }_1-w^T{\mu }_2\right)}^2=w^T\left({\mu }_1-{\mu }_2\right){\left({\mu }_1-{\mu }_2\right)}^{T}w=w^T{d}_{k}w.}$ The Eq. (18) can be written as (24)${\displaystyle J\left(w\right)=\frac{w^T{d}_{k}w}{w^T{d}_{w}w}.}$ The optimal vector is (25)${\displaystyle w=d_w^{-1}\left({\mu }_1-{\mu }_2\right).}$ The test voice set can be easily divided into two groups by Eqs. (12) and (13) when the w is obtained.

Hidden Markov model

The hidden Markov model (HMM) can be considered as a generalization of a mixture model. The hidden variables are related through a Markov process, and the observation is controlled by the hidden state. The state is not directly visible in a HMM, but output observation is visible and dependent on the state. Each state has a probability distribution corresponding to the possible output. Therefore, the output sequence generated by an HMM presents some information about the sequence of invisible states.

The random variable x(t) presents the hidden state at time t (with x(t) ∈ {x₁, x₂, x₃}). The random variable y(t) is considered as the speech observation at time t (with y(t) ∈ {y₁, y₂, y₃, y₄}). According to the basic theory of Markov process, it is clear that the conditional probability distribution of the hidden variable x(t) only depends on the value of the x(t − 1). The values at time t − 2 and before have no influence. The value of the speech observation y(t) also depends on the value of the x(t).

Two types of parameters, called transition probabilities and output probabilities, are contained in a HMM. The hidden state at time t is determined by hidden state at time t − 1 according to the transition probabilities. There is also a set of output probabilities to describe the distribution of the observed variable.

There are some important parameters in a HMM.

1. N, the number of states in the model.

2. M, the number of observation symbols per state.

3. The transition probability distribution. (26)${\displaystyle a_{ij}=P\left(x_{t+1}=s_j|x_t=s_i\right).}$

4. The output observation probability distribution. (27)${\displaystyle b_j\left(k\right)=P\left(y_t=o_k|x_t=s_j\right).}$

5. The initial state distribution. (28)${\displaystyle {\pi }_i=P\left(x_t=s_i\right).}$

The famous forward-backward algorithm, EM algorithm, and Viterbi algorithm can be used to train the models and solve the recognition task (Rabiner & Schafer, 2007; Matthias & John, 2009). Although the HMM is a traditional method used in recognition systems, it is still a suitable model in deceptive speech detection with time series speech signals.

Speech database

The liar’s appearance has a direct relationship with individual personality, culture background, conversation content and the cost of being seen through the lie. Therefore, the speech sources should be collected in a real circumstance. According to the concealed information test theory (Verschuere, Ben-shakhar & Meijer, 2011), we designed an interesting game, and the speech database is selected from the game records. There are two groups in the game, which are called A-group and B-group. Every person in A-group should tell a story, and the persons in B-group can ask all kinds of questions according to the story. Due to the different stories told by peoples in A-group, the questions and answers are different from each other. Since the persons in B-group do not know whether the story is experienced the storyteller himself/herself, they should decide the true or false through the teller’s answers. If the persons in B-group speculate the correct result, they win the game and obtain some reward. Otherwise, the person in A-group wins. So if the story is a lie, the teller should try his/her best to keep the secret from every one to win the game. People in B-group should ask as many questions as possible (generally more than 10 questions) to make the liars nervous and make mistakes. Here, we select the every fake stories and fake answers as the deceptive speech samples. Then the corresponding tellers should record a set of normal speech in a calm environment, the topics can include such as self-introduction, hobbies, and daily life topics and so on. These records should long enough to cover as much as possible syllables in Mandarin Chinese. So the normal speech samples are collected.

At last, we reserved useful records of 50 participants, including the 25 men and 25 women. Due to some limitations, the participants are mainly 25 to 35 years old. The SNR of all the samples are more than 25dB. All speech is mono sampled at 8 kHz and quantified with 16 bits. The frame length is 20 ms, and the overlap is 10 ms when the speech is under short-time analysis. The data set is divided into two parts, namely the training set and test set. The experiment is under a unified standard to divide the data set due to the different length of the every people’s speech sample. The training set contains 30% of whole record, and the remaining data are regarded as the test set.

Human Ethics

This research was approved by the Institutional Review Boards of Soochow University School of Electronics and Information Engineering, and Suzhou University of Science and Technology School of Electronics and Information Engineering. The speech set recording is carried out in a game style, so all the participants are confirmed with verbal consent.

The experiment results for LDA model

The experiment step is expressed as follows:

1. Divide the speech signals into short frames with the length of 20 ms and overlap of 10 ms.

2. Extracted the FrCC parameters from every frame, and 12 FrCC coefficients and 12 delta FrCC coefficients are used as the FrCC vector from one speech frame. The range of angle is α ∈ (0, π), with the 0.01π as the step. So there are 100 FrCC vector groups in every frame.

3. Select 30% of the total data as the train set, and use LDA algorithm to calculate the optimal vector w.

4. Take the remaining 70% of data as test set. Map the test set into low-dimensional space by w.

Then use Eqs. (12) and (13) to make the decision and the statistical accuracy can be obtained at last.

In this experiment, the recognition results of MFCC parameters are taken as a benchmark to compare with that of FrCC parameters. The results are shown in Tables 1, 2 and Figs. 3, 4. The first line of these tables denotes the people’s ID. The second line indicates the corresponding α of highest accuracy for FrCC. The accuracy of FrCC and MFCC are shown in line 3 and 4, respectively.

In order to further refine the improvement of FrCC, the vector variance is introduced to compare the clustering performance of the two parameters. The vector variance is shown in Eqs. (29) and (30). (29)${\displaystyle R_1=\frac{\sum _{i=1}^N{\left(normfrc{c}_i-\overline{normfrcc}\right)}^2}{\sum _{i=1}^N{\left(normmfc{c}_i-\overline{normmfcc}\right)}^2}}$ (30)${\displaystyle R_2=\frac{\sum _{j=1}^M{\left(decfrc{c}_j-\overline{decfrcc}\right)}^2}{\sum _{j=1}^M{\left(decmfc{c}_j-\overline{decmfcc}\right)}^2}.}$ Here, the normfrcc_i presents the FrCC of normal speech, and the $\overline{normfrcc}$ presents the mean vector. The normmfcc_i and $\overline{normmfcc}$ present the MFCC of normal speech and MFCC mean vector, respectively. The R₁ in Eq. (29) denotes the vector variance ratio between FrCC and MFCC of normal speech. The R₂ in Eq. (30) denotes the vector variance ratio between FrCC and MFCC of deceptive speech. The results are presented in Tables 3 and 4.

The experiment results of HMM model

There are many sophisticated tools for HMM training and testing, such as HTK or Matlab software package. The speech signals should also be divided into short frames. Then the speech characters such as FrCC and MFCC can be regarded as the observations, the psychophysiology status is regarded as the invisible states. The 30% of the total data is regarded as the train set, and the remaining data is the test set. The speech characters changed frame to frame, and the hidden Markov chain can present the process of the psychophysiology changes. The experiment results are shown in Tables 5 and 6. The comparison between the men set and women set are in Figs. 5 and 6.

Results analysis and discussion

In the sections ‘The experiment results for LDA model’ and ‘The experiment results of HMM model,’ the experiment results show that the identification accuracy of FrCC parameters under certain angles is higher than that of MFCC parameters. The FrCC coefficients introduced to the LDA model make the clustering performance much better. The accuracy will be increased when HMM model is used to enhance the contextual information. The following paragraphs give some brief explanation to the experiment results.

(A) In the LDA recognition system, the men groups’ average accuracy of FrCC with best angle is 59.9%, and MFCC is 56.3%. The average of best angle is $\overline{\alpha }=0.51\pi$, and the variance of α is D(α) = 0.23π. The women groups’ average accuracy of FrCC with best angle is 56.2%, and MFCC is 50.3%. The average of best angle is $\overline{\alpha }=0.59\pi$, and the variance of α is D(α) = 0.22π. The best angle of 10, 22 in men’s group and 8 in women’s group is $\frac{\pi }{2}$. In these cases, the FrCC coefficients are equal to MFCC coefficients. In the other cases, the identification performance of FrCC under LDA model is better than that of MFCC. The accuracy increased from 36.7% to 56.0% when FrCC is introduced in the 16th men. And the accuracy of many people is increased over 10%. Due to individual differences, the accuracy is only a little increased with some people. Overall, when FrCC coefficients are involved, the average accuracy is increased by 3.6% in men group and 5.9% in women group, respectively. Although the average growth of accuracy is not very large, there was great progress with some individuals. The FrCC parameters can therefore improve the deceptive detection performance.

(B) The performance of FrCC parameters may be changed with different α of FrFT. Due to the diversity and non-stationary characteristics of speech, and personality difference of the speakers, it is impossible to determine the optimal α before the experiment. The best α is selected by the highest accuracy. So the mechanism of selection algorithm should be further studied.

(C) Most of the R₁ and R₂ is less than 1 in Tables 3 and 4. It is shown that the clustering performance of certain FrCC is much better than that of MFCC. The FrCC data is concentrated to the clustering center. The existence of α enhanced the appearance of the change of articulator, when people are lying or under stress. These faint details could not be reflected by MFCC. These conclusions may be explained by phase change statement described in section ‘The Fractional Fourier Transform application foundations,’ and should be further verified by physiology research.

(D) The experiment results show that the performance of FrCC is better than that of MFCC. That is to say the new character may be more suitable in some speech based psychophysiology information processing field. There are 25 men and 25 women participants, and the range of angle is α ∈ (0, π), with the 0.01π as the step. There are 2,500 recognition accuracy results in each men set and women set. The distribution of FrCC (with all angle α) based man and woman deceptive speech recognition accuracy is presented in the Figs. 7 and 8, respectively. The black solid line is accuracy distribution, and the red dotted line is the average accuracy of MFCC. According to the statistical result, the men groups’ average accuracy of MFCC is 56.3%, so approximately 20.7% (518/2,500) of the FrCC based recognition accuracy is higher than 56.3%. The women groups’ average accuracy of MFCC is 50.3%, so approximately 60.8% (1,520/2,500) of the FrCC based recognition accuracy is higher than 50.3%.

(E) Each speech frame is regarded as the basic unit for deceptive speech detection under LDA model. It reveals the advantages of FrCC coefficients for this classification task. The vector w is a global optimal vector, but the speech flow is a time-varying process. If the context information can be introduced in the identification system, the speech signal can be mapped onto the state flows. Then, if statistical model are used for classification, there may be a better result.

(F) The speech is a time series signal, and the contextual information may be hidden among the speech frames. The HMM model can mine this information and reveal the relations among the adjacent speech frames. In the HMM recognition system, the men groups’ average accuracy of FrCC with best angle is 71.0%, and MFCC is 66.4%. The women groups’ average accuracy of FrCC with best angle is 70.2%, and MFCC is 65.0%. When the HMM model is involved, the average accuracy is increased by 11.1% and 14.0%, respectively, in two groups. The highest accuracy of FrCC is 82.0% in the men set, and 85.4% in the women set. The largest individual accuracy increase is 31.2% in the women set (from 39.9% to 71.1%, 13th woman) and 18.5% in the men set (from 45.5% to 64.0%, 12th man). The FrCC based deceptive detection accuracy comparison between LDA and HMM are shown in Figs. 9 and 10.

(G) The ROC curve (Fawcett, 2006) is usually used to analysis the performance of the identification system. Here, the deceptive detection is a binary classification problem, in which the outcomes are labeled either as positive (p) or negative (n). There are only four outcomes from a binary classifier, the true positive (TP), false positive (FP), true negative (TN) and false negative (FN). Therefore, we select two parameters, the true positive rate (TPR, sensitivity) and true negative rate (TNR, specificity) to evaluate the performance of the LDA and HMM model. The sensitivity defines how many correct positive results occur among all positive samples during the experiments. Specificity defines how many correct negative results occur among all negative samples during the experiment. The definition equations are shown in (31) and (32). The statistical results are presented from Figs. 11–14. The difference between the sensitivity and specificity of every participant is not large, so LDA and HMM are the suitable tool for dividing the normal or deceptive speech. (31)${\displaystyle sensitivity=\frac{TP}{TP+FN}}$ (32)${\displaystyle specificity=\frac{TN}{FP+TN}.}$

In summary, the experimental results at least reflected that the acoustic feature is effective for lie detection. Faint difference between truthfulness and deceptiveness can be expanded under some improved acoustic features such as FrCC, and these characteristics may play an important role in deceptive speech identification.