Addressing human speech characteristics in single-channel speaker extraction networks

The speech extraction problem

We formally describe the speech extraction problem as follows. Let the speech signal of the target speaker be denoted as $x(t)$. The mixed speech can then be represented as follows:

(1) $$y(t)=x(t)+\sum _{i=1}^n{n}_i(t).$$Here, $x(t)\in {\mathbb{R}}^{1\times T}$, and $n_i(t)\in {\mathbb{R}}^{1\times T}$ denote the speech signal of the target speaker and the interference sources, respectively, $t$ refers to the time frame, and $i$ is the number of interference sources. The objective of speech extraction is to extract $x(t)$ from $y(t)$.

The architecture of the scheme

Drawing inspiration from the design principles of speech extraction network frameworks proposed in previous research (Luo & Mesgarani, 2019), we have devised a novel deep learning-based speech extraction network, as vividly illustrated in Fig. 1. In this figure, Fig. 1A offers a comprehensive block diagram depicting the general process of speaker extraction, while Fig. 1B presents a detailed system flowchart of the proposed network architecture. The proposed network model is composed of five key components: the Mixture Encoder, Reference Speech Encoder, Speech Extractor, Speaker Encoder, and Speech Decoder. The Mixture Encoder and Reference Speech Encoder undertake the crucial task of transforming the time-domain waveform signals of mixed speech and reference speech, respectively, into high-dimensional time-domain feature representations. These representations capture the essential characteristics of the input speech data.

The Speaker Encoder, by contrast, represents an evolved design from our earlier previous work, engineered to more effectively differentiate the unique acoustic attributes of the target speaker from those of other speakers. Whereas the previous work exhibited limitations in capturing fine-grained acoustic variations, this updated encoder generates robust identity embedding information through enhanced discriminative learning. This information serves as a distinctive identifier for the target speaker within the network and is disseminated throughout the model, providing a solid foundation for subsequent processing steps. The integration of identity embedding information into the network architecture builds directly on our previous framework, addressing its prior constraint of suboptimal embedding utilization. Its primary role is to endow the model with the ability to selectively extract target speech from complex mixtures, a capability that was partially realized in the previous work but now refined through optimized information propagation. By leveraging this embedded identity information—now more deeply integrated across model layers—the model achieves higher precision in identifying and isolating the target voice, marking a significant advancement in speech extraction performance.

The Speaker Encoder, by contrast, is engineered to differentiate the unique acoustic attributes of the target speaker from those of other speakers. Through this discrimination process, it generates robust identity embedding information, which serves as a distinctive identifier for the target speaker within the network. This identity embedding information is then disseminated throughout the model, providing a solid foundation for subsequent processing steps. The integration of identity embedding information into the network architecture is of paramount importance. Its primary role is to endow the model with the ability to selectively extract the speech of the target speaker from complex auditory mixtures. By leveraging this embedded identity information, the model can more accurately identify and isolate the target speaker’s voice, thereby enhancing the overall performance of the speech extraction process.

The Speech Decoder is architected to reconstruct the time-domain waveform signals at varying resolutions. Its operational mechanism involves leveraging the time-domain masking information generated by the core Speech Extractor component, enabling the recognition and subsequent reconstruction of the target speech waveform. This process is crucial for restoring the original speech signal quality after the extraction process. Ensuring the fidelity of the extracted speech necessitates that the Speech Extractor is equipped with the capacity to analyze the intricate speech features and their inherent interdependencies, while simultaneously adapting to the dynamic and complex acoustic environments. To address these requirements, we employ a non-causal convolutional model with adjustable dilation rates to build a long-sequence processing architecture. This design allows the model to effectively capture the long-range contextual information within speech signals, thereby facilitating a more comprehensive and nuanced analysis of the speech content. The detailed design and implementation of the Speech Extractor are further elaborated in ‘Speech Extractor’.

Mixture encoder

The speech encoder utilizes one-dimensional convolution to perform multi-scale encoding (with convolution kernels of varying lengths) to juice the time domain features of human speech. This operation bypasses the challenge of phase estimation in short-time Fourier transforms, ultimately improving the performance of speech extraction.

The length of the convolution kernel is determined as $L_1(short)$, $L_2(middle)$ and $L_3(long)$ respectively, the step size is set to $(L_1/2)$. The output obtained through the speech encoder is a N-dimensions vector $W_i$ with length K:

(2) $$W_i=ReLU(F_i(y)),i=1,2,3$$where $W_i\in {\mathbb{R}}^{K\times N}$, and K can be calculated by $K=2(T-L_1)/L_1+1$. ReLU is the activation function, $F_i(\cdot )$ is the one-dimensional convolution layer, and $y$ is the original mixed speech. The complete output obtained by multi-scale encoder is a concatenation of the three vectors $w_1,w_2,$ and $w_3$.

Similaly, in the proposed network model, we share the same convolutional layer and weights of mixed speech coding with the reference speech coding, the adoption of the twin operation can effectively improves the performance of speech separation that was validated in the SpEx+ model (Ge et al., 2020). The output of the reference speech coding can be obtained as $V_i=ReLU(F_i(s)),i=1,2,3$.

Reference speech encoder

The reference speech encoder design is depicted in Fig. 2. The reference speech signal is first passed through one-dimensional convolutional encoders (Conv1Ds) to obtain time domain features. Simultaneously, the reference speech signal undergoes short time Fourier transform (STFT) operation to obtain a two-dimensional time-frequency map. The time-frequency map is then convolved using a 2-D convolution (Conv2D) to obtain two-dimensional time-frequency features. The time domain features obtained by Conv1Ds are concatenated along the channel, while the time-frequency domain features from the Conv2D are concatenated along the frequency direction. These features are then combined to create new features.

However, the one-dimensional convolution has three types of convolutional kernel lengths (20, 80, and 160) with a step size of 10. These kernel lengths correspond to time window lengths of 2.5, 10, and 20 ms respectively when the sampling rate is 8,000.

As shown in Fig. 2, $Conv1D_{S}hort$ denotes the encoder with a kernel length of 20, while the other two encoders have kernel sizes of 80 and 160. The process for obtaining temporal features is calculated as follows:

(3) $$\{\begin{array}{c}Fconv\mathrm{\_}1=Conv1D\mathrm{\_}s(x)\hfill \\ Fconv\mathrm{\_}2=Conv1D\mathrm{\_}m(x)\hfill \\ Fconv\mathrm{\_}3=Conv1D\mathrm{\_}l(x)\hfill \end{array}$$where $Conv1D\mathrm{\_}s(x)$, $Conv1D\mathrm{\_}m(x)$ and $Conv1D\mathrm{\_}l(x)$ are convolution operations with kernels of varying lengths.

To obtain time-frequency features using STFT, we set the window length to 20 and the frame shift to 10 to match the kernel length. We apply STFT to the input speech signal $x$, and the calculation process is as follows:

(4) $$X(M)=\sum _0^{N-1}x(n)[\cos \left(\frac{2\pi nm}{N}\right)-j\sin \left(\frac{2\pi nm}{N}\right)].$$

We separate the real and imaginary parts of it to obtain:

(5) $$X(M)=X_{real}+j{X}_{imag}.$$

Then, we obtain the magnitude spectrum $X_{mag}$ as:

(6) $$X_{mag}=\sqrt{{X_{real}}^2+{X_{imag}}^2}.$$

We input $X_{mag}$ into a two-dimensional convolution layer to obtain time-frequency features:

(7) $$F_{freq}=Conv2D(X_{mag}).$$

After completing these steps, we obtain both temporal and spectral features, concatenate them, and input them into the speaker encoder to learn the embedded vector.

Speaker encoder

After encoding the reference speech with time domain features, it must meet with further training by the network to obtain a vector that can be used for extracting. The speaker encoder, as depicted in blue in Fig. 1B, aims to produce an identifiable embedding vector for the speech extraction network. Unlike traditional voiceprint extraction networks that use i-vector (Dehak et al., 2010) and x-vector (Snyder et al., 2016) to classify speakers, our speaker encoder concentrates on discovering similar embedding vectors in the time domain system to provide references for speech extraction networks.

The speech encoder encodes the reference speech and generates a high-dimensional identity feature vector V. To further enhance the features obtained by different scale encoders, layer normalization (LN) and a $1\times 1$ convolution are enforced. The resulting high-dimensional features are then subject to channel integration and cross-channel fusion, ultimately resulting in a dimension change from $3N$ to $e$.

In order to obtain a desirable embedding vector, we have developed the Spk block, which is illustrated in Fig. 3A, we explain the structure of it as follows.

The Spk block: The Spk block utilizes a depth-wise separable convolution (dConv) to separate the signal into $e$ channels, followed by s batch normalization (BN). Subsequently, a $1\times 1$ convolution layer is applied to transform the channel number from $e$ to $f$, with a PReLU non-linear layer, then another $1\times 1$ convolution layer is adopted, the number of channels still maintained as $f$. Next, the channel attention module SE block (Hu, Shen & Sun, 2018) was utilized to employ the attention mechanism to each channel, thereby increasing the weight of channels that have positive effects. To prevent the loss of shallow features and the vanishing gradient problem in the training process, the input and output of the SE block were added by skip-connection in the figure. Subsequently, a threshing operation was performed by the maximum pooling layer, which boosted the generalization performance of the trained embedding vector and reduced its length.

In the Speaker encoder, Spk block has been repeatly utilized $N_s$ times. Then a convolutional layer was employed to set the output dimension to N′, and the global average layer was exploited to obtain the identity embedding vector $Emb$ with length 1 and dimension N′.

The target speaker identity embedding obtained from the aforementioned analysis serves two purposes. Firstly, it is used to guide the speech extraction network to estimate the correct target masking. Secondly, the identity is embedded through the linear layer and softmax for speaker label prediction. Manipulating co-training of SI-SDR loss and cross-entropy loss, can result in a better match between the identity embedding and the speaker identity, thereby enabling the model to more accurately name the target speaker’s voice and achieve preferable extraction performance.

Speech extractor

In this article, we propose a novel time-domain speech extraction network for obtaining the target speaker’s mask. Recently, the time-domain model based on TCN block (Bai, Kolter & Koltun, 2018) has achieved remarkable success in speech separation tasks. However, the hollow convolution’s receptive field is severely limited due to the convolution kernel size constraint. The ConvNeXt block model (Liu et al., 2022), which uses convolution kernels with larger adaptation, can theoretically achieve a larger receptive field. However, the original application scenarios of ConvNeXt is image processing, it still unable to meet the requirement for a large receptive field of temporal sequence data in speech signal processing. Therefore, we have elevated the ConvNeXt block model to TD-ConvNeXt model to improve its suitability for processing time series data.

The TD-ConvNeXt block: The ConvNeXt (Liu et al., 2022), which is derived from RseNet50 (He et al., 2016) and builds on the Swin Transformer (Liu et al., 2021), has demonstrated superior performance compared to Swin Transformer. To process one-dimensional time series data, we have revamped the ConvNeXt blocks, which is referred to as Time-Domain ConvNeXt block (TD-ConvNeXt block) and is illustrated in Fig. 3C. The original ConvNeXt nonlinear layer and the normalized layer are GELU and LN, respectively. We have modified them to PReLU and gLN.

Compared to the original ConvNext structure, we put the dConv convolution in the first layer, the TD-ConvNext structure has fewer nonlinear and normalized layers. The convolution kernel of dConv has a size of $1\times P$ and an expansion factor, we put gLN and PReLU between the two convolutions. This approach reduces the number of parameters that need to be saved during training, making it easier to construct a deeper model. TD-ConvNeXt blocks are also a stacked structure in speech extractor, as depicted in Fig. 4B. They are stacked B times with the expansion factor of dConv gradually increasing from $2^0$ to $2^{B-1}$ to enhance the receptive field of modules.

Considering the excellent performance of TCN block in processing time series, we have incorporated it as one of the components in our proposed model. The block comprises two $1\times 1$ convolution layers and a dConv layer, as illustrated in Fig. 3B. The first layer is a $1\times 1$ convolutional layer, which changes the channel from $a$ to $b$. Nonlinear layer PReLU and global layer normalization (gLN) (Luo & Mesgarani, 2019) are then added to accelerate network convergence. The middle convolution layer is a deeply separable convolution with a convolution kernel size of $1\times 3$ and an expansion factor. Since speech data is a long sequence of one-dimensional data, increasing the receptive field can ensure the integrity of the output speech to some extent. A PReLU+gLN layer is added to nonlinearize and normalize the intermediate output features. The last $1\times 1$ convolution layer communicates and fuses it across channels. To prevent the vanishing gradient issue during training, each module in the entire block is equipped with a skip-connection. As the TCN blocks are stacked B times, as shown in Fig. 4A, we refer to it as Stacked TCN blocks, where the expansion factor of each block increases from $2^0$ to $2^{B-1}$.

Once the output vector W of the mixed speech encoder and the embedded vector $Emb$ of the speaker encoder are obtained, normalization is performed followed by dimension reduction using a convolution layer. This reduces the dimension of the feature vector from $3N$ to $a$. The feature vector includes both the target speaker and jammer, which necessitates integration of the speaker-encoded embedded vector to guide the subsequent speech extraction process. As shown in purple in Fig. 1B, the speech extractor comprises $N_c$ stacked TD-ConvNeXt blocks and $N_t$ stacked TCN blocks. The speaker feature embedding vector $Emb$ is fused with each large module to provide continuous stimulus to the speech extractor, thereby guiding the speech extraction process.

Furthermore, the $Emb$ size is ${\mathbb{R}}^{1\times N^{\prime }}$, and it is copied to the same length as W′. Consequently, the fusion result size is $Concat(W^{\prime },Emb)\in {\mathbb{R}}^{K\times (c+N^{\prime })}$, which is then input into each block. Between stacked TD-ConvNeXt blocks and stacked TCN blocks, $1\times 1$ convolution is used for dimension conversion. Therefore, the feature size of the first block of the stacked input TCN blocks in the model is ${\mathbb{R}}^{k\times (a+N^{\prime })}$. The time domain feature code obtained after connecting the two basic modules, is processed through a combination of $1\times 1$ convolution and ReLU to obtain a mask similar to IBM and IRM (Wang, Narayanan & Wang, 2014). Because we employ the multi-scale scheme, the mask can be denoted as $Mas{k}_i,i=1,2,3$, where $i=1,2,3$ after training. Consequently, the final speech encoded features can be expressed as follows.

(8) $$\widehat{S_i}=Mas{k}_i\otimes W_i,i=1,2,3.$$Here, $\otimes$ represents the dot multiplication between the corresponding elements of the matrix, and $\widehat{S_i}$ denotes the estimated target speaker speech coding at a certain scale.

Speech decoder

The speech extraction network mixes the embedding vector of the auxiliary network with the time domain features of the mixed speech obtained by the encoder, followed with convolution operations to obtain the speech coding of the target speaker. Finally, the speech coding is reconstructed into a time series through the decoder, resulting in the reconstructed target speech data. This process can be expressed as:

(9) $$\widehat{x_i}=D_i(\widehat{S_i}),i=1,2,3.$$Here, $\widehat{x_i}$ represents the reconstructed target speaker speech, where $i$ denotes the estimation at different scales. $D_i(\cdot )$ refers to the deconvolution, which is the decoder with convolution kernels corresponding to the encoder with lengths $L_1$, $L_2$, and $L_3$, respectively. The decoder generates estimated speech at multiple resolutions, with the estimated speech exhibiting the minimum discrimination rate serving as the output of the model.

Multi-mission learning

Target speaker speech extraction can be viewed as a multi-task process. Firstly, the speech features of the target in the mixed speech are extracted and reconstructed into clean speech containing only the target speaker. Second, the reference speech is trained as the feature representation of the target speaker, which corresponds to the correct label classification, thereby providing a positive incentive for the speech extraction of the target speaker. Therefore, the overall target loss function can be divided into two parts. For the speech extraction of the target speaker, the scale invariant signal-to-noise ratio SI-SDR is used as the loss function, while the cross-entropy loss function is employed for the speaker label. The total loss function can be represented as follows:

(10) $$\mathcal{L}={\mathcal{L}}_{SI\text{-}SDR}+\gamma {\mathcal{L}}_{CE}$$where $\gamma$ denotes the weight factor occupied by the cross-entropy loss function. The loss functions for the two subtasks are presented separately below.

SI-SDR is a commonly used speech evaluation metric that has better robustness than SDR (Roux et al., 2019). To encode and decode operations in multi-scale scenarios, we use convolution kernels of different sizes. This enables the speech to have output in different resolution windows. Therefore, the entire SI-SDR loss during the training process can be expressed as follows:

(11) $$\begin{array}{rl}{\mathcal{L}}_{SI\text{-}SDR}=& -[(1-\alpha -\beta )\ast \rho (\widehat{x_1},x)\\ & +\alpha \ast \rho (\widehat{x_2},x)+\beta \ast (\rho \widehat{x_3},x)].\end{array}$$

In Eq. (11), $\widehat{x_i},i=1,2,3$ represents the estimation of the mixed speech $y(t)$ obtained under different resolution windows after extraction. $\rho (\widehat{x_i},x)$ denotes the SI-SDR loss.

The cross-entropy loss function (CE loss) is commonly employed for classification tasks, and can be represented as follows.

(12) $${\mathcal{L}}_{CE}=-\sum _1^{T_s}{I}_i\log (q(I_i))$$where $T_s$ is the number of speakers, $I_i$ is the true vector of speaker labels, and $q(I_i)$ is the estimated predicted probability.

The data set

Our experimental dataset is the 100-h libriSpeech dataset (Panayotov et al., 2015), which comprises 251 speakers and approximately 28,000 clean sentences as the training and validation sets. The test set consists of 40 speakers and 2,620 clean sentences, with all original sentences sampled at a 16 k sampling rate. Each statement has a duration of approximately 10 s.

To simulate real-world environments, we used the WHAM! (Wichern et al., 2019) noisy dataset as ambient background noise. We randomly selected a clean sentence without background noise from two different speakers. We chose the first speaker as the target source and selected another sentence from the target speaker’s speech as the reference speech. The other speakers were regarded as the interference sources.

The mixed speech was generated using the direct stacking method. Firstly, environmental noise was mixed with the speech of another speaker. Then, the mixed speech was randomly combined with the speech of the target speaker at signal strengths of −5, 0, and 5db. A total of 20,000 statements were generated as the training set, with 5,000 selected for the validation set. In the test set, we used three signal-to-noise ratios (SNRs), and each test statement was a hybrid generation of LibriSpeech and WHAM! The reference sentence was randomly selected from all the speech of the target speaker, which was not repeated by the target speaker in the mixed speech.

Experiment environment

The hardware used for learning and training in this article was an NVIDIA Tesla V100 graphics card with 32G video memory. For the software environment, we used Pytorch version 1.10, CUDA version 10.2, and Python version 3.7.6. The main parameters used in model training are presented in Tables 1–3. The optimizer was Adam (Kingma & Ba, 2014), with an initial learning rate of $1e^{-3}$ and a batch size of 10. The initial $L_1$ was set to 20, representing a time window length of 2.5 ms, as the sampling rate of the initial speech in the dataset was 16 k, which was resampled to 8 k in the experiment. Similarly, the initial time window length of $L_2$ and $L_3$ were 10 and 20 ms, respectively. B represents the number of times small blocks are stacked into large blocks in the model. For instance, TCN block was stacked eight times into TCN blocks, and the expansion factor of dConv convolution kernel was increased from $2^0$ to $2^7$. The number of Spk blocks was set to four, with a speech sample length of four seconds. $\alpha$ and $\beta$ denote the weight of SI-SDR loss under medium window length and long window length in the loss function, which were pre-set to 0.1. $\gamma$ represents the weight factor of the cross-entropy loss function, which was set to 10.

The influence of different configuration of speech extractor on model performance

The speech extractors comprise different stacking patterns of TD-ConvNeXt blocks and TCN blocks, and the number of modules processed has a significant impact on speech extraction performance. This section mainly investigates the influence of different components of the speech extractor on the model’s performance through experiments, aiming to obtain the best configuration of the speech extractor.

Table 2 presents the SI-SDR results in the environment of two-person speech mixing and noise under different model structures and reference models. We created the speech extraction model using a stack of different combinations of TD-ConvNeXt blocks and TCN blocks, while keeping the speaker encoder section unchanged. It can be observed from Table 2 that configuration 7 has an average SI-SDR evaluation index that is 0.38 dB higher than that of the benchmark model. In model configurations 9–12, when the number of TD-ConvNeXt block stacks increased from 2 to 4, the system performance leveled off and slightly decreased compared to the 2-layer stacks. Therefore, the structure of the speech extractor is based on configuration 7, and the parameters c and d are set as 160 and 320, respectively, which correspond to the number of convolutional channels in Fig. 3B.

Based on the results presented in Table 2, it can be observed that the best performance is achieved when the size of the dConv convolution kernel is $1\times 7$, larger convolution kernels slightly reduce the model’s performance. Compared to model 7, the SI-SDR index of Model 14 is approximately 0.1 dB lower, at the cost of greater computational complexity.

In addition, we made some changes to the configuration parameters $a$ and $b$ for TCN blocks. However, as seen in model structures 15–27 in the table, for the optimal model structure 7, the number of parameters is reduced, and the corresponding SI-SDR also decreases. Therefore, in this experiment, we keep $a=256$ and $b=512$ as the optimal configuration parameters.

The influence of different configurations of the speaker encoder

Previous works have mainly focused on the design of the speech extractor, while neglecting the role of the speaker encoder (embedding vector). In this section, based on the optimal speech extractor obtained in the previous experiment, we modify the structure of the speaker encoder to investigate the impact of different configurations on the overall model, aiming to obtain the configuration structure of the speaker encoder with the best performance.

Table 3 presents the results of our experiments. In Table 2, we selected the best speech extractor structure obtained in the previous section (Structure No. 7). The table shows that the models with Spk block outperform the benchmark model, SpEx+. Moreover, by fine-tuning parameters $e$, $f$ and the size of the dConv convolution kernel in the Spk block, the average SI-SDR of model 21 is 0.14 dB higher than that of model 7, indicate that with the same conditions, the system can achieve better performance and better anti-interference, which means that the extracted speech is closer to the real speech.

Effect of encoder window length on model performance

Multi-scale encoders consist of multiple convolution kernel lengths. The encoded and decoded speech at different lengths contain information of different scales and resolutions, and the detailed features of speech at high resolution significantly impact the human auditory experience. Therefore, in addition to the SI-SDR evaluation indicator, we added PESQ (the speech perception evaluation) and STOI (the short-term intelligibility) to our evaluation metrics, we tested different combinations of window lengths to verify the model’s performance.

Table 4 presents the experimental results of different encoder window lengths. As shown in the table, compared to model 21, model 25 also uses a codec with a window length of 40. The SI-SDR slightly improved, the PESQ values are similar, and the STOI have significantly improved. Models 25–27 indicate that a higher STOI can be achieved with more short-window codecs.

The impact of the weight setting of the loss function on the performance of the model

The multi-scale speech extraction model used in this article encounters a weight assignment problem during the loss calculation process. In the previous experiment, we pre-set $\alpha =0.1$ and $\beta =0.1$, resulting in a weight of 0.8 for the small scale with a window length of 20, and 0.1 for both the medium and long scales. In this section, we fine-tune the weights $\alpha$ and $\beta$ of the loss function to investigate the impact of weight settings on the model’s performance.

Table 5 presents the experimental results after selecting the best performing model 25 from the previous section. Models 28 to 30 set different weights for Model 25. From the results, it can be observed that the SI-SDR and STOI of Model 25 are higher than those of Models 28 to 30. Therefore, we continue to keep $\alpha$ and $\beta$ as 0.1 for the loss function weight setting.

Comparison of the proposed network with the state of the art

Through the experiments described in the previous sections, we obtained the current best-performing network model 25, which we named TDNext. In this section, we reproduced five baseline models, SpEx+ (Ge et al., 2020), SpEx (Xu et al., 2020), TD-SpeakerBeam (Delcroix et al., 2020), SpeakerBeam (Delcroix et al., 2018) and sDPCCN (Han et al., 2022), and compared TDNext with these five baselines using the three evaluation indicators of SI-SDR, PESQ, and STOI to verify the effectiveness of our proposed algorithm.

From Table 6, it can be seen that TDNext outperforms all the baselines in SI-SER and STOI metrics at the same SNR levels. The above experimental results demonstrate the effectiveness of TDNext in the target speech extraction task.

To visually observe the effectiveness of different models in extracting speech, we present typical spectrograms of the original mixed speech, clean speech, and the speech extracted by our proposed model and the comparative models in Fig. 5. From the figures, it can be seen that there is a high-frequency blank area without speech energy distribution in the beginning area of clean speech. In contrast, the interference speech energy distribution can be observed in this area in the spectrogram of the mixed speech. The performance of SpEx+ model and our proposed model is better, with clear blank areas in the silent region, achieving a good noise reduction effect. Correspondingly, there is a clear boundary between sentences in the blank area in the middle of clean speech, only our model shows a clear separation with a distinct boundary between each sentence, which may result in clearer auditory perception and better speech understanding, as demonstrated in the spectrogram.

Ablation experiment

To further verify the effectiveness of the TD-ConvNeXt block proposed in this article, an ablation experiment was conducted in this section with and without removing the TD-ConvNeXt block and its expansion factor. The results are presented in Table 7.

In Table 7, it can be observed that after removing the expansion factor from the TD-ConvNeXt block of the whole network and returning to an ordinary convolutional neural network (W/O D.Conv), all three indicators decreased. Moreover, after removing the TD-ConvNeXt block (W/O TD) from the whole network, the three indicators decreased further. The increased decline rate confirms the performance improvement effect of the TD-ConvNeXt block proposed in this article for the whole network.