Musical timbre style transfer with diffusion model

Perceptual compression

This work builds on the work of Rombach et al. (2022) by integrating the idea of perceptual compression to improve the efficacy of training the diffusion model for generating high-quality CQT spectrogram. This feature enhances the computational efficiency of the diffusion model by conducting sampling in a low-dimensional space.

This article employs a convolutional VAE to formally encode the CQT spectrogram $X\in R^{T\times F}$ into a potential space Z. Here, $Z\in R^{C\times \frac{T}{r}\times \frac{F}{r}}$, T, and F represent the time and frequency dimensions, C denotes the number of channels, and r signifies the compression level of the potential space. In order to achieve high computational efficiency and sample quality, the values of C and $r$ are set to $8$ and $4$, respectively. Both the encoder E and the decoder D consist of stacked convolution modules, with each block comprising convolution layers and residual connections. During the generation phase, the decoder is utilized to reconstruct Z in order to produce a CQT spectrogram $\tilde{X}\in R^{T\times F}$ for a given potential representation.

In order to train VAE, this study presents the creation of three loss functions: Gaussian constraint loss, adversarial loss, and CQT spectrogram reconstruction loss. The CQT spectrogram reconstruction loss is employed for computing the average discrepancy between the input sample X and the reconstructed CQT spectrogram. The PatchGAN (Demir & Unal, 2018) discriminator was employed to enhance the quality of reconstruction in the context of adversarial loss. Gaussian constraints are utilized to impose structure on the potential spaces of VAE, promoting the learning of continuous and organized potential spaces that can better capture the underlying structure of the data. In summary, the general training objectives of VAE can be delineated as follows:

(1) $${\mathcal{L}}_{VAE}={\mathcal{L}}_{rec}(x,D(\epsilon (x)))+\lambda {\mathcal{L}}_{adv}(\psi D(\epsilon (x)))+\gamma K{L}_{Gau}(\mu ;{\sigma }^2)$$where ${\mathcal{L}}_{VAE}$ denotes the reconstruction loss, ${\mathcal{L}}_{rec}$ represents the adversarial loss, $\lambda {\mathcal{L}}_{adv}$ stands for the Gaussian constraint loss, $\psi$ signifies the discriminator utilized in the adversarial process, and $\mu$ and $\sigma$ denote the mean and variance of the VAE potential space.

Latent diffusion models

This article utilizes a methodology akin to the Palette (Saharia et al., 2022) image-to-image conversion technology to instruct the LDM as a timbre transfer decoder. In the initial phase, an efficient and low-dimensional latent space was successfully acquired through the perceptual compression model, wherein high frequencies and subtle details are abstracted. This has a significant impact on the extraction of musical attributes such as pitch, loudness, and timbre. In the subsequent sections, the features extracted from the perceptual compression model will be utilized as input for the constructed LDM model to represent the CQT spectrogram within the potential space and to facilitate the transformation between domain X and domain Y. Upon reflection of LDM, it can be broadly stated that LDM operates by acquiring the ability to produce data from noise through two distinct processes. The initial stage involves the forward process, during which Gaussian noise $\gamma \sim N(0,1)$ is incrementally introduced to the input until the two become indistinguishable. The subsequent stage involves the reverse process, also known as the de-noising process, during which the decoder acquires the ability to undo the forward process and reconstruct the data from the noise. LDM represents an enhanced iteration of DDPM, sharing the same training process as DDPM. However, distinctions arise in the modeling process. The training process for LDM occurs in a potential space, enabling faster reasoning time.

Given a CQT spectrogram X with a compressed potential encoding $Z_0\sim q(Z_0)$. In the forward process, starting with the initial data $Z_0$, the diffusion time can be represented as $t\in \{0,1,...T-1\}$ when T steps are considered. The given code is subject to the incremental introduction of Gaussian noise in accordance with the predefined variance table ${\beta }_1,{\beta }_2,...,{\beta }_T$. Following the iterative step T, a series of potential noise variables, denoted as $Z_1,Z_2,...,Z_T$, is subsequently generated

(2) $$q(Z_{1:T}|Z_0):=\prod _{t=1}^{T}q(Z_t|Z_{t-1})$$

(3) $$q(Z_t|Z_{t-1}):=\mathcal{N}(Z_t;\sqrt{1-{\beta }_t}{Z}_{t-1}.{\beta }_{t}I)$$where $Z_T\in N(Z_T;0,I)$ is pure Gaussian noise.

In the inverse procedure, commencing with the Gaussian noise distribution $Z_T$ and the desired instrument CQT spectrogram contained within $E^y$, the denoising process, conditioned on $E^y$, progressively produces the potential encoding $Z_0$ of the target CQT spectrogram through the subsequent steps

(4) $$p(Z_t|Z_{t-1},E^y):=\mathcal{N}(Z_{t-1};{\mu }_{\theta }(Z_T,t,E^y),{\sigma }_t^{2}I)$$

(5) $$p_{\theta }(Z_{0:T}|E^y)=p(Z_T)\prod _{t=1}^T{p}_{\theta }(Z_{t-1}|Z_t,E^y)$$where $\theta$ represents a parameterized neural network defined by Markov chains. In this context, we utilize U-Net, a commonly used model in image synthesis. In practice, the mean ${\mu }_{\theta }$ and variance ${\sigma }_t^2$ are parameterized as follows

(6) $${\mu }_{\theta }(Z_T,t,E^y)=\frac{1}{\sqrt{{\alpha }_t}}(Z_t-\frac{{\beta }_t}{\sqrt{1-{\overline{\alpha }}_t}}{\epsilon }_{\theta }(Z_t,t,E^y))$$

(7) $${\sigma }_t^2=\frac{1-{\overline{\alpha }}_{t-1}}{1-{\overline{\alpha }}_t}{\beta }_t$$where ${\alpha }_t=1-{\beta }_t$, ${\overline{\alpha }}_t=\prod _{i=1}^t{\alpha }_i$, and ${\epsilon }_{\theta }(Z_{t,t})$ are predicted generated noise.

In this article, a reweighted noise training objective is used in practice

(8) $${\mathcal{L}}_{simple}(\theta )={\mathbb{E}}_{\epsilon (x),ϵ\sim \mathcal{N}(0,1),t}||{ϵ}_{\theta }(Z_t,t,E^y-ϵ)|{|}_2^2$$where $\epsilon \sim \mathcal{N}(0,I)$ is derived from a diagonal Gaussian distribution.

The style transfer module is utilized for modeling the CQT spectrogram in potential space and executing complete style transfer, as depicted in Fig. 3. Given a source audio sample CQT spectrogram X, its potential representation $Z_{t_0}$ is calculated by adding step $t_0<T$ noise according to Formula (3). Subsequently, commencing with $Z_{t_0}$ as the initial stage of the reverse process, the conditional mechanism is employed to access the potential encoding of the target instrument CQT spectrogram Y

(9) $$p_{\theta }(Z_{0:t_0}|E^y)=p(Z_{t_0})\prod _{t=1}^T{p}_{\theta }(Z_{t-1}|Z_t,E^y)$$

Therefore, in order to accomplish the transformation of style. If $t_0$ controls the outcomes, the original audio information will not be preserved in the case of $t_0\approx T$.

This research employs a $2\times 2$ convolution layer, which is a two-dimensional convolution layer, in experiments to enhance the capabilities of the underlying U-Net. To improve the U-Net backbone, a cross-attention mechanism is incorporated so that it can produce CQT spectrogram of the target domain according to the requirements needed to accomplish timbre transfer. Three subsampling blocks, each with four filters and residual blocks of $64$, $128$, and $256$, make up the encoder in U-Net. To condense the possible encoding of the CQT spectrogram, average pooling with a pooling size of two is applied after each subsample block’s output. A cross-self-attention block follows the encoder’s last block. The bottleneck part obtained by the encoder is processed by a residual block with $512$ filters and then passed to the decoder. The only difference between the symmetric decoder and encoder is that the decoder uses transposed convolution to create an upsampled layer that increases the feature dimension. The cross-self-attention layer succeeds the last subsampling block, the encoder’s bottleneck, and the decoder’s first higher sampling layer.

Conditioning mechanisms

The conditional mechanism primarily comprises encoder ${\tau }_{\theta }$, which is structured in a manner similar to the encoder used in perceptual compression. The encoder ${\tau }_{\theta }$ projects the input target instrument CQT spectrogram, represented by $y$, into an intermediate representation ${\tau }_{\theta }(y)\in {\mathbb{R}}^{M\times d_{\tau }}$ after encoding it. The cross-attention layer is then used to map this intermediate representation to the middle layer of the U-Net, making it easier to create a CQT diagram that depicts the target instrument’s timbre under condition $y$. The formula for the cross-attention mechanism is expressed as follows

(10) $$Attention(Q,K,V)=softmax(\frac{Q{K}^T}{\sqrt{d}})\cdot V$$where $Q=W_Q^{(i)}\cdot {\phi }_i(Z_t)$, $K=W_K^{(i)}\cdot {\tau }_{\theta }(y)$, $V=W_V^{(i)}\cdot {\tau }_{\theta }(y)$. ${\phi }_i(Z_t)\in {\mathbb{R}}^{N\times d_{\epsilon }^i}$ represents the U-Net intermediate representation that implements ${\epsilon }_{\theta }.W_V^{(i)}\in {\mathbb{R}}^{d\times d_{\epsilon }^i}$, $W_K^{(i)}\in \mathbb{R}{ }^{d\times d_{\tau }}$ and $W_Q^{(i)}\in \mathbb{R}{ }^{d\times d_{\tau }}$ are projection matrices that map intermediate representations from ${\tau }_{\theta }(y)$ to the target domain, thereby enabling the transfer of timbre. The objective function can be rewritten as

(11) $${\mathcal{L}}_{LDM}(\theta )={\mathbb{E}}_{\epsilon (x),y,ϵ\sim N(0,1)}||ϵ-{ϵ}_{\theta }(Z_t,t,{\tau }_{\theta }(y))|{|}_2^2$$where ${\tau }_{\theta }$ and ${\epsilon }_{\theta }$ can be optimized by the objective function.

Description of the data

We used the MusicNet and MAESTRO datasets, together with the real-world recordings from other instruments that we gathered from YouTube, to train and assess the model described in this research. The dataset comprises instruments such as piano, flute, guitar, clarinet, violin, trumpet, organ, strings, and vibraphone. Each instrument collected approximately 2 h of data, during which the tracks underwent quasi-switching to mono and resampling to 16 kHz. The WAV format was used for preprocessing on all data sets. The data set was split up in an 8:1:1 ratio between a training set, a test set, and a validation set.

Implementation details

In this study, the Adam optimizer with a batch size of 16 was employed to train the proposed timbre transfer model over 50,000 iterations, with a learning rate set at 2e-5. The compression ratio of the potential space is 64. During the timbre transfer process, the denoising size was configured to 0.55, the seed size to 42, the steps per sample to 50, and the guidance to 7. The DiffWave model trained over 20,000 iterations at an initial learning rate of 2e-5 using the CQT spectrogram condition and an Adam optimizer with a batch size of 32. It is important to note that all models developed in this study are built upon the Pytorch library. All models were trained using thre NVIDA RTX3090Ti graphics processing units.

Evalutaion metrics

The performance of the proposed model is assessed using both objective evaluation and subjective evaluation methods in this study.

In the objective evaluation, the following indicators were adopted:

• Jaccard distance: The Jaccard distance is utilized to quantify the disparity between two sets of pitches, A and B, in order to evaluate the extent to which the produced tracks vary in pitch profile. The Jaccard distance can be computed using the following formula: (13) $JD(A,B)=1-|\frac{A\cap B}{A\cup B}|$

  The Jaccard distance values range from $0$ to $1$, where lower values indicate fewer mismatches, indicating a greater degree of similarity between the generated pitch profiles. The pitch profile is determined through the utilization of the multi-pitch version of MELODIA (Salamon & Gómez, 2012) incorporated in the Essentia library (Bogdanov et al., 2013), with the pitch being rounded to the nearest semitone.

• SSIM: The SSIM (Structural Similarity Index) (Setiadi, 2021) is employed for assessing the perceived quality disparity between the original image and its reconstructed counterpart. Here, the reconstructed CQT spectrogram are compared with the original spectrogram to evaluate the fidelity of the model reconstruction. The SSIM value ranges from −1 to 1, with a higher value indicating greater structural similarity between images.

• Fréchet Audio Distance (FAD): FAD (Roblek et al., 2019) is utilized to quantify the similarity between synthesized audio and authentic audio. Based on the PyTorch implementation, the FAD was calculated by embedding the VGGish model (Shi et al., 2019). The approach considers the embeddings as a continuous multivariate Gaussian distribution and computes the Frechet distance between the real and generated data. The formula for calculating FAD is as follows: (14) $FAD=||{\mu }_r-{\mu }_g|{|}^2+tr(\sum _r+{\mu }_g-2\sqrt{\sum _r\sum _g})$where $({\mu }_r,\sum _r)$ represents the embedded mean and covariance of the real data, and $({\mu }_g,\sum _g)$ represents those of the generated data. A smaller FAD value results in a more realistic sample set being generated.

• Accuracy: We have developed multiple instrument classifiers through training. A network resembling AlexNet (Iandola et al., 2017) was trained using audio clips extracted from a dataset to categorize the transmitted audio based on different instruments. The output of the classifier undergoes processing by the Sigmoid function, which yields the classification probability for each segment. In this article, the classification probability is articulated as a measure of confidence.

The mean opinion score (MOS) was utilized for subjective evaluation. The MOS score was calculated based on an anonymous listening test involving 50 participants. The scoring system ranges from 1 (low) to 5 (high) and encompasses three dimensions:

• 1. Success in style transfer (ST): evaluating the degree to which the timbre of the transfer version aligns with the target in perception.

• 2. Content retention (CP): the degree to which the transmitted version of the music content matches the original version.

• 3. Sound quality (SQ): How does the audio sound as a whole?

Evaluation of timbre transfer

This section will assess the impact of timbre transfer. As demonstrated in the prior study (Lu et al., 2019), our objective was to assess our methodology using three criteria: (1) content retention, which measures the degree to which the tonal content of the input is preserved in the output; and (2) style fit, which evaluates the extent to which the output aligns with the target timbre. (3) The quality of the audio. In this regard, we employed both subjective and objective indicators to assess the measurements.

Subjective evaluation. To subjectively evaluate the ability of timbre transfer, the MOS of the listening test was gathered from 50 participants. The participants in the study primarily consisted of individuals with a strong affinity for music. The target mixture was utilized as the reference standard for each work in each evaluation cycle, and the appropriate translated output was given after that. The conditions and examples were presented in a random order in each segment, which the tester was not aware of beforehand. The test is segmented into two parts, involving one-to-one and many-to-many conversions.

The initial stage of the assessment involved a one-on-one instrument conversion segment, during which the participant listened to eight musical pieces representing four types of transitions: piano to guitar, flute to trumpet, trumpet to piano, and flute to organ. In each round of scoring, the order of conditions and examples in each individual part of the test was randomized. The results obtained through the hearing test in the first stage are presented in Table 1.

The results indicate that our model demonstrates strong performance in the task of one-to-one instrument timbre transfer. The MOS for both style transfer and content retention are around four, indicating that our model performs well on these tests. It has been noted that our approach works well in cases where the target instrument has a relatively “stable” sound, such as in the conversion of a piano. Meanwhile, the sound quality scores for all one-to-one conversion tasks are high, indicating that our method performs well in generating audio, although the generated audio is quite a bit different from real-world recordings.

In the subsequent phase, we assessed the task of one-to-many timbre style transfer. In the context of one-to-many timbre style transfer, this study has chosen six instruments from the test set and conducted transfers between each instrument and the other five instruments, as well as the original instrument, resulting in a total of 36 transfer pairs ( $6\times 6=36$). Simultaneously, we also took into account four different tracks. Two of the tracks pertain to the conversion from vibraphone/clarinet to piano/violin, while the other two are related to the reverse conversion. The conditions and examples in the test were randomized, and the order was not known to the participants beforehand. Also, all participants rated the output in isolation. The MOS score is presented in the three dimensions of ST, CP, and SQ and subsequently averaged to obtain the final assessment score. A heat map was generated to illustrate the hearing test outcomes of one-to-many transfer for a single timbre, as depicted in Fig. 5. Table 2 presents the auditory test results of multi-pair multi-transfer for mixed instruments.

Differences in performance can be observed among various timbre transfers in the context of many-to-many transfers of a single timbre. Additionally, it is evident that the timbre reconstruction effect surpasses the transfer effect. The transfer of timbre from other instruments to the piano is notably effective, while the transfer to the trumpet is comparatively less successful. One possible explanation is that the timbre of the piano changes more regularly compared to the trumpet, because the decay of the piano timbre after each note is struck is relatively predictable. This relative regularity of timbral characteristics makes piano timbral characteristics easier to learn in timbre space. Conversely, the trumpet’s timbre exhibits such significant variation that it is challenging to master and manipulate. The data presented in the table indicates that our approach is comparatively less effective in facilitating many-to-many conversion of multi-instruments as opposed to single-instrument many-to-many conversion. This disparity may be attributed to the challenges associated with extracting and distinguishing the timbre of mixed instruments.

Objective evaluation. In the objective evaluation, this study will employ the Jaccard distance to assess the content retention ability of the model, and FAD to examine the perceived similarity between the generated audio and the original audio. The SSIM is used to measure the perceived similarity between the reconstructed CQT spectrogram and the original spectrogram in order to evaluate the model’s ability to generate a high-quality CQT spectrogram. An instrument classifier is utilized to assess the precision of the model’s timbre transfer. The Mel-frequency cepstrum coefficient (MFCC) is widely regarded as a reliable approximation of timbre (Richard, Sundaram & Narayanan, 2013). To facilitate a more intuitive observation of the timbre transfer of the instrument, Fig. 6 displays the MFCC diagram of several sets of input and output audio. The results of the objective evaluation are presented in Table 3.

The timbre transfer between the input and output audio may be clearly seen in the picture. Consistent with the subjective evaluation, the objective evaluation statistics show that one-to-one conversions for single instruments perform better than many-to-many conversions for mixed instruments. From the data, it can be observed that the model proposed in this article has high values in terms of accuracy, indicating that the model performs well in the forced-choice classification task. Although high accuracy means that the output sound is closer to the timbre of the target instrument, accuracy alone does not provide a comprehensive measure of the quality of timbre transfer, and further sonic assessment and subjective evaluation are necessary. It is also noteworthy that the generated pitch contour closely resembles the pitch contour of the target input, as indicated by the low JD values. This illustrates how well the content was retained between the input and the generated audio. Higher SSIM values indicate that the CQT spectrograms generated by the model are of good quality, which is one reason why subsequent music generation has a lower FAD. Concurrently, the low FAD value indicates that, as expected, the audio output of the model has better perceptual quality.

The analysis of the objective results largely corresponds to the subjective reviews, which is as we expected. This suggests that our model is capable of handling both one-to-one and many-to-many timbre transfer tasks. This confirms that the diffusion model performs well on timbre transfer and music synthesis tasks, offering researchers a more intriguing direction for their work. However, in contrast to the one-to-one style transfer task, the many-to-many task was relatively poor, suggesting that the model is underperforming for learning and extracting timbres from mixed instruments. This suggests that the model struggles to accurately separate and learn the timbres of mixed instruments, which is a direction that needs to be pursued in the future.

Additional properties

Apart from the primary assessment outcomes, we investigated several aspects of the model, specifically the compromises between sampling step and quality, as well as the relationship between compression ratio and quality. Figure 7 displays the outcomes of our trials, which involved the piano2guitar and vibraphone/clarinet2piano/violin tasks. These tasks represent one-to-one and many-to-many timbre transfer tasks, respectively.

Trade-off between sampling steps and quality. From the results in the Fig. 7, it can be noticed that increasing the number of sampling steps in the potential diffusion model improves the quality, which may be due to the fact that the generated potential space is more detailed and at the same time produces better overall structured music. However, at 50-100 sample steps, there is no significant decrease in the FAD values with increasing sample steps, which may be related to the fast fitting of the model. Considering the speed relationship, we weighed the selection of 50 as the sampling step. This is because model generation becomes slower as the number of sampling steps increases.

Trade-off between compression ratio and quality. From the results in Fig. 7, we can see that lowering the compression ratio improves the quality of the generated music, because a low compression ratio is closer to the original data, but it also slows down the running speed of the model. Considering that we need to process higher dimensional data later, we weigh the performance of the model and choose a compression ratio of 64.

Comparison with baseline model

To validate the effectiveness of the method proposed in this article, we consider a comparison with three baseline models. For the baseline models, we consider VAE-GAN (Bonnici, Benning & Saitis, 2022), Music-Star (Alinoori & Tzerpos, 2022) and DiffTransfer (Comanducci, Antonacci & Sarti, 2023) in this article. The VAE-GAN architecture combines a variational encoder and a generative adversarial network for constructing meaningful representations of source audio and generating realistic target audio. It enables many-to-many timbre style transfer, where the dataset of this article is used for training according to the procedure described in Bonnici, Benning & Saitis (2022). Music-Star is an audio translation system based on the WaveNet autoencoder, which is capable of converting audio waveforms into the styles of different musical instruments. The model used in this article was trained based on the description found in the literature (Alinoori & Tzerpos, 2022). DiffTransfer is a timbre transfer method for both single and multi-instrument use that is based on a denoising diffusion implicit model. In this study, the model is trained using the description of Comanducci, Antonacci & Sarti (2023). The findings of an objective and subjective assessment of the proposed method in comparison to the baseline model are given. We consider piano-to-guitar and piano-to-vibraphone timbre transfers, for one-to-one instrument transfers. For many-to-many transformations involving audio from multiple instruments, we consider timbre transfer from vibraphone/clarinet to piano/strings. Table 4 presents the outcomes of the subjective assessment. In every section of the test, the conditions and examples were presented in a random order, and the participants were unaware of the method used to generate them. Table 5 and Fig. 8 show the outcomes of the objective assessment, where the results in Fig. 8 are presented to verify the accuracy of the proposed model with respect to the success of the style transfer from the baseline model. The classification criteria use the same instrument classifiers trained on a network similar to AlexNet (Shi et al., 2019).

The suggested approach performs better than the two baseline models, VAE-GAN and Music-Star, according to the subjective scores in Table 4. On the other hand, in comparison to the DiffTransfer model, the performance is comparable. This demonstrates the favorable performance of our method. We note that our method shows good sound quality relative to the baseline model, suggesting that DiffWave is a promising direction for the field of music synthesis.

The classification results presented in Fig. 8 demonstrate that the audio output from the approach suggested in this research performs much better in terms of resemblance to the target instrument timbre than both VAE-GAN and Music-Star. Regarding DiffTransfer, the classification outcomes show a slight gain over the baseline model, but the difference is not significant. This demonstrates that within the topic of music style transfer, diffusion modeling is a promising direction. The technique suggested in this article performs much better in constructing single and multiple instrument timbre transfer tasks than the two baseline models, VAE-GAN and Music-Star, as can be seen by a brief review of the data provided in Table 5. The results also demonstrate that, in terms of FAD values, the approach suggested in this research marginally outperforms the DiffTransfer baseline model; this result may be explained by the usage of the DiffWave vocoder model in this work. Nevertheless, the approach suggested in this article is somewhat less than the DiffTransfer baseline model when the JD value is taken into account. This could be because the latent diffusion model used in this article necessitates compression in the potential space.

Consider that the choice of vocoder may have an impact on the performance of timbre transfer. For a more comprehensive comparison, we used the DiffWave vocoder with the original SoundStream vocoder in our experiments with the baseline model DiffTransfer. The results show a slight performance advantage of DiffWave over SoundStream in terms of sound quality. Specifically, in the Piano2Guitar task, the DiffWave version of DiffTransfer had a FAD value of 3.20, which was slightly better than the SoundStream version of DiffTransfer at 3.34. In the Piano2Vibraphone and Vibraphone/Clarinet2Piano/Strings tasks, the performance is also slightly better. This shows that DiffWave is a promising direction in the field of music synthesis.

Upon conducting a concise analysis of both subjective and objective outcomes, it is evident that the proposed approach demonstrates favorable performance in the transfer of timbre. Compared to the baseline model, the diffusion model performs better in the timbre transfer task. This illustrates how effectively diffusion models perform in music synthesis and timbre transfer tasks when compared to models like VAEs and GANs, and it offers an appealing option for subsequent research.