Semiautomated generation of species-specific training data from large, unlabeled acoustic datasets for deep supervised birdsong isolation

Introduction

An increasing number of research efforts have been focused on collecting and analyzing wildlife vocalizations in recent years (Laiolo, 2010; Aide et al., 2013; Browning et al., 2017; Sugai & Llusia, 2019; Sugai et al., 2019; Teixeira, Maron & Van Rensburg, 2019; Desjonquères, Gifford & Linke, 2020). Bioacoustic monitoring can be used to detect species, count the number of vocalizations per species, analyze changes in vocalization structure over time, and trilaterate the approximate location of individual animals (Frommolt & Tauchert, 2014; Teixeira, Maron & Van Rensburg, 2019; Besson et al., 2022). Because monitoring can be performed passively and continuously over long periods (e.g., months), large datasets often accumulate, requiring automated analytical tools such as BirdNET, CARACAL, and Vesper (Mills, 2017; Wijers et al., 2019; Kahl et al., 2021; Stowell, 2022). However, the acoustic richness in these datasets can cause automated analytical approaches to perform poorly (Evrendilek & Akcan, 2011; Akcan & Evrendilek, 2013; Zhang et al., 2014; Funosas et al., 2023). For example, a signal of interest in a behavioral study may be a single faint birdsong within a complex dawn chorus, or a bird competing to be heard against road noises in an urban-rural gradient (Priyadarshani et al., 2016; Wang, Venkataramani & Smaragdis, 2020; Dai et al., 2021). Furthermore, each recording site is different, compounding the complexity of the problem. Here, we focus on birdsongs recorded in these noisy environments and how site-specific deep supervised source separation (DSSS) can be used to analyze them.

Any large-scale birdsong dataset analysis often begins by finding the birdsong of interest within potentially thousands of hours of audio. By running raw audio samples through a classification algorithm (e.g., BirdNET; Kahl et al., 2021), the confidence of having recorded a bird species can be determined for a given time interval (Acevedo et al., 2009). Furthermore, if calibrated against known call rates, it is possible to further estimate abundances by counting the number of songs from each species (Dawson & Efford, 2009; Pérez-Granados & Traba, 2021). However, many classification algorithms are trained on mostly clean birdsongs with little interference from natural and anthropogenic sources, and their accuracy degrades with an increasing amount of competing signals (Apol, Valentine & Proppe, 2020). BirdNET, for instance, is trained on birdsongs from the Macaulay Library, eBird, and Xeno-Canto. In contrast to most passively collected datasets, these datasets primarily comprise clean, professionally recorded birdsongs, with only a minor presence of noisy recordings (Planqué & Vellinga, 2005; Sullivan et al., 2009; Kahl et al., 2021; Funosas et al., 2023).

Birdsong feature labeling and analysis are also informative in behavioral ecology studies (Vehrencamp et al., 2013). Features such as peak frequency and song amplitude can provide insights into reproductive, defense, and social behaviors (Brumm & Todt, 2004; Teixeira, Maron & Van Rensburg, 2019). These insights allow us to measure how different environmental changes and anthropogenic activities impact bird behavior. For example, an upward shift in song frequency has been observed in several species in response to anthropogenic noise (Halfwerk & Slabbekoorn, 2009; Cardoso & Atwell, 2012; Nemeth et al., 2013). In many cases, birdsong feature labeling and analysis have focused on the minimum and maximum frequency of songs. The labeling process often involves manually annotating the spectrograms because background noise makes automating this process challenging. Even then, some human-labeled error is as high as 300–1,800 Hz (Brumm et al., 2017). It has been argued that the subjective process of identifying minimum frequencies on a spectrogram would be better served by an objective, power-based approach (Cardoso & Atwell, 2012). Yet, these approaches typically require clean audio. Any interference that exceeds the joule-per-second power threshold could be mislabeled as the minimum or maximum frequency.

Another application that is negatively impacted by background noise is trilateration, where the source of a sound is spatially located. By spacing out several microphones, the time difference of arrival between microphones can be used to trilaterate the location of an animal (Grande et al., 2019). This can be used to autonomously track bird movement to and from their nest (Nathan et al., 2022). Or, if obtaining population counts, trilateration can be used to prevent birds from being counted twice. However, with any background noise, finding the time difference of arrival becomes increasingly hard (Evrendilek & Akcan, 2011; Akcan & Evrendilek, 2013).

Due to interference during recording, the original ground-truth birdsong is often lost, making it challenging to objectively define a clean birdsong. Consequently, we loosely define a clean birdsong based on the degradation’s impact on the specific bioacoustics application. For birdsong species classification tasks, a clean birdsong can be confidently labeled by an expert ornithologist, while a noisy birdsong is challenging to identify due to interference. Similarly, for birdsong feature analysis, clean birdsongs can be confidently labeled by an ornithologist, whereas noisy birdsongs may pose difficulties. In trilateration applications, the noisiness of a birdsong can be measured by the resulting error in the estimated position.

One upstream solution to these issues is to develop robust methodologies for extracting clean birdsongs from audio recordings. However, this is challenging for several reasons. At first glance, one may consider using visual semantic separation. This technique assigns each pixel from an image to a source, often using a machine learning model. For instance, for an autonomous car, pixels from the road lines would be assigned accordingly (Rahman, Khan & Barnes, 2020; Ren et al., 2023). Audio waveforms can be converted into spectrograms using Fourier transformations and then each time-frequency bin could be treated as a pixel. However, unlike pictures, audio is additive: each time-frequency pair could have multiple sources. For instance, if a certain frequency is heard during a concert, it could have come from several instruments at the same time. Furthermore, some bird vocalizations can be very similar, making it even harder to differentiate between them.

Some birdsong source separation (SS) approaches use unsupervised machine learning which learns from unlabeled datasets where there is no ground truth. For example, (Sun, Yen & Lin, 2022) use non-negative matrix factorization (NMF) to separate audio samples into similar categories. For instance, it may group low road noises into one category and higher human speech into another (Sun, Yen & Lin, 2022). However, NMF is limited in complexity because it is purely a linear method. A neural network is preferred because it can better represent more complex function spaces such as birdsong SS (Thakur & Konde, 2021). This is what Denton, Wisdom & Hershey, 2021a; Denton, Wisdom & Hershey, 2021b do with Bird MixIT, a bird SS tool that uses deep unsupervised mixture invariant training (MixIT). By learning to separate mixtures of noisy birdsongs from one another, the model learns to separate individual birdsongs from the original noisy recordings (Wisdom et al., 2020; Denton, Wisdom & Hershey, 2021a).

However, this approach lacks control and is less customizable. Besides constructing the training dataset, there is little one can do to control the final model. These models often learn to separate the audio in a random order, splitting some sources and combining others. For instance, looking at example 4 from Bird MixIT’s Caples sample outputs, the Rebut class is combined with the background noise using the first model, and the Btywar class is combined with the background noise using the second (Denton, Wisdom & Hershey, 2021b). Because the training process is unsupervised, it is very challenging to address this or any other undesirable behaviors.

In contrast, supervised learning turns the SS into a training dataset creation problem. Because supervised learning models learn to replicate the training dataset, a high-quality training dataset often means a high-quality model. This offers greater control and is more customizable. One can specify exactly what sounds constitute the foreground and the background, and the model will learn to separate them accordingly. For instance, if trying to separate the Rebut and Btywar birdsong classes from background noise, each could be assigned a specific output channel of a supervised SS model, and there would be no ambiguity in the model’s desired behavior.

Fortunately, several deep supervised audio SS models exist such as ConvTasNet (Convolutional Time-Domain Audio Separation Network), DPRNNTasNet (Dual-path Recurrent Neural Network Time-Domain Audio Separation Network), and SuDORMRFNet (Successive Downsampling and Resampling of Multi-Resolution Features Network) where ConvTasNet and SuDORMRFNet use convolutional neural networks (CNN) and DPRNNTasNet uses a recurrent neural network (RNN) (Luo & Mesgarani, 2019; Luo, Chen & Yoshioka, 2020; Tzinis, Wang & Smaragdis, 2020). While these and several other supervised SS models exist, the overwhelming majority of these models require clean signals from each source as the ground truth for training. This is seen in the music SS dataset MUSDB18 where each instrument is recorded separately and without interference (Rafii et al., 2017). However, for birdsong SS, obtaining a clean signal of both the birdsongs and the background is challenging because the recording does not occur in a controlled environment. (Priyadarshani et al., 2016; Wang, Venkataramani & Smaragdis, 2020; Dai et al., 2021). And, if SS models are trained on audio with even a little interference, the final output will have that same interference (Lones, 2023).

Conveniently, several publicly-available birdsong datasets exist with cleanly-recorded birdsongs, and some approaches have trained deep supervised SS models on these datasets. By mixing together different species of animals whose vocalizations interfere, these approaches were successfully able to separate the different vocalizations (Izadi, Stevenson & Kloepper, 2020; Bermant, 2021; Zhang et al., 2022). However, due to site-specific variations in birdsongs and background noise, these approaches’ practicality for actual field use is questionable. We show that ideally, each source separation model should be fine-tuned for the recording sites of interest to account for this variation.

Here, to the best of our knowledge, we are the first to train a deep supervised source separation (DSSS) model using site-specific, passively-recorded data. We introduce a site-specific DSSS workflow that addresses the challenges associated with supervised SS with a two-step process. First, we create a site-specific synthetic dataset by extracting, denoising, and mixing sufficiently clean birdsongs and background noise from passive acoustic datasets. Second, we train on the dataset. Unlike previous DSSS methods, our model learns to remove background noise specific to that site, and compared to unsupervised SS, our approach has more control and is more customizable.

We then test our methodology using traditional SS metrics such as source-to-distortion ratio, source-to-interference ratio, and source-to-artifact ratio (SDR, SIR, and SAR) as well as bioacoustics-inspired metrics such as automated feature labeling accuracy and species-specific trilateration accuracy. These additional metrics both highlight potential downstream SS applications and more rigorously vet our model.

Materials & Methods

We present a semiautomated workflow to train a site-specific DSSS model using raw field recordings to isolate Golden-Cheeked Warbler (GCW) birdsongs. The Golden-Cheeked Warbler (Setophaga chrysoparia) is a federally endangered migratory songbird that breeds only in mature Ashe juniper (Juniperus ashei) and oak (Quercus spp.) woodlands within central Texas. This species vocalizes three types of songs: A, B, and C; however, C songs do not occur often, and we do not include them in our study (Groce et al., 2010). Our analysis proceeded as follows:

Results

We found that the SuDORMRFNet model yielded the highest SDR, SIR, and SAR after seeing 250,000 six-second samples. Training ConvTasNet, another convolutional model, also yielded a high SDR, SIR, and SAR suggesting that convolutional neural networks outperform recurrent neural networks in this domain.

After retraining SuDORMRFNet without site-specific audio, the SDR, SIR, and SAR decrease by 9.33 dB, 24.07 dB, and 3.60 dB respectively, highlighting the importance of site-specific audio in source separation. Notably, the performance drop in the source-to-interference ratio (SIR) suggests that this model was unable to handle background interference specific to this site (Fig. 1A).

Model Robustness

Our five robustness tests all obtained an SDR, SIR, and SAR above 19 dB. For reference, the leading model for the music SS dataset MUSDB18 only obtained an SDR of 9.200 dB (Rafii et al., 2017; Rouard, Massa & Défossez, 2022). Because we used site-specific data and a robust data generator, we were able to maintain high SS metrics across the entire training distribution. This means our model was able to interpolate well from examples it saw during training. Notably, we had very few training samples for A songs with no trailing hook; however, by inflating its occurrence in our data generator, we maintained the same model performance as other song types (Fig. 1B). Our birdsong occurrence frequency test was also inside the training distribution. Even if there are 20 birdsongs in a minute, with some potentially overlapping each other, the model obtained metrics above 22 dB (Fig. 1C). The same was true for our birdsong-background ratio test: for all recordings, we obtained metrics above 19 dB even though recordings with ratios at or below 0.02 are barely perceivable to the human ear (Fig. 1D).

Outside of the training distribution, our model also achieved SS metrics above 27 dB, including for different background and virtual forest types. During training, we only used one unique background recording at a time with no mixing; however, for one test, we mixed two unique backgrounds on top of one another. Despite this, the results remained relatively unphased with the extra background noise (Fig. 1E). We also convolved the isolated birdsong with random forest impulse responses. Despite not having trained with this functionality, our model still obtained SS metrics above 27 dB (Fig. 1F). The model learned to preserve this convoluted distortion solely from the dataset’s diversity and data augmentation. Looking closer, we visually inspected isolated birdsongs with and without convolution, and we saw that the convolution distortion is maintained (Figs. 2C and 2D).

Visual inspection

We then ran our model on several noisy birdsongs that we reserved for testing. The resulting spectrograms suggest that our model removes most of the background noise (Fig. 3). We also ran our model on several outlier birdsongs, and our model appeared to have preserved all features (Fig. 4).

Discussion and Conclusions

Overall, we implemented a site-specific deep supervised source separation (DSSS) approach to isolate birdsongs from field recordings containing interference from humans and other bird species. Across our five robustness tests, our model obtained consistently high SDRs, SIRs, and SARs even with faint birdsongs and different background noise levels.

However, while SDR, SIR, and SAR are commonly used metrics for SS, they have shortcomings in bioacoustics. For example, a model could have a high SDR, SIR, and SAR, but distort the minimum and maximum frequencies. Because these signal boundaries are only a fraction of the SDR, SIR, and SAR metrics, they could easily be overlooked. This is problematic because distorted signal boundaries could drastically alter research results, such as in the numerous studies observing an upward shift in song frequency in response to anthropogenic noise (Halfwerk & Slabbekoorn, 2009; Cardoso & Atwell, 2012; Nemeth et al., 2013).

Consequently, we also tested our model specifically for downstream bioacoustics applications. Visual inspection confirmed unique birdsong variations like high maximum frequencies and distortions from environmental effects like forest reverberations were accurately captured. This suggests our model learned the nuances of different birdsongs rather than overfitting the training data. Preserving these distinctive features enables automating sophisticated analyses like peak frequency and buzz length measurement. Currently, birdsong labeling analysis is often manual because background noise makes automating this process very challenging. However, if automated with source separation (SS), this would allow large-scale feature analysis without human bias. By using our DSSS model, we were able to algorithmically extract minimum and maximum birdsong frequencies with single-digit mean percent error. While not tested, we predict other features such as buzz duration could be similarly extracted.

The current approaches for trilateration also break down with background noise (Evrendilek & Akcan, 2011; Akcan & Evrendilek, 2013; Zhang et al., 2014), and our DSSS workflow is again a potential solution. By isolating a single species of bird, the time differences of arrival can be extracted using cross-correlation. Then, the location of each birdsong can be calculated with a mean absolute error of 2.58 m.

Finally, while our DSSS model did not improve BirdNET’s performance, we hypothesize this is caused by BirdNET’s lack of robustness. As seen in Fig. 5, our model eliminated a lot of the background noise, but surprisingly, BirdNET’s performance decreased. In addition to very noisy recordings, we hypothesize that BirdNET was not trained on such very clean recordings and consequently failed during our tests. While our model likely helps in some cases, this benefit is canceled out by BirdNET’s fragility with distribution shifts. Looking forward, other classification models may work better. Denton, Wisdom & Hershey (2021a) and Denton, Wisdom & Hershey (2021b) saw improvement in their custom classification model after running their source separation model. However, they used unsupervised learning without site-specific audio. Or, SS could instead be used to augment the classifier’s training set. A SS model could be used to isolate birdsongs for various species. Then, the isolated birdsongs could be mixed with diverse background audio to create a more robust dataset and classification model. Unlike BirdNET, this classifier would be robust to recordings with very little noise if some isolated birdsongs were left unmixed.

Overall, this research builds on previous methods because it is more customizable and has improved site-specific performance. Because our approach is supervised, the inputs and outputs are fully controllable. And, by training on site-specific recordings, we boost site-specific performance by 9.33 dB, 24.07 dB, and 3.60 dB for SDR, SIR, and SAR respectively.

While testing was limited to Golden-Cheeked Warblers, we expect our approach to be easily adaptable to all species of birds. Likewise, while only testing a few downstream applications, our approach should be adaptable to any application needing clean birdsongs.

In the future, DSSS will enable researchers to conduct more accurate bird vocalization feature tracking, trilateration for precise location estimation, and, potentially, species identification. Going forward, integrating our DSSS methodology with existing bird monitoring systems and expanding it to handle other species could greatly enhance our understanding of ecology and contribute to more effective conservation efforts.

Supplemental Information