Multimodal system for recording individual-level behaviors in songbird groups

The sound-isolation chamber

All our experiments were performed in a sound-isolation chamber (Industrial Acoustic Company, Winchester, UK) of dimensions 124 cm (width) × 90 cm (depth) × 130 cm (height). The isolation chamber has a special silent ventilation system for air exchange at a rate of roughly 500 l/min. Two types of light sources made of light emitting diodes (LEDs, Waveform Lighting) are mounted on aluminum plates on the ceiling of the chamber: (1) LEDs with natural frequency spectrum (consuming a total electric power of 80 W); and (2) ultraviolet LEDs of wavelength 365 nm (consuming a total electric power of 13 W).

The ceiling of the chamber contains three circular holes of 7 cm diameter through which we inserted aluminum tubes of 5 mm wall thickness that serve as holders for three cameras. The tubes also conduct the heat of the LEDs (93 W) and the cameras (13 W) to the outside of the chamber where we placed silent fans to keep the cameras below 55 °C.

Where possible, we covered the sidewalls of the chamber with sound-absorbing foam panels. A door sensor measures whether the door of the chamber is open or closed. The temperature inside the chamber was roughly 26 °C and the humidity was on average 24 $\pm$ 1% (std).

Cameras and video acquisition system

Into the three circular holes of the chamber, we placed industrial 3-Megapixel cameras (Basler acA2040-120uc) with zoom lenses (opening angles: top view: 45° × 26°, back view: 55° × 26°, side view: 26° × 35°) and exposure times of 3 ms. To visualize nests in the dimly lit nest boxes, we used monochrome infrared cameras (2 MP, Basler daA1600-60um) and fisheye lenses (143° × 112°). The frame rate of the video is about 48 frames per second (frame period 21 ms). The spatial resolution of the main cameras is about 2.2 pixels/mm.

With the cameras, we filmed the arena directly from the top and indirectly from two sides via two tilted mirrors in front of the glass side panels. This setup yields a large object distance of about 1–1.5 m, which allows for a small perspective distortion. We relayed the uncompressed camera outputs (approx. 400 MB/s in total) to a host computer via a USB3 Vision interface. Each camera received frame trigger signals generated by the radio interface.

On the host computer that controlled the acquisition system, we ran two custom applications: BirdVideo, which acquires and writes the video data to a file, and BirdRadio, which acquires and writes the microphone and transmitter signals to a file (see Extension section, ‘Software and data’). The generated file pairs are synchronized such that for each video frame there are 512 audio samples (an audio frame) and 512 transmitter signal samples (a radio frame). While the recording is gapless, it is split into files of typically 7 mins duration.

BirdVideo is written in C++, makes use of the OpenCV library (Bradski, 2000) and FFMPEG (Tomar, 2006) and undistorts the nest camera images with a fisheye lens model and transforms the five camera images into a single 2,976 × 2,016 pixel-sized composite image (Fig. 1B). The composite images we compress with the h264 codec on a NVIDIA GPU and save the video as an MP4 (ISO/IEC 14496-14) file. We used constant-quality compression with a variable compression ratio in the range 150–370, resulting in a data rate of 0.72–1.8 MB/s, depending on the number of birds and their activity in the arena. Compression ratios in this range do not significantly decrease key point tracking performance (Mathis & Warren, 2018).

Microphones

We installed four microphones on the four side walls of the isolation chamber and one microphone on the ceiling. We mounted two more microphones in the nest boxes and one microphone outside the isolation chamber.

Antennas

We mounted four whip antennas (referred to as $\mathrm{A},\mathrm{B},\mathrm{C}$, and $\mathrm{D}$) of 30 cm length (Albrecht 6157) perpendicular to the metallic sidewalls and ceiling of the chamber (using magnetic feet). We fed the antenna signals to a USRP containing a large FPGA (Fig. 7A).

The bird arena

Inside the chamber, we placed the 90 cm × 55 cm (floor size) bird arena. To minimize acoustic resonances and optical reflections, we tilted the four 40-cm high side panels of the arena. We tilted opposite panels by 11° and 13° toward the inside, respectively. Two of the side panels are made of glass for videography and the two opposite panels are made of perforated plastic plates. We covered the floor of the arena with a sound-absorbing carpet. The upper part of the arena we enclosed with a pyramidal tent made of a fine plastic net that reached up to 125 cm above the arena. At a height of 35 cm, we attached two nest boxes to the side panels, each equipped with a microphone and a camera. On the floor, we placed perches, a sand-bath, and food and water trays.

Transmitter device

Our transmitter devices are based on the FM radio transmitter circuit described in Ter Maat et al. (2014) (Fig. 2). To distinctly record birds’ vocalizations irrespective of external sounds, we replaced the microphone with an accelerometer (Anisimov et al., 2014) that picks up body vibrations and acts as a contact microphone. The accelerometer (Knowles BU21771) senses acceleration with a sensitivity of 1.4 mV/(m/s²) in the 30 Hz–6 kHz frequency range (Knowles Electronics, 2017). Inspired by Ter Maat et al. (2014), we performed the frequency modulation of the accelerometer signal onto a radio carrier using a simple Hartley oscillator transmitter stage with only one transistor and a coil that functions both as the inductor in the resonator circuit and as the antenna. We set the carrier frequency ${\omega }_c$ of the radio signal in the vicinity of 300 MHz, which corresponds to an electromagnetic wavelength of about 1 m.

The total weight of the transmitter including the harness and battery is 1.5 g, which is ca. 10% of the body weight of a zebra finch. The transmitters are powered by a zinc-air battery (type LR41), which lasted more than 12 days.

The radio circuit on the transmitter uses a single transistor and an inductor coil as emitting antenna to modulate the measured acceleration signal $a_T\left(t\right)$ (where $T$ stands for Transmitter and $t$ is the time) onto a (relatively stable) radio carrier frequency ${\omega }_c\left(t\right)$. The accelerometer signal $a_T\left(t\right)$ is encoded as (instantaneous) transmitter frequency ${\omega }_T\left(t\right)\simeq {\omega }_c\left(t\right)+c{a}_T\left(t\right)$ with $c$ some constant. The high-pass filtered tracking frequency ${\omega }_{\alpha }\left(t\right)$ (with a cutoff of 200 Hz) is our estimate of the accelerometer signal $a_T\left(t\right)$ (Figs. 3, 4, 5 and 6). We refer to it as the demodulated (transmitter) signal.

Before mounting a device on a bird, we adjusted the coil to the desired carrier frequency in the range of 250–350 MHz by slightly bending the coil wires. Thereafter, we fixed the coil and electronics in dyed epoxy. The purpose of the dyed epoxy is to help identify the birds in the video images, protect the printed circuit board, and stabilize the coil’s inductance (Figs. 2B and 2C). We mounted the devices on birds using a rubber-string harness adapted from Alarcón-Nieto et al. (2018) (Fig. 2D).

Stage 1: Multi-antenna radio signal reception

We fed the antenna signals to a universal software radio interface (USRP-2945; National Instruments, Austin, TX, USA), which comprises four independent antenna amplifiers, the gains of which we set to 68 dB, adjustable in the range −20 dB to +90 dB.

Stage 2: Intermediate band

At the input stage of our demodulation technique, the radio interface generates from the amplified radio signal a down-converted signal. The radio receiver filters out an 80 MHz wide band around the local oscillator frequency ${\mathrm{\omega }}_{\mathrm{L}\mathrm{O}}$, which we typically set to 300 MHz.

These four signals then become digitally available on the FPGA as complex valued signal of 100 MHz sampling rate. We call them the intermediate signals $z^a\left(t\right),a\in \left\{\mathrm{A},\mathrm{B},\mathrm{C},\mathrm{D}\right\}$ (intermediate band in the frequency domain; Fig. 7B). On the FPGA, they are 100 MS/s complex-valued signals of 2 × 18 bits precision. For details about the analog down conversion and the sampling of complex valued signals, see the National Instrument manual of the USRP and the documentation of our custom radio software.

Stage 3: digital down-conversion and tracking

On the FPGA, we instantiated eight demodulators that simultaneously demodulated up to eight transmitter channels. Each demodulator contains four digital down-converters that extract from the four intermediate signals $z^a\left(t\right)$ the four baseband signals $u^a\left(t\right)$, one for each antenna $a\in \left\{\mathrm{A},\mathrm{B},\mathrm{C},\mathrm{D}\right\}.$ Because we accommodate up to eight transmitters, there are in total 32 baseband signals $u_i^a\left(t\right),$ one for each antenna and transmitter device.

The baseband signal for transmitter device $i\in \left\{1,...,8\right\}$ is extracted around the tracking frequency ${\omega }_i\left(t\right)$ (Figs. 7C and 7D). The decimation filter of this down-conversion has a flat frequency response within a $\pm 100$kHz band. Given the decimation factor of 128, the complex baseband signals $u_i^a\left(t\right)$ have a sample rate of 781.25 kHz and a precision of 2 × 25 bits. Since the digital signals are complex valued, we interchangeably call them vectors and their phases we refer to as angles.

The tracking frequency ${\mathrm{\omega }}_i$ we set with PLLs, one PLL for each transmitter (channel) $i.$ A PLL generates an internal oscillatory signal of variable tracking frequency $\omega \left(t\right)$; it adjusts that frequency to maintain a zero phase with respect to the received (input) signal. As long as the zero-phase condition is fulfilled, the PLL’s tracking frequency ${\omega }_i\left(t\right)$ follows the instantaneous transmitter frequency ${\omega }_{T,i}\left(t\right)$ of the $i^{\mathrm{\prime }}$th transmitter. Each PLL generates its tracking frequency ${\mathrm{\omega }}_i$ by direct digital synthesis with a 48-bit phase accumulation register and a lookup table.

A PLL is driven by the main vector $u_i^{\mathrm{M}}\left(t\right)=u_i^{\mathrm{A}}\left(t\right)+u_i^{\mathrm{B}}\left(t\right)+u_i^{\mathrm{C}}\left(t\right)+u_i^{\mathrm{D}}\left(t\right)$ that we formed as the sum of the four baseband vectors. The error signal used in the PLL’s feedback controller is given by the phase of the main vector ${\theta }_i\left(t\right)=\mathrm{a}\mathrm{r}\mathrm{g}\left(u_i^{\mathrm{M}}\left(t\right)\right)$ (Fig. 7D).

The tracking frequency ${\mathrm{\omega }}_i$ is dynamically adjusted to keep the phase of the main vector close to zero. We implemented the calculation of the angle ${\theta }_i=\mathrm{a}\mathrm{r}\mathrm{g}\left(u_i^{\mathrm{M}}\right)$ of the main vector with the CORDIC algorithm (Volder, 1959) and unwrapped the phase up to $\pm 128$ turns before using it as the error signal for a proportional-integral-derivative (PID) controller that adjusts the tracking frequency of the PLL.

The PID controller was implemented on the FPGA in single precision floating point (32-bit) arithmetic. The controller included a limiting range and an anti-windup mechanism (Astrom & Rundqwist, 1989). The unwrapping of the error phase was crucial for the PLL to quickly lock-in and to keep the lock even during large and fast frequency deviations of the transmitter. To tune the PID parameters, we measured the closed-loop transfer function of the PLL by adding a white-noise signal to the control signal (tracking frequency), and then we adjusted the PID parameters until we observed a closed-loop transfer function with low-pass characteristic. We achieved a closed loop bandwidth of about 30 kHz.

Stage 4: Phase compensation

To avoid destructive interference in the summation of the baseband vectors, we compensated their phases ${\alpha }_i^a\left(\mathrm{t}\right)=\measuredangle \left(u_i^{\mathrm{M}}\left(\mathrm{t}\right),u_i^a\left(\mathrm{t}\right)\right),a\in \left\{\mathrm{A},\mathrm{B},\mathrm{C},\mathrm{D}\right\}$ relative to the main vector. We introduced individual phase offsets (also called compensation phases) ${\mathrm{\Delta }\phi }_{\mathrm{i}}^a\left(t\right)$ that were set in feedback loops to drive the phases ${\alpha }_i^a\left(t\right)$ towards zero (Fig. 7E).

For a given PLL $i$, we compensated the relative phases under which a radio signal arrives at the four antennas $a\in \left\{\mathrm{A},\mathrm{B},\mathrm{C},\mathrm{D}\right\}$ to align the four baseband vectors $u_i^a$. The alignment was achieved by providing a phase offset ${\mathrm{\Delta }\phi }_i^a$ to each down-converter, where it acts as offset to the phase accumulation register of the direct digital synthesis. To compute the angle of a baseband vector relative to the main vector, we rotated the baseband vectors $u_i^a$ by the phase ${\theta }_i$ of the main vector to result in the rotated vector $r_i^a=u_i^a{e}^{-i{\theta }_i}$. After averaging the rotated vectors across 512 samples, we computed its angle ${\mathrm{\alpha }}_i^a=\mathrm{a}\mathrm{r}\mathrm{g}(⟨r_i^a{⟩}_{512})$. We then compensated that angle by iteratively adding a fraction $\mathrm{\gamma }\in \left[0,1\right]$ of it to the phase offset: ${\mathrm{\Delta }\phi }_i^a:=\mathrm{\Delta }{\phi }_i^a+{\gamma \alpha }_i^a$. The parameter $\mathrm{\gamma }$ is the phase compensation gain (Fig. 7D), typically set to $\gamma =0.2$.

The PLL and phase compensation form independent control loops. When the PLL is unlocked (e.g., off), the tracking frequency does not match the instantaneous transmitter frequency and the baseband vectors rotate at the difference frequency (Fig. 7F, left). When the PLL is switched on and locked, the baseband vectors do not rotate, and the main signal displays a phase ${\theta }_i\left(t\right)\simeq 0$ (Fig. 7F, middle). When the phase alignment is switched on, the baseband signals align, and their sum maximizes the magnitude of the main vector (Fig. 7E, right).

Because birds’ locomotion is slower than their rapid vocalization-induced vibratory signals, the phases ${\alpha }_i^a\left(t\right)$change more slowly than the tracking frequency ${\omega }_i\left(t\right)$ and therefore we updated phase offsets less often than the PLL’s tracking frequency at a rate of about 1.5 kHz (781.25 kHz/512).

Operation

The intermediate band of $z^a\left(\omega \right)$is wide enough (80 MHz) to accommodate up to eight FM transmitters. Provided the transmitters’ FM carrier frequencies are roughly evenly spaced, the transmitter frequencies do not cross, even during very large frequency excursions. Nevertheless, we limited the tracking frequency ${\mathrm{\omega }}_i$ of channel $i$ to the range $\left[{\mathrm{\Omega }}_i-\mathrm{\Delta }\mathrm{\Omega },{\mathrm{\Omega }}_i+\mathrm{\Delta }\mathrm{\Omega }\right]$, where ${\mathrm{\Omega }}_i$ is the center frequency and $\mathrm{\Delta }\mathrm{\Omega }=1$ MHz is the common limiting range of all channels (Fig. 7B). The center frequency ${\mathrm{\Omega }}_i$ of a channel we manually set at the beginning of an experiment to the associated FM carrier frequency. The limiting range of 1 MHz we found narrow enough for the PLL to rapidly relock after brief signal losses. However, when the PLL loses tracking, the tracking frequency ${\mathrm{\omega }}_i\left(t\right)$ always stuck at one of the two limits of its range. Therefore, we detected tracking loss events heuristically as the time points $t$ that satisfied $\left|{\omega }_i\left(t\right)-{\mathrm{\Omega }}_i\right|>=\mathrm{\Delta }\mathrm{\Omega }$.

Synchronization

To ensure that the data streams from the video cameras, microphones, and transmitter devices are synchronized, we recorded videos with industrial cameras and digitized sounds on a National Instruments data acquisition board, both of which received sample trigger signals from the USRP. The base clock of the USRP was 200 MHz and was divided by 2¹³ to generate the audio sample rate of 24.414 kHz, and with a further division by 2⁹, we generated the 47.684 Hz video frame rate.

Central control software

On the host computer we run our central control software (BirdRadio programmed with LabView) that acquires the microphone and transmitter signals and writes them to a TDMS file (Fig. 1C). BirdRadio also logs both power and variance trajectories of the baseband signals $u^a\left(t\right)$ and $r^a\left(t\right)$ associated witch a given transmitter, which allows us to evaluate the radio signal-to-noise ratio $RSN{R}^a$ and $RSN{R}^{\mathrm{M}}$ of both single-antenna signals ( $a)$ and multi-antenna ( $M$) signals on the same dataset (see Section Radio Signal-to-Noise Ratio).

Furthermore, BirdRadio sends user datagram protocol (UDP) control signals to BirdVideo, it automatically starts the recording in the morning and stops it in the evening, it controls the light dimming in the sound-isolation chamber with simulation of sunrise and sunset, it triggers an email alarm when the radio signal from a transmitter device is lost, and it automatically adjusts the center frequency of each radio channel every morning to adjust for carrier frequency drift.

Data management

The BirdPark is designed for continuous recordings over multiple months, producing data at a rate of 60 GB/day for two birds and 130 GB/day for eight birds. On a subset of the data analyzed here we implemented the FAIR (findable, accessible, interoperable, and reusable) principles of scientific data management (Wilkinson et al., 2016) as follows:

Our recording software splits the data into gapless files of 20,000 video frames (ca. 7 min duration). At the end of a recording day, all files are processed by a data compilation script that converts the TDMS files into HDF5 files and augments them with rich metadata. The HDF5 files are self-descriptive in that they contain metadata as attributes and additionally, every dataset in the file contains its description as an attribute. We use the lossless compression feature of HDF5 to obtain a compression ratio of typically 2.5 for the audio and accelerometer data. The script also adds two microphone channels to the video files. Although this step introduces redundancy, the availability of sound in the video files is very useful during manual annotation of the videos. Furthermore, the script also exports the metadata as a JSON file and copies the processed data onto a network attached storage (NAS) server. At the end of an experiment, the metadata is uploaded onto our internal openBIS (Bauch et al., 2011) server and is linked with the datafiles on the NAS.

Sound attenuation

The isolation chamber attenuates sounds by 30 dB, which we measured by playing a 3-s continuous white noise stimulus on a loudspeaker inside the chamber and by recording it both inside and outside the chamber. We defined the sound attenuation as the difference in log-RMS amplitude in the 400 Hz to 5 kHz frequency band (corresponding roughly to the vocal range of zebra finches).

The background acoustic noise level inside the chamber is roughly 37 dBA (measured with a Voltcraft SL-10 sound-level meter). The acoustic reverberation time of the chamber inside is 36 ms: First we measured the impulse response function by playing a white noise signal on a loudspeaker and recording the sound with a microphone, both inside the chamber and then correlating the two signals. We estimated the reverberation time as the exponential decay time of the squared impulse response function over a decay range of 60 dB.

Transmitter properties

We found that the carrier frequency ${\omega }_c$ of FM transmitters depends on temperature at an average of −73 kHz/°C (range −67 to −87 kHz/°C, n = 3 transmitters). Towards the end of the battery life (over the course of the last 3 days), we observed an increase of ${\omega }_c$ by an average of 500–800 kHz. These slow drifts can easily be accounted for by tracking and high-pass filtering the momentary frequency. The measured end-to-end sensitivity of the frequency modulation is 5 kHz/g, with g being the gravitational acceleration constant.

Radio signal-to-noise ratio (RSNR)

While animal carried an operational transmitter, we calculated the radio signal-to-noise ratio $RSN{R}^a\left(t_k\right)$ of the signal $r^a\left(t\right)$ with $a\in \left\{\mathrm{A},\mathrm{B},\mathrm{C},\mathrm{D},\mathrm{M}\right\}$ in the (non-overlapping) $k$-th 21-ms radio frame $F_k$ defined by its center time $t_k$ as a function of the signal power (at zero frequency): $p^a\left(t_k\right)={|⟨r^a{\left(t\right)⟩}_{\left(t\in {\mathrm{F}}_k\right)}|}^2$ (where averages $⟨\cdot ⟩$ are calculated over all samples in a radio frame and the noise variance $v^a\left(t_k\right)={⟨{\left|r^a\left(t\right)\right|}^2⟩}_{\left(t\in {\mathrm{F}}_k\right)}-{|⟨r^a\left(t\right){⟩}_{\left(t\in {\mathrm{F}}_k\right)}|}^2$:

$$RSN{R}^a\left(t_k\right)=10\mathrm{l}\mathrm{o}{\mathrm{g}}_{10}\left[{\displaystyle \frac{p^{\mathrm{a}}\left(t_k\right)}{v^a}}\right].$$

To obtain smooth estimates of the variance $v^a\left(t_k\right)$, we estimated the latter as the median value $v^a$ across all n = 20,000 radio frames per 7-min long file. In Fig. 3A, we compared the multi-antenna $RSN{R}^{\mathrm{M}}\left(t_k\right)$ to the $RSN{R}^{\ast }\left(t_k\right)$ of the best single antenna defined as:

$$RSN{R}^{\ast }\left(t_k\right)=\underset{a\in \left\{\mathrm{A},\mathrm{B},\mathrm{C},\mathrm{D}\right\}}{max}⟨RSN{R}^a\left(t_k\right){⟩}_k.$$

In Fig. 3A, the $RSN{R}^{\mathrm{M}}$ measurements are roughly constant (max. deviation of RSNR of 1 dB) in each of the examples shown on top. We estimate the noise power $P_{\mathrm{N}}$ of the (high-pass filtered) demodulated signal as the power spectral density (PSD) averaged over periods without vocalization and integrated from 0 to 8 kHz (Fig. 3A). In Fig. 3B, we plot $P_{\mathrm{N}}$ against $RSN{R}^{\mathrm{M}}$ on the noise segments in Fig. 3A and on additional n = 216 noise segments within periods in which $RSN{R}^{\mathrm{M}}$ was roughly constant; these noise segments were selected as follows from a large dataset. First, we selected segments in which $RSN{R}^{\mathrm{M}}$ was inside intervals $\left[kdB,\left(k+1\right)dB\right],k\in \left[1,\dots ,50\right]$ for at least 100 ms. Second, segments that contained apparent signal components generated by the bird were removed by manual inspection of the spectrogram.

In Fig. 3C, we compare signal-to-noise ratios of multi vs single-antenna demodulation. In theory, when the PLL locks (when phases are aligned and $r^a\left(t\right)$ equals $u^a\left(t\right)$) and assuming independence of radio amplifier noises, the noise power (variance) of the phase-compensated and summed signal $u^{\mathrm{M}}\left(t\right)$ is four times larger than that of a baseband signal $u^a\left(t\right)$ for a given antenna and the signal power of $u^{\mathrm{M}}\left(t\right)$ is 16 times larger than that of a single antenna signal $u^a\left(t\right)$. Taken together, we expect multi-antenna demodulation to increase the $RSNR$ by 6 dB.

Detection of radio frequency jumps

We detected jumps of the transmitter tracking frequency ${\mathrm{\omega }}_{\mathrm{i}}$ as frequency deviations larger than 50 kHz from the running median, computed within 0.2 s sliding windows. Frequency jumps were concatenated when they were separated by less than 50 ms. The modulation (or jump) amplitude was defined as the maximum absolute deviation.

Detection of wing flaps and video alignment

The transmitter signals evoked by proximity effects during wing flap movement were downward and dip-like (Fig. 6). We detected dips on the male’s transmitter signal by thresholding its derivative after down-sampling by a factor of 32. A wing flap event was recognized when the derivative initially falls below a negative threshold, $-W$ = −60 MHz/s, and subsequently rose above a positive threshold, $W$ within a 2–12 ms window following the initial threshold crossing. The detected dips corresponded closely with the center time point of the wing’s down stroke (Fig. 6B).

We time-stamped the video frames as follows. For each detected wing-flap event, we examined the transmitter’s signal range. Events associated with a signal range less than 200 kHz we discarded from further analysis, where we defined the signal range as the maximum minus the minimum transmitter signal in a time window [−10.5, 10.5] ms (corresponding to 1 video frame period) centered on the minimum of the signal dip. Similarly, we excluded abrupt signal changes exceeding 1,000 kHz.

The video frame with exposure onset nearest to the dip minimum we defined as the first event frame, and the subsequent frame as the second event frame. We then ordered all event frames according to their time stamp: the time lag between the exposure onset and the dip minimum.

By randomly sampling video frames in which the flapping birds were clearly visible and continuously flying (from n = 5 birds), we obtained the illustration of wing flapping shown in Fig. 6, nicely capturing the phases of the wing flap cycle. All the frames in Fig. 6 are derived from a comprehensive dataset of five birds, with each event capturing a wing flap from a randomly selected bird.

Manual voice activity detection

In each of two bird-pair experiments (copExpBP08 and copExpBP09), we manually segmented all vocalizations of both birds in a randomly chosen file of 7 min duration. In experiments with four birds (juvExpBP01) and eight birds (juvExpBP03), we segmented all bird vocalizations in the first 1 min each of seven randomly chosen files (to obtain more diversity). We excluded files with a signal loss rate above 0.1%. We high-pass filtered (with a passband frequency of 200 Hz) the raw demodulated transmitter signals and microphone signals and produced spectrograms (in windows of 384 samples and hop size of 96 samples) that we manually segmented using Raven Pro 1.662 or BpBrowser (our custom data annotation software). We performed three types of vocal segmentations in the following order:

• (1)

  Transmitter-based vocal segments: On all transmitter channels, we separately segmented all vocalizations, precisely annotating their onsets and offsets. In the released data sets, these segments are referred to as transmitter-based vocal segments. We visually checked for crosstalk (whether we saw a faint trace of a vocalization on another bird’s transmitter spectrogram). We excluded crosstalk segments from further analysis (except for the crosstalk statistics): If a segment was detected as crosstalk, we set the label Crosstalk to the transmitter channel of the bird that generated the crosstalk, otherwise we set the tag to ‘No’. When we were uncertain whether a sound segment was a vocalization or not, we also looked at spectrograms of microphone channels and listened to sound playbacks. If we remained uncertain, we tagged the segment with the label ‘Unsure’: we treated such segments as (non-vocal) noises and excluded them from further analysis (except for the uncertainty statistics).

• (2)

  Microphone-based vocal segments: We simultaneously visualized all microphone spectrograms (we ignored nest mics when the nest was not accessible to the birds) using the multi-channel view in Raven Pro (we ignored Mic7, which was located in the second nest that was not accessible to the birds). On those, we annotated each vocal segment on the first microphone channel on which it was visible (e.g., a syllable that is visible on all microphone channels is only annotated on Mic1). Overlapping vocalizations were annotated as a single vocal segment. When we were uncertain whether a sound segment was a vocalization or not, we also looked at spectrograms of transmitter channels and listened to sound playbacks. If we remained uncertain, the segment was tagged with the label ‘Unsure’, such segments were treated as (non-vocal) noises and were excluded from further analysis.

• (3)

  Consolidated vocal segments: All consistent (perfectly overlapping down to a temporal resolution of one spectrogram bin) transmitter- and microphone-based vocal segments, we labelled as consolidated vocal segments. We then inspected all inconsistent (not perfectly overlapping) segments by visualizing all channel spectrograms. We fixed inconsistencies that were caused by human annotation errors (e.g., lack of attention) by fixing the erroneous or missing transmitter- and microphone-based segments. From the inconsistent (partially overlapping) segments that were not caused by human error, we generated one or several consolidated segments by trusting the modality that more clearly revealed the presence of a vocalization (hence our reporting of ‘misses’ in Table 2).

In our released annotation table, we give each consolidated vocal segment a Bird Tag (e.g., either ‘b15p5_m’ or ‘b14p4_f’) that identifies the bird that produced the vocalization, a Transmitter Tag that identifies the transmitter channel on which the vocalization was identified (either ‘b15p5_m’ or ‘b14p4_f’ or ‘None’), and a FirstMic Tag that identifies the first microphone channel on which the segment was visible (‘Mic1’ to ‘Mic6’, or ‘None’). We resolved inconsistencies and chose these tags as follows:

• If a microphone (-based vocal) segment was paired (partially overlapping) with exactly one transmitter segment, a consolidated segment was generated with the onset time set to the minimum onset time and the offset time set to the maximum offset time of the segment pair. The Bird and Transmitter Tags were set to the transmitter channel name, and the FirstMic Tag was set to the microphone channel name.

• If a transmitter segment was unpaired, a consolidated segment was created with the same onset and offset times. The Bird and TrCh Tags were set to the transmitter channel name, and the FirstMic Tag was set to ‘None’.

• If a microphone segment was unpaired, we tried to guess a transmitter channel name based on the vocal repertoire and noise levels on both transmitter channels. We visually verified that the vocal segment was not the result of multiple overlapping vocalizations (which was never the case). Then we created a consolidated segment with the same on- and offset, set the FirstMic Tag to the microphone number, the Bird Tag to the guessed transmitter channel name or ‘None’ if we were unable to make a guess, and the Transmitter Tag to ‘None’.

• If a microphone segment was paired with more than one transmitter segment, a consolidated segment was created for each of the transmitter segments. The onsets and offsets were manually set based on visual inspection of all spectrograms. Bird and Transmitter Tags were set to the transmitter channel name and the FirstMic Tag was set to the first microphone channel on which the respective segment was visible.

• We never encountered the case where a transmitter segment was paired with multiple microphone segments.

Vocal statistics

The statistics in Table 2 were calculated as follows:

# vocalizations = (number of consolidated segments)

# voc. missed on tr. channel = (number of consolidated segments with Transmitter=None)

# voc. missed on all mic. channels = (number of consolidated segments with FirstMic=None)

# voc. missed on Mic1 = (number of consolidated segments with FirstMic≠Mic1 and FirstMic≠None)

# voc. unassigned = (number of consolidated segments with Bird=None)

# voc. with overlaps = (number of consolidated segments which overlap with at least one other syllable by at least one spectrogram bin)

# voc. with crosstalk = (number of transmitter-based segments with Crosstalk)

# uncertain segments = (number of transmitter-based and microphone-based segments with ‘Unsure’ tag)