Species-specific audio detection: a comparison of three template-based detection algorithms using random forests

Template computation

The template refers to the combination of all ROIs in the training data. To create a template, we first start with the examples of the specific call of interest (i.e., ROIs) that were annotated from a set of recordings for a given species and a specific call type (e.g., common, alarm). Each ROI encompasses an example of the call, and is an instance of time between time t₁ and time t₂ of a given recording and low and high boundary frequencies of f₁ and f₂, where t₁ < t₂ and f₁ < f2. In a general sense, we combine these examples to produce a template of a specific song type of a single species.

Specifically, for each recording that has an annotated ROI, a spectrogram matrix (SM) is computed using the Short Time Fourier Transform with a frame size of 1024 samples, 512 samples of overlap and a Hann analysis window, thus the matrices have 512 rows. For a recording with a sampling rate of 44,100 Hz, the matrix bin bandwidth is approximately 43.06 Hz. The SM is arranged so that the row of index 0 represents the lowest frequency and the row with index 511 represents the highest frequency of the spectrum. Properly stated the columns c₁ to c₂ and the rows from r₁ to r₂ of SM were extracted, where: ${\displaystyle c_1=\lfloor t_1\times 44,100\rfloor ,c_2=\lfloor t_2\times 44,100\rfloor ,r_1=\lfloor f_1∕43.06\rfloor \text{and}{r}_2=\lfloor f_2∕43.06\rfloor .}$ The rows and columns that represent the ROI in the recording (between frequencies f₁ and f₂ and between times t₁ and t₂) are extracted. The submatrix of SM that contains only the area bounded by the ROI is define as SM_ROI and refer in the manuscript as the ROI matrix.

Since the ROI matrices can vary in size, to compute the aggregation from the ROI matrices we have to take into account the difference in the number of rows and columns of the matrices. All recordings have the same sampling rate, 44,100 Hz. Thus, the rows from different SMs, computed with the same parameters, will represent the same frequencies, i.e., rows with same indexes represent the same frequency. After the ROI matrix, SM_ROI, has been extracted from SM, the rows of SM_ROI will also represent specific frequencies. Thus, if we were to perform an element-wise matrix sum between two ROI matrices with potentially different number of rows, we should only sum rows that represent the same frequency.

To take into account the difference in the number of columns of the ROI matrices, we use the Frobenius norm to optimize the alignment of the smaller ROI matrices and perform element-wise sums between rows that represent the same frequency. We present that algorithm in the following section and a flow chart of the process in Fig. 4.

Template computation algorithm:

1. Generate the set of SM_ROI matrices by computing the short time Fourier Transform of all the user generated ROIs.

2. Create matrix SM_max, a duplicate of the first created matrix among the matrices with the largest number of columns.

3. Set c_max as the number of columns in SM_max.

4. Create matrix T_temp, with the same dimensions as SM_max and all entries equal to 0. This matrix will contain the element-wise addition of all the extracted SM_ROI matrices.

5. Create matrix W with the same dimensions of SM_max and all entries equal to 0. This matrix will hold the count on the number of SM_ROI matrices that participate in the calculation of each element of T_temp.

6. For each one of the SM_i ROI matrices in SM_ROI:

   1. If SM_i has the same number of columns as T_temp:

      1. Align the rows of SM_i and T_temp so they represent equivalent frequencies and perform an element-wise addition of the matrices and put the result in T_temp.

      2. Add one to all the elements of the W matrix where the previous addition participated.

   2. If the number of columns differs between SM_i and T_temp, then find the optimal alignment with SM_max as follows:

      1. Set c_i as the number of columns in SM_i.

      2. Define (SM_max)_I as the set of all submatrices of SM_max with the same dimensions as SM_i. Note that the cardinality of (SM_max)_I is c_max − c_i.

      3. For each Sub_k ∈ (SM_max)_I:

         1. Compute d_k = NORM(Sub_k − SM_i) where NORM is the Frobenius norm defined as: ${\displaystyle NORM\left(A\right)=\sqrt{\sum _{\left(i,j\right)}\left|a_{i,j}^2\right|}}$ where a_i,j are the elements of matrix A.

      4. Define Sub_min{dk} as the Sub_k matrix with the minimum d_k. This is the optimal alignment of SM_i with SM_max.

      5. Align the rows of Sub_min{dk} and T_temp so they represent equivalent frequencies, perform an element-wise addition of the matrices and put the result in T_temp.

      6. Add one to all the elements of the W matrix where the previous addition participated.

7. Define the matrix T_template as the element-wise division between the T_temp matrix and the W matrix.

The resulting T_template matrix summarizes the information available in the ROI matrices submitted by the user and it will be used to extract information from the audio recordings that are to be analyzed. In this article each species T_template was created using five ROIs.

In Fig. 5A a training set for the Eleutherodactylus coqui is presented and in Fig. 5B the resulting template can be seen. This tool is very useful because the user can see immediately the effect of adding or subtracting a specific sample to the training set.

Algorithm to Create the Similarity Vector:

1. Compute matrix SPEC, the submatrix of the spectrogram matrix that contains the frequencies in T_template. Note that we are dealing with recordings that have the same sample rate as the recordings used to compute the T_template.

2. Define c_SPEC, the number of columns of SPEC.

3. Define c_template, the number of columns of T_template. Note that c_SPEC ≫ c_template since the SPEC matrix have the same number of columns as the whole spectrogram and that the T_template matrix fits c = c_SPEC − c_template + 1 times inside the SPEC matrix. There are c submatrices of SPEC with the same dimensions as T_template.

4. Define step, the step factor by which T_template will progressed over the SPEC matrix.

5. Define $n=⌊\frac{c_{SPEC}-c_{template}}{step}⌋+1$. Note that if step = 1 then n = c. In this work, however, this parameter was selected as step = 16 as a trade-off for speed.¹

6. Define SPEC_i as the submatrix of SPEC that spans the columns from i × step to i × step + c_template.

7. Set i = 1.

8. While i ≤ n

   1. Compute the similarity measure meas_i for SPEC_i (the definition of meas_i for each of the three variants is provided in the following section).

   2. Increase i by 1. Note that this is equivalent to progressing step columns in the SPEC matrix.

9. Define the vector V as the vector containing the n similarity measures resulting from the previous steps. That is, V = [meas₁, meas₂, meas₃, …, meas_n].

Recognition function

We used three variations of a pattern match procedure to define the similarity measure vector V. First, the Structural Similarity Index described in Wang et al. (2004) and implemented in Van der Walt et al. (2014) as compare_ssim with the default window size of seven unless the generated pattern is smaller. It will be referred in the rest of the manuscript as the SSIM variant. For the SSIM variant we define meas_i as: ${\displaystyle mea{s}_i=SSI\left(T_{template},SPE{C}_i\right),}$ where SPEC_i is the submatrix of SPEC that spans the columns from i × step to i × step + c_template and the same number of rows as T_template and V = [meas₁, meas₂, meas₃, …, meas_n] with ${\displaystyle n=⌊\frac{c_{SPEC}-c_{template}}{step}⌋+1.}$

Second, the dynamic thresholding method (threshold_adaptive) described in Van der Walt et al. (2014) with a block size of 127 and an arithmetic mean filter is used over both T_template and SPEC_i before multiplying them and applying the Frobenius norm and normalized by the norm of a matrix with same dimensions as T_template and all elements equal to one. Therefore, meas_i for the NORM variant is defined as: ${\displaystyle mea{s}_i=FN\left(DTM\left(T_{template}\right).\ast DTM\left(SPE{C}_i\right)\right)∕FN\left(U\right),}$ where again SPEC_i is the submatrix of SPEC that spans the columns from i × step to i × step + c_template, FN is the Frobenius norm, DTM is the dynamic thresholding method, U is a matrix with same dimensions as T_template with all elements equal to one and .∗ performs an element-wise multiplication of the matrices. Again, V = [meas₁, meas₂, meas₃, …, meas_n] with ${\displaystyle n=⌊\frac{c_{SPEC}-c_{template}}{step}⌋+1.}$

Finally, for the CORR variation we first apply the OpenCV’s matchTemplate procedure (Bradski, 2000) with the Normalized Correlation Coefficient option to SPEC_i, the submatrix of SPEC that spans the columns from i × step to i × step + c_template. However, for this variant, SPEC_i includes two additions rows above and below, thus it is slightly larger than the T_template. With these we can define: ${\displaystyle mea{s}_{j,i}=CORR\left(T_{template},SPE{C}_{j,i}\right)}$ where SPEC_j,i is the submatrix of SPEC_i that starts at row j (note that there are five such SPEC_j,i matrices).

Now, we select five points at random from all the points above the 98.5 percentile of meas_j,i and apply the Structural Similarity Index to the 5 strongly-matching regions. The size of these regions is eight thirds (8∕3) of the length of T_template, 4∕3 before and 4∕3 after the strongly-matched point that was selected. Then, define FilterSPEC as the matrix that contains these 5 strongly-matching regions and FilterSPEC_i as the submatrix of FilterSPEC that spans the columns from i to i + c_template then, the similarity measure for this variant is define as: ${\displaystyle mea{s}_i=SSI\left(T_{template},FilterSPE{C}_i\right)}$ and the resulting vector V = [meas₁, meas₂, meas₃, …, meas_n] but this time with ${\displaystyle n=5\times \left(⌊8∕3\times c_{template}⌋+1\right).}$

It is important to note that no matter which variant is used to calculate the similarity measures, the result will always be a vector of measurements V. The idea is that the statistical properties of these computed recognition functions have enough information to distinguish between a recording that has the target species present and a recording that does not have the target species present. However, notice that since c_SPEC, the length of SPEC, is much larger that c_template the length of the vector V for the CORR variant is much smaller than the other two.

Random forest model creation

After calculating V for many recordings we can train a random forest model. First, we need a set of validated recordings with the specific species vocalization present in some recordings and absent in others. Then for each recording we compute a vector V_i as described in the previous section and extract the statistical features presented in Table 2. These statistical features represent the dataset used to train the random forest model, which will be used to detect recordings for presence or absence of a species call event. These 12 features along with the species presence information are used as input to a random forest classifier with a 1,000 trees.