Fully-automated identification of fish species based on otolith contour: using short-time Fourier transform and discriminant analysis (STFT-DA)

Preprocessing

Discrimination among different fish species was based on the 1D representation of the otolith outline. Firstly, the external outline of the surface contours of the otolith had to be extracted and then, distances between the center of gravity and the contour points had to be calculated. For this purpose, the grayscale image of the otolith was converted to the binary image with the threshold value of 0.1. Choice of this threshold value (0.1) resulted in obtaining the binary images for the otoliths with a wide range of transparency. After clearing the borders and filling the holes, the small objects (objects that had fewer than 50,000 pixels) were removed from the binary images. Then, coordinates of the boundary (outline) pixels as well as the center of gravity were calculated. By having these coordinates, characteristic 1D signals, which are the distances between the boundary pixels and center of gravity as a function of the corresponding angles, were determined. Figure 2 shows an image of the otolith with its representative 1D signal.

Feature extraction

1D spatial-domain signals obtained from the previous stage were down-sampled to 1,000 points (samples) by interpolation using fast Fourier transform (FFT). In this study, short-time Fourier transform (STFT) was applied as a feature extraction method on the resampled signals. STFT of the original (1D) signals were determined by using Gaussian window function. Repeated trials of many combinations of two parameters, the number of points of the window function and the number of overlapped samples, were made to achieve the best classification result. The best match of 100 points of the window and 40 overlapped samples resulted in the division of each signal into 16 segments. The type of windowing function also affected the performance of the identification system. To explore this effect, results obtained using different windowing techniques were compared in the next section. Figure 3 shows the spectrogram (using STFT with the sampling frequency (f_s) of 2π) obtained from the 1D spatial-domain signal illustrated in Fig. 2. The color bar in Fig. 3 indicates the power spectral density (PSD) estimate of each segment. Each segment of the original signal consisted of 129 frequency components. Absolute values and phase angles of the frequency components of each segment were determined and then standardized by calculating the corresponding z-scores (Z_ABSs: z-scores of the absolute values and Z_ANGs: z-scores of the angles). In each segment of the signal, two important parameters were determined: maximum of the Z_ABSs (MAX_ABS) and maximum of Z_ANGs (MAX_ANG). Having 16 segments in each signal, 32 attributes (16 MAX_ABS + 16 MAX_ANG) were extracted from each representative signal. In this way, each otolith image was converted to a 32-element vector in which the first 16 elements were MAX_ABS values and the rest were the values of MAX_ANG. The contribution of each feature type (absolute and angle) to the performance of the model was also explored and the obtained results are demonstrated in the next section.

Classification

The characteristic vectors obtained from the previous stage were utilized as inputs to the Discriminant Analysis (DA) classifier in order to train and test the identification system. Fourteen species from three different families were used in this study (Table 1). All otoliths were extracted from fish obtained from fish landing sites or the wet markets. No ethics clearance was required from the University of Malaya—Institutional Animal Care and Use Committee (UM-IACUC).

Engraulidae family

Three species namely Coilia dussumieri, Setipinna taty and Thryssa hamiltonii from the Engraulidae family were used in this study. From each species, 20 specimens (otolith images) were used for training the model. Then, the trained model was tested with 10 specimens per species (total of 30 images for testing the model). Table 2 demonstrates the confusion matrix obtained from the predicted species in this family.

All of the 10 specimens from the Coilia dussumieri and Setipinna taty species were classified correctly. For the Thryssa hamiltonii species, one specimen was misclassified as the Setipinna taty species. Overall, 29 out of 30 specimens from the Engraulidae family (∼97%) were correctly predicted as the target species.

Sciaenidae family

Five species of the Sciaenidae family were also used to evaluate performance of the identification system. In this family, 19 specimens per species (total number of 95 specimens) were used to train the system, and then the trained model was tested with 50 specimens (10 specimens per species). The predicted results of this family are presented in Table 3. Among five species in this family, three species (Johnius belangerii, Johnius carouna and Panna microdon) were identified with 100% accuracy. Two other species (Dendrophysa russelli and Otolithes ruber) had one misclassified specimen each. In this family, similar to the Engraulidae family, there was no species with classification accuracy of less than 90%. The proposed model identified five species of the Sciaenidae family with an overall accuracy of 96%.

Ariidae family

Six species from the Ariidae family were also used in this study. The number of specimens per species for training and testing the model were 18 and 10, respectively. The classification results obtained from this family are shown in Table 4. Overall accuracy of the model in this family was ∼93% which is slightly less than the other two families. The lowest classification accuracy (80%) in this family was for the Nemapteryx caelatus. Two specimens of the Nemapteryx caelatus species were predicted as the Cryptarius truncatus. Three species namely Arius maculatus, Hexanematichtys sagor and Plicofollis argyropleuron had 100% correct prediction results. The accuracy of the model for the Cryptarius truncatus and Osteogeneiosus militaris species was 90%. Both of these species had one specimen that was misclassified as Nemapteryx caelatus.

All three families

To test the model with more species, all three families were combined (total number of 14 species) and the results of the classification are demonstrated in Table 5. From each species, 18 and 10 specimens were used to train and test the model, respectively (total numbers of 252 images for the training and 140 images for the testing). All 14 species were predicted by the proposed model with an overall accuracy of ∾92%. Eight of these species, three from the Sciaenidae, three from the Ariidae, and two from the Engraulidae family, were classified with the accuracy of 100%. Three species showed the identification accuracy of less than 90% (Dendrophysa russelli: 80%, Nemapteryx caelatus: 70%, and Cryptarius truncatus: 70%). Nemapteryx caelatus and Cryptarius truncates, both from the Ariidae family, had the most numbers of misclassified specimens among the 14 species used in this study. The classification accuracy for Otolithes ruber, Osteogeneiosus militaris, and Setipinna taty was 90%. It is worth-noting that there was no cross-family misclassification for all six species that had at least one misclassified specimen (all six species had specimens correctly classified in their families). As a result, developing a model that first identifies the family and then species cannot lead to an improvement in the overall accuracy of the system.

Contribution of MAX_ABS and MAX_ANG to the system performance

To explore the contribution of MAX_ABS and MAX_ANG to the performance of the system, each feature type was separately used to train and test the model. Table 6 shows the classification results obtained by using each feature type separately (16-element vector) and their combined features (32-element vector). For all four data sets used in this study, the best identification result was achieved using the 32-element vector. For the Engraulidae, Sciaenidae and the combined families, using only the MAX_ANG resulted in higher accuracy compared to using only the MAX_ABS. However, the performance of the model was better for the Ariidae family by using the 16-element vector obtained from the absolute features (MAX_ABS). This result suggests that both phase and absolute features should be taken into account when the model is trained with different fish families.

Effect of the windowing function

As mentioned in the previous section, the windowing function used to calculate STFT of the representative signals could influence the performance of the model. To explore this effect, the identification system was trained and tested with several types of the window function. However, the number of points of window (100) and the number of overlapped points (40) were fixed for all types of window function tested. The overall accuracy obtained from three families, as well as the combined families, are compared and shown in Table 7.

Using the Gaussian window function led to the highest classification accuracy (97%) in the Engraulidae family. In the Sciaenidae family, the best result (96%) was achieved by using four functions namely Gaussian, Hamming, Kaiser, and Rectangular. The most accurate prediction (93%) in the Ariidae family was obtained by using the Gaussian function. In the combined families, using the Rectangular function resulted in the highest overall accuracy (94%). However, utilizing the Rectangular windowing function led to relatively poor performance of the model in the Engraulidae (87%) and Ariidae (83%) families. Taking into accounts all the results obtained using these 16 functions, the Gaussian window function was selected in this study due to its good performance in all the four data sets.