Cephalopod species identification using integrated analysis of machine learning and deep learning approaches

Sample collection

Sampling trips were conducted from November 2017 to January 2018. A total of 174 cephalopod samples were collected from two major fisheries landing sites located on the west coast of Peninsular Malaysia, namely Hutan Melintang, Perak and Pasir Penambang, Kuala Selangor. The specimens were selected to represent the diversity of distinctive morphological groups and size classes available during sampling. Seven putative cephalopod species were selected including four putative species for squid (n = 96), two for cuttlefish (n = 49) and one for octopus (n = 29). Samples collected were kept on ice in the field and frozen immediately upon arrival in the laboratory.

Species identification and confirmation

After defrosting, the specimens were measured for dorsal mantle length (mm) and wet body mass (g) and were photographed (see steps described below). Initial species identification using morphological characters were conducted using available taxonomic references (Reid, Jereb & Roper, 2005; Jereb et al., 2016) and were cross-checked against current species checklists in reputable databases, such as the Malaysia Biodiversity Information System (Malaysia Biodiversity Information System, 2020), the World Register of Marine Species (MolluscaBase, 2020), and SeaLifeBase (Palomares & Pauly. Editors, 2020). The upper and lower beaks were then extracted from the buccal mass and preserved separately in labelled jars containing 80% alcohol.

Species identification from morphological characteristics was subsequently confirmed with molecular approaches using the mitochondrial 16S rRNA gene which was amplified with the universal primers 16Sar and 16Sbr (Simon et al., 1994).

Software and hardware

The hardware used for the image acquisition included a lightbox, a laptop (Intel i7 with 4 GB RAM), and a smartphone (16MP camera with 1440 ×250 pixels). The Adobe Photoshop CS6 (version 13.1.2) and Python 3 software were used in the feature extraction and identification step.

Image acquisition

The images of both upper and lower beaks were captured using the Samsung Galaxy Note 4 smartphone camera. Samples were placed against a white background, i.e., lightbox with white light, and were centred on the camera screen to ensure high-quality images. A distance of 10 cm between the smartphone camera and the beak samples was fixed and no zooming was used for all specimens photographed (Fig. 2). Only the left lateral views of the upper and lower beaks were captured. All beak images were stored using the JPEG format.

Image processing and feature extraction

A digital image has to be processed before performing feature extraction and computational analysis to enhance the image, remove the noises, minimize the amount of storage needed and to increase the computation performance. Firstly, unwanted parts in the images were cropped. Next, all the beak images were downscaled to 10% of its original size to eliminate unnecessary pixel information without compromising the quality of the images. The original image resolution was 5312px ×2988px while the rescaled image resolution was approximately 532px ×299px. Gaussian smoothing was then carried out on the rescaled images to reduce the noises on the images.

Image segmentation was performed to convert the rescaled images into red-green-blue (RGB) colour, grayscale and binary images to obtain the region of interest (ROI), which is the beak. Firstly, the colour space of the image was converted to the common RGB colour space. The colour image was then converted to a grayscale image by using Eq. (1) (OpenCV, 2020).

(1)${\displaystyle Y=0.299\left(R\right)+0.587\left(G\right)+0.114\left(B\right).}$

A thresholding algorithm was used to convert the grayscale image into a binary image by classifying all the pixels into two groups using a threshold value set at 240. Those below the threshold value (0–239) formed the image background and those above the threshold (240–255) formed the object of interest, i.e., the beak. The boundary of the beak (ROI) was then determined, outlined and extracted from the image.

Next, HOG feature descriptor was calculated by determining the occurrences of gradients and orientation in localized portions of each image, i.e., an image is broken down into small regions and the gradients and orientation were calculated for each region. The magnitude of the gradient increases wherever there is a sharp change in intensity (brightness) of the image. The magnitude of the gradient, G was computed using Eq. (2) (Mallick, 2016). (2)${\displaystyle \left|\mathrm{G}\right|=\sqrt{{\mathrm{G}}_{\mathrm{x}}^2+{\mathrm{G}}_{\mathrm{y}}^2}.}$ The gradient orientation, was computed using Eq. (3) where G_x is horizontal gradient in the X direction and G_y is vertical gradient in the Y direction (Mallick, 2016). (3)${\displaystyle \Theta =\arctan \frac{{\mathrm{G}}_{\mathrm{x}}}{{\mathrm{G}}_{\mathrm{y}}}.}$ A single image was divided into several cells, where each cell formed a square region which contained 128 ×128 pixels each. Using calculated gradient magnitude and orientation, every pixel within a cell casted a weighted vote for an orientation based histogram. The orientation bin was set as nine and this generated a frequency table for nine angles, i.e., 0, 20, 40, 60, 80, 100, 120, 140 and 160 (Singh, 2019). Using the set number of bins, a 9 ×1 matrix histogram was counted for each cell. Next, 2 ×2 cells were grouped to form a block, resulting in a total of 3 (3 ×1) blocks of 128 ×128 pixels for each image. Each block contained four 9 ×1 or a single 36 ×1 matrix and a total of 108 (3 × 36 ×1) features were produced for each image. In this manner, HOG was used in extracting the colour features from grayscale and RGB images and converted them into feature datasets.

The MSD consisted of both geometrical and morphological features (Aakif & Faisal Khan, 2015) and extracted the shape features from the highlighted region in the binary images. The binary image is used commonly due to clear delineation of the boundary of the object. Firstly, the image was computed to determine the contour of the object. Then, all the noises were eliminated by adjusting the threshold. After the contour was shown, ten morphological features, including beak size, were extracted from each image (Table 1).

In addition to the traditional methods, three deep learning CNN models, namely VGG19, InceptionV3 and ResNet50, were used to automatically extract features from the images. The VGG19 model consists of 3 ×3 convolutional layers. The original beak images were firstly resized to 224 ×224 pixels of RGB images as the inputs for VGG19. The VGG19 model pre-processed the images by subtracting the mean RGB value from each pixel of the image. Next, the max-pooling layer was used to reduce the feature dimensionality and two fully connected layers were used to produce the feature vector, with the presence of 4,096 neurons (Mateen et al., 2019).

The InceptionV3 model requires the input RGB image size of 299 ×299 pixels. Unlike the fixed kernel size for VGG19, the InceptionV3 allowed the features of the images to vary within an image frame and include multiple sizes of kernels in the same layer. There are four operations constructed in parallel, including a 1 ×1 convolutional layer for depth reduction, a 3 ×3 convolutional layer for capturing distributed features, a 5 ×5 convolutional layer for capturing the global features and a max-pooling layer for capturing the low-level features. Each layer is responsible to extract the deep features from the images, concatenate and pass them to the next layer (Anwar, 2019).

The ResNet50 model requires the input RGB image size of 224 ×224 pixels. ResNet50 is composed of 48 convolutional layers with 7 ×7 and 1 ×1 kernel size, max pooling layer and the fully connected layer. One of the important features of ResNet50 is the shortcut connections, that skip one or more layers, in order to solve the problem of vanishing gradient in deep neural networks by allowing the gradient to flow through the layer (Wen, Li & Gao, 2020).

Machine learning identification

Machine learning techniques are capable of analysing weighty amounts of image data accurately and successfully. The supervised machine learning methods used for the cephalopod classification problem were Artificial Neural Network (ANN), Support Vector Machine (SVM), Random Forest (RF), Decision Tree (DT), k-Nearest Neighbours (kNN), Logistic Regression (LR), Linear Discriminant Analysis (LDA), and Gaussian Naïve Bayes (GNB). Table 2 shows the list of parameters adjusted for each classifier.

The ANN method is a machine learning technique that processes information by adopting the way neurons of human brains work and consists of a set of nodes that imitates the neuron and carries activation signals of different strengths (Daliakopoulos, Coulibaly & K. Tsanis, 2005).

The SVM method is another popular algorithm in solving classification problems with limited amount of data. First, each sample data is plotted as a point in n-dimensional spaces, where n is the number of features obtained from the image, also known as the support vectors. The SVM algorithm finds the best-fit hyperplane that maximizes the margin between the nearest support vectors of both classes with the hyperplane chosen (Yu & Kim, 2012).

A decision tree (DT) model is constructed for each feature selected. It starts from the root node and splits at the leaf nodes. Each leaf node determines the best split approach while the Gini impurity function is used to measure the quality of the split. The final leaf node shows the final prediction of each DT (Fan, Ong & Koh, 2006).

The RF method is a meta-estimator that fits several decision trees (DTs). The RF model is constructed and trained by the bagging method (decision tree) while the result is based on majority voting. All the predictions resulting from each DT are voted and the majority is the final output of the RF model (Svetnik et al., 2003).

For the kNN model, each data point is coordinated in the n-dimensional space while an unknown sample is introduced, the distance between the unknown sample with each data point is calculated based on the Euclidean distance matrix (Alimjan et al., 2018).

The multinomial LR model has two or more discrete outcomes where the most frequent species in the dataset was chosen as the reference category while the probability of other categories was compared against the reference category. This resulted in n–1 binary regression models, where n = number of species in the classification problem. Prediction of the LR model is based on the Maximum Likelihood Estimation (MLE) approach, where the MLE determines the mean and variance that best describe the sample (Brownlee, 2019).

The LDA algorithm assumes that each feature has the same variance and calculates the between-class and within-class variance (Balakrishnama & Ganapathiraju, 1998). The LDA approach maximizes the between-class variance and minimizes the within-class variance. Once the LDA model is trained, the probability of an unknown sample is calculated by using the Bayes Theorem (Hamsici & Martinez, 2008). The result of the prediction is chosen based on the highest probability of the species.

The normal distribution method is also applied in the GNB model that estimates the mean and standard deviation of each species from the training data given. The Bayes Theorem is applied to calculate the probabilities of each species. Species with the highest probability matched will be selected (Pattekari & Parveen, 2012).

The features extracted from each feature extraction methods were used as the input datasets in the machine learning identification step. Each dataset was split into 80% of training and 20% of the testing set. The training set was used to trained all eight models by minimizing the error formed while the model performance was evaluated using the testing dataset. Five-fold cross-validation (CV) stratified shuffle split method was used and tested for 10 times to avoid any overfitting. A significant feature of stratified shuffle split is the ability to preserve the ratio of the training and testing set in the dataset. Figure 3 shows an example of sample splitting for one of the ANN models.

Performance evaluation

The performance of the classification model was evaluated through the confusion matrix, i.e.,a table that displays the number of true positives (TP), true negatives (TN), false positives (FP) and false negatives (FN). The testing accuracy, precision and recall were calculated using Eqs. (7) to (9) respectively. (7)${\displaystyle \mathrm{Testing accuracy}=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}}$ (8)${\displaystyle \mathrm{Precision}=\frac{TP}{TP+FP}}$ (9)${\displaystyle \mathrm{Recall}=\frac{TP}{TP+FN}}$

In addition, the performance of each classifier model was visualized from the area under the Precision-Recall (PR) curve (AUC), where the precision value was plotted on the y-axis while the recall value at the x-axis (Narkhede, 2018). The precision and recall values were computed from the testing set for each cephalopod species and the average AUC of each model was calculated. The higher the AUC, the better the model performance in identifying cephalopod species from the beak images. According to Saito & Rehmsmeier (2015), the PR curve is more informative and reliable than the Receiver Operating Characteristic (ROC) curve for imbalanced datasets.

Data Collection and Feature Extraction

Seven species of cephalopod were morphologically identified (Table 3) and confirmed through 16S rRNA sequencing (Table 4). Out of these, two species were cuttlefish (Sepia aculeata, and Sepia esculenta), four were squids (Sepioteuthis lessoniana, Loliolus uyii, Uroteuthis chinensis and Uroteuthis edulis) and one was an octopus (Amphioctopus aegina).

A range of 10 to 108 features such as shape and colour features were extracted from the left lateral beak images using HOG and MSD descriptors. Using deep learning methods of VGG19, ResNet50 and InceptionV3, 2048 to 4096 features were extracted. Table 5 lists the number of traditional and deep features extracted by each descriptor.

Traditional features

Firstly, the extracted features were tested individually as a single descriptor of grey HOG, colour HOG and MSD. Each of the descriptors was fit into eight different classifiers to test for the model performance in identifying the cephalopod species. Five-fold cross-validation with 80%–20% stratified shuffle splitting was used to avoid overfitting. Each test was continuously run for 10 times to achieve more reliable results. The testing results were averaged and the PR curves were plotted based on the descriptors used. Table 6 shows the average testing accuracy of each classifier with traditional descriptors for the upper and lower beak images. Grey HOG descriptors achieved the highest testing accuracy of 55.43% (± 0.14) for the kNN model using upper beak images while HOG descriptors with ANN achieved the best accuracy of 68.69% (± 0.12) with lower beak images. MSD descriptors with GNB achieved the best accuracy at 64.00% (± 0.11) using the upper beak images.

Next, the descriptors were combined as the hybrid descriptors which are (grey HOG + MSD) and (colour HOG + MSD) and tested. Table 6 shows the average testing accuracy of each classifier with hybrid descriptors. The hybrid descriptors of (Grey HOG + MSD) and (colour HOG + MSD) with ANN had the best testing accuracy at 61.09% (± 0.14) for the upper beak image and 73.09% (± 0.12) for lower beak images. An improvement in the testing accuracy was observed in comparison with the single descriptor with the increment from 68.69% to 73.09% for the ANN model with lower beak images. Figure 4A shows the confusion matrix from one of the runs of the ANN model for hybrid descriptor (colour HOG + MSD) and lower beak image. The ANN model appeared to classify the test samples of S. esculenta and L. uyii perfectly. The performance of the ANN model can be observed through the precision recall-curve (Fig. 4B).

Deep features

Both upper and lower beak images were used as inputs into CNN models to extract the deep features. Table 7 shows the average testing accuracy of each classifier with deep features extracted. All CNN models achieved good results with the best result of 91.14% (± 0.09), 87.54% (± 0.10), and 86.86% (± 0.09) for VGG19, InceptionV3 and ResNet50 respectively. Figure 5A shows the confusion matrix from one of the runs of the ANN model for deep features extracted from the VGG19 with lower beak images. The ANN model was shown to classify most of the species perfectly, except for S. esculenta and L. uyii. The performance of the ANN model can be observed through the precision recall-curve (Fig. 5B).