Fishing operation type recognition based on multi-branch convolutional neural network using trajectory data

Geohash

The Geohash algorithm, which was proposed by Gustavo Niemeyer in 2008, is a geocoding method in the public domain (Yin & Chen, 2022; Zhou et al., 2021). What’s particularly elegant about this algorithm is that it follows the pattern of “left is 0, right is 1; bottom is 0, top is 1”, which is used to represent the area after subdivision, achieved by continuously and alternately dichotomizing the global longitude [−180°, 180°] [−180°, 180°] and latitude [−90°, 90°] ranges. In the Geohash approach, the corresponding binary value is recorded until the encoding length reaches the required accuracy. The longitude and latitude are then rearranged into a new binary string. Finally, the corresponding Geohash string is obtained using Base32, which encodes every five binary digits (Suwardi et al., 2015; Li et al., 2020). The alphabetically ordered Geohashes trace out a Z-order curve. As shown in Fig. 2, at the first level of Geohash encoding, different global ranges can be represented by separate 32 American Standard Code for Information Interchange (ASCII) characters. If the positions of the characters are connected with curves in the usual order, the curves appear Z-shaped. This is so called as a Z-order filling curve.

Thus, the Geohash algorithm converts the 2D longitude and latitude coordinates of a point into a 1D string to represent the corresponding geographical location (Zhou et al., 2022, 2020). The length of the Geohash string depends on the encoding accuracy; a longer string provides higher accuracy, and vice versa. A Geohash is generally encoded at Levels 1–12, corresponding to a grid range from 5,000 × 5,000 km² to 0.000000372 × 0.000000186 km² like Table 2. Each Geohash code represents a grid region on Earth and is the same for all coordinate points within that grid. As the Geohash string length increases, the extent of the grid it represents decreases (Irshaid et al., 2021; Jin et al., 2019; Guo et al., 2019).

Embedding

Embedding techniques were initially used in natural language processing tasks to convert string text into numeric vectors, primarily focusing on representation of words and documents (Mikolov et al., 2013; Zhao et al., 2020; Sonbol, Rebdawi & Ghneim, 2022). This method can be used to convert a word into a vector; that is, the text is converted into a fixed-length vector of consecutive real numbers. Words in a vocabulary are mapped to a potential vector space that summarizes their syntactic and semantic information. The basic assumption of the embedding technique is that words with the same meaning in a text should have similar representations and, thus, embedding vectors that are closer in the vector space are more similar in textual meaning. Therefore, the embedding representation can reveal the background information, that is the textual semantic information (Babić, Martinčić-Ipšić & Meštrović, 2020; Wang, Nulty & Lillis, 2021). Current implementations of embedding methods for natural language processing include Word2vec, a static embedding method based on local semantic information (Mikolov et al., 2013), and Embeddings from Language Models (ELMo), a method based on dynamic semantic information (Wang et al., 2020).

In this study, embedding is implemented using the TensorFlow embedding layer. We do not directly use functions from the Gensim library, which is a popularly used library for Word2vec algorithm’s implementations. The embedding layer in TensorFlow being used to process text sequences can be considered an implement of the Word2vec algorithm. An embedding layer is primarily used to encode a sentence to produce a distributed representation of a word or character. After a sentence passes through this layer, the word vector of each word or character can be obtained, where k denotes the dimensions of the word vector and $X_i$ denotes the i-th word. A sentence of length n can then be expressed as

(1) $$\begin{array}{c}{X}_{1:n}=X_1\oplus X_2\cdots \oplus X_n\end{array}$$where $\oplus$ denotes the concatenation operator and $X_{i:i+j}$ denotes the matrix of feature vectors formed by the word vectors $X_i$, $X_{i+1}\dots ,X_{i:i+j}$.

In the proposed model, the convolutional, pooling, and fully connected layers for the classification task are positioned behind the embedding layer. This design enables the embedding layer to process semantic information for weight adjustment considering both the contextual information of the word and the classification task, thereby improving the semantic information extraction performance. Then the embedding layer is realized using the function tf.keras.layers.Embedding in TensorFlow to establish the spatiotemporal contextual features of the trajectory segment by an end-to-end method. Table 1 lists the parameters used in this study.

Neural network model construction and comparison

Datasets

Experimental data were obtained from the 2020 Digital China Innovation Competition (https://tianchi.aliyun.com/competition/entrance/231768/introduction), which provided trajectory data for three fishing-vessel operational types: trawl net, purse seine, and gillnet fishing. In these datasets, a trajectory segment comprises several track points of the vessel at sea based on a unique vessel ID, along with the fishing type corresponding to that track segment.

In this study, these data were preprocessed to remove operational track segments having fewer than five track points and heading directions outside the range of 0–360°. The final data volume was 8,163. The data fields comprised the fishing vessel ID, longitude, latitude, speed, heading, timestamp, and registered fishing method or operational type. The longitude, latitude, speed, and direction ranges were [0°, 127.803°], [0°, 44.824°], [0, 11] knots, and [0°, 360°], respectively. The distribution of each feature value domain in the dataset is shown in Fig. 3. The length distribution of the single operational trajectories and the data volumes for the different fishing behaviors are shown in Fig. 4. The track-point sequence length of a single-vessel trajectory segment ranged from 21 to 5,841, with an average length of ~598. From Fig. 4, it is evident that the majority of trajectory sequences have a length of less than 800. Consequently, the trajectory length for the model’s input has been determined to be 800. The dataset was divided into training, validation, and test datasets at a ratio of 7:2:1. The details of the dataset division are presented in Table 3.

The spatiotemporal contextual features of the fishing vessel trajectories were extracted from the original dataset and used as inputs for the recognition model. The model input features included the tensors of the spatiotemporal contextual features of the fishing vessel trajectories, times, speeds, and headings. The spatiotemporal context tensor of the fishing-vessel trajectories was obtained by encoding the longitude and latitude data using the Geohash method and performing extraction by embedding using natural language processing. The time feature was obtained by converting the “Month‒Day” data to a character string; this information was then represented by a vector obtained by embedding of nature language processing. The maximum-minimum normalisation method (Bhanja & Das, 2018) is used to preprocess the data on speed and direction of heading, so as with the non-geohash transformation geolocation data for the comparison.

Evaluation metrics

Learner or model generalizability is a performance metric that requires valid and feasible experimental estimation methods and evaluation criteria. The performance metrics reflect the task requirements, and different metrics can often yield different results when the capabilities of different models are compared.

Following the construction of the complete classifier, we used a series of evaluation metrics to determine its effectiveness. Accuracy is one of the most commonly used performance measures in traditional classifier evaluation metrics, calculated as the ratio of correctly classified samples to the total number of samples. Typically, this metric can be applied to standard balanced data having a positive-to-negative sample ratio of 1:1. However, the number of samples is usually inconsistent between categories. In such cases, Accuracy should not be the only metric used to evaluate the classifier. For example, for a sample with a positive-to-negative sample ratio of 999:1, the model Accuracy can reach 99.9% even if the classifier predicts only the negative sample as positive. However, only the negative-sample information must be considered.

Consequently, another evaluation metric F1 is introduced, for which a more significant value indicates better model classification. F1 takes into account both Recall and Precision. Recall denotes the proportion of all positive samples predicted to be correct in the classification process, as well as Precision denotes the proportion of samples predicted to be in the positive class by the model during classification, calculated as the percentage of the correct predictions made by the model.

To reduce the resulting instability caused by the randomization of the neural network parameters, for both Accuracy and F1, we used the mean ± standard deviation values of the maximum and minimum values obtained after 12 training sessions under the same conditions for the same model.

Influence of spatiotemporal contextual information on the classification accuracy of fishing vessel operation types

To investigate the impact of spatiotemporal contextual information on the classification accuracy of fishing vessel operation types, the comparison experiment using two different sets of trajectories data was designed. One set utilized latitude and longitude data directly for model training and prediction. The other set incorporated the spatiotemporal context information of the trajectories for the same purpose. The spatial contextual feature extraction of the trajectory were extracted using the GeoHash and Embedding techniques. Subsequently, the MB-1dCNN model was employed for training and prediction, achieving accuracy and F-score of 86.93% and 83.84% respectively on the test set.

As shown in Table 4, the comparison of these two experimental sets reveals that the classification accuracy and F-score improved by 14.13% and 22.36% respectively when using the Embedding method based on GeoHash transformation of vessel trajectories, compared to the model without spatiotemporal contextual information. This demonstrates that incorporating spatial contextual information from trajectories into the fishing vessel operation type classification model (MB-1dCNN) is highly effective in enhancing classification accuracy. Consequently, the spatiotemporal contextual information of trajectories promises to play a significant role in identifying fishing vessel types.

Impact of different Geohash lengths on results

To address the issue of identifying fishing vessel operation types, we examined the effect of varying Geohash lengths on the MB-1dCNN model’s ability to identify operation types using the spatiotemporal contextual features of fishing vessel trajectories by comparing the accuracy. Figure 5 illustrates a plot of the change in recognition accuracy with the Geohash length. As the Geohash length increased, both the weighted Accuracy and weighted F1 scores of the model exhibited an upward trend followed by a decline. The highest classification accuracy was achieved at a Geohash length of 5; the weighted Accuracy was 0.8595 ± 0.0082 and weighted F1 being 0.8573 ± 0.0085. The corresponding spatial extent was approximately 4.89 × 4.89 km².

As previously noted, each Geohash code represents a grid region on Earth, with the code length denoting the grid-region size. Figure 6 presents results for the gillnet, trawl net, and purse seine fishing operations in columns from left to right, respectively. The first row of Fig. 6 shows the original fishing vessel trajectories for each of these operations. The second to fourth rows (from top to bottom) present overlays of the Geohash geographic grids on the vessel trajectories when the Geohash lengths were 2, 5, and 6, respectively. When the Geohash length was too short (as seen in the second row of Fig. 6, with length = 2), the geospatial region was larger, with most of the trajectory points enrolled in one region. This led to information loss under the Geohash coding, thereby reducing the model classification accuracy. As the Geohash length gradually increased, the region size became more reasonable, and the specific fishing behavioral characteristics of the different fishing vessel types became evident after sequence encoding. Consequently, model classification accuracy improved (see the third row of Fig. 6, with length = 5). However, when the Geohash length was excessively long, the region division granularity became too fine, introducing noise and reducing model classification accuracy (see the fourth row of Fig. 6, with length = 6).

Impact of different network structures on the model

We examine the recognition accuracies of the different models for the three fishing vessel operational types under investigation in this study. As noted previously, we utilized the trajectory-point latitude and longitude to construct the spatiotemporal contextual feature. It was then used as input feature for the models, alongside embedded-time, speed, and heading. As depicted in Table 5, the SB-1dCNN model achieved a weighted average Accuracy of 0.8660 ± 0.0037 and F1 score of 0.8636 ± 0.0038. The MB-1dCNN model showed slightly higher values, with a weighted average Accuracy of 0.8671 ± 0.0058 and F1 score of 0.8651 ± 0.0058. Both models demonstrated superior performance to the SB-FCNN and MB-FCNN models.

The SB-1dCNN and MB-1dCNN models outperformed their SB-FCNN and MB-FCNN counterparts. This indicates that CNN models are better suited for fishing-vessel operational types recognition than FCNN models. The success of CNN models may be due to their ability to mitigate the impact of noise between neighboring trajectory points, which can arise from becalmed scenarios or missed reports in certain trajectory segments. By using a convolution kernel of a specific size to extract features from trajectory segments, CNN models effectively reduce the influence of such noise, unlike FCNN models, which are more susceptible to noise-induced classification accuracy reductions. Thus, the results confirm that 1D-convolution is superior to fully connected networks for feature extraction in sequential data.

Effect of single- and multi-branch structures on model performance

As previously illustrated in Fig. 1, two distinct model-building strategies have been developed for 1D-sequence data: single- and multi-branch strategies. In the single-branch approach, multiple factors are concatenated and fed into the model sequentially as depicted in Figs. 1A and 1B. In contrast, the multi-branch method involves inputting multiple factors into separate substructures, and then combining the extracted information before feeding it into the subsequent network for learning as depicted in Figs. 1C and 1D.

The experimental results presented in Table 3 indicate that, for both the FCNN and CNN structures, the multi-branch-based models exhibit higher classification accuracy than the single-branch models. Specifically, the MB-1dCNN model demonstrated improvements of 0.0011 in weighted Accuracy and 0.0015 in weighted F1 score compared to the SB-1dCNN model. Similarly, the MB-FCNN model showed enhancements of 0.0524 in weighted Accuracy and 0.0835 in weighted F1 score in comparison to the SB-FCNN model. This difference may be due to the number of neural networks used to handle feature extraction. The multi-branch structure allows for separate feature extraction and fusion of inputs, avoiding the noise impact that occurs when inputs are directly concatenated in the single-branch structure. Consequently, when dealing with multiple types of different inputs, a multi-branch structure can more effectively extract the features of the inputs and enhance the model classification accuracy. Overall, this outcome further demonstrates the advantages of the proposed MB-1dCNN model in identifying fishing vessel operational types.