Field-road classification for agricultural vehicles in China based on pre-trained visual model

Datasets

The proposed methodology was trained and tested on the four trajectory datasets collected by wheat harvesters and paddy harvesters in China. The four trajectory datasets include a wheat harvesting trajectory dataset collected in 2021 (Wheat 2021), a paddy harvesting trajectory dataset collected in 2021 (Paddy), a wheat harvesting trajectory dataset collected in 2023 (Wheat 2023), and a wheat harvesting trajectory dataset collected in 2024 (Wheat 2024). Furthermore, Wheat 2021 and Paddy are collected by Chen et al. (2022).

Each harvester was installed with a GNSS positioning terminal, and the positioning accuracy was 5 m. Once a harvester got started in a day, a daily trajectory began to be recorded. Each point of a trajectory contains five parameters: timestamp (YYYY:MM:DD-hh: mm:ss), longitude (°), latitude (°), speed (m/s), and direction (°). As shown in Table 1, the four datasets are different in data statistics, such as recording updated frequency (5s vs. 30s for the wheat and paddy harvesting trajectories). Moreover, the trajectories of the datasets are spatially distributed across fourteen provinces of China, from north regions to south regions.

Furthermore, for each trajectory, existing data preprocessing (Chen et al., 2021) was applied, which includes smoothing noise points and removing duplicated points. The processed data were manually annotated, and each point of a trajectory was marked as either “field” or “road”. Moreover, each dataset is randomly divided into the training set and test set according to the ratio of 9:1. The datasets are available at: https://zenodo.org/records/10561232.

Overview

As illustrated in Fig. 2, our multi-view field-road classification framework comprises three parts: input feature extraction, multi-view fusion feature extraction, and linear classification. The uni-view feature extraction (Visual encoder and Physical encoder) provides input features to field-road classification consisting of the multi-view fusion feature extraction and the linear classification.

(1) Input feature extraction: For each point in a trajectory, two uni-view encoders (physical encoder E_phi and visual encoder E_vis) are applied to extract two input feature vectors (a physical feature vector and a visual feature vector) from the physical view and visual view, respectively. The physical feature provides the physical motion characteristics, and the visual feature gives the shape characteristic. Moreover, in encoder E_vis, an image is generated for each point, and then a visual feature is extracted from the image using an image recognition model which has been developed through fine-tuning the pre-trained ResNet model.

(2) Multi-view fusion feature extraction: a multi-view fusion encoder E_mv is used to extract a multi-view fusion feature vector F_mv for each point based on its two input feature vectors.

(3) Linear classification: a point is classified using a linear layer based on its F_mv.

Physical encoder

We follow the feature extraction used by Chen et al. (2022) to extract the physical motion characteristics of a point. Specifically, the physical encoder E_phi represents a point as a vector with seven features (speed, longitude_diff, latitude_diff, direction_diff, acceleration, jerk, bearing rate). The speed is the recorded speed on the point. Longitude_diff, latitude_diff, and direction_diff are the differences in the longitude, latitude, and direction between the current point and its previous point, respectively. Acceleration, jerk, and bearing rate are given in Dabiri & Heaslip (2018).

Visual encoder

As illustrated in Fig. 2, there are three components in the visual encoder E_vis: image generation, vision model’s fine-tuning, and visual feature extraction. The image generation module creates an image for each trajectory point, containing the contextual information specific to that point (e.g., several parallel strips, the speed of the point and its neighboring points). Vision model’s fine-tuning provides an image recognition model through fine-tuning the existing pre-trained ResNet model (He et al., 2016). Based on the fine-tuned ResNet model, the visual feature extraction module extracts a visual feature vector for each point in a trajectory.

Image generation: In order to capture the shape formed by a point and its neighboring points, we generate a 2D image for the point, and such an image contains points located in a rectangle region of the point. For example, n images (Fig. 3C) are generated for n points in a trajectory (Fig. 3A) one by one, where each point is represented by a row in Fig. 3B, and each image is represented by the format of Fig. 3D. Since both all images input to the ResNet model must be the same-sized pixels in either dimension and the distance information in a trajectory needs to be kept. We follow the work of Endo et al. (2016) to clip a fix-sized rectangle region centered on the point, where a rectangle region is defined by ranges of latitude and longitude (the side length of the rectangle is 0.0002° in this study). Then, following the image generation method proposed by Chen et al. (2023), we generate a colorful image for each rectangle region, where the three color channels (Red-Green-Blue; RGB) are used to encode three physical motion characteristics (speed, direction, speed+direction), respectively. Specifically, the value in the R channel is the ratio of the speed at the current point to the maximum speed in the entire trajectory, the value in the G channel is the ratio of the direction at the current point and 360° (the maximum direction value), the value in the B channel is the average of the values in the R and G channel, the calculated formulas of R, G, and B as Eqs. (1)–(3). Notice that unlike an image per point in our method, Chen et al. (2023) generated an image for the whole trajectory, and the detailed information on each point is missing in such an image. (1)${\displaystyle R=\frac{speed\left(i\right)}{max\left(speed\right)}}$ (2)${\displaystyle G=\frac{direction\left(i\right)}{360}}$ (3)${\displaystyle B=\frac{R+G}{2}}$ where speed(i) is the speed of i-th trajectory point, max(speed) is the maximum speed in the entire trajectory, direction(i) is the direction of i-th trajectory point, and 360 represents all directional degrees in a two-dimensional coordinate system.

Vision model’s fine-tuning: the significant progress of the pretraining-finetuning paradigm has achieved great success in the vision understanding domain (Chu et al., 2016; Tajbakhsh et al., 2016), which consists of two phrases, as shown in Fig. 4. In the pre-training phase, a visual model is trained based on a deep neural network architecture and large-scale high-quality natural images. Moreover, such a visual model usually can effectively learn the general knowledge in natural images. In the fine-tuning phase, the pre-trained ResNet model is loaded, and the model parameters are re-trained with task-specific images so that the model can learn the task-specific knowledge in the images. To utilize both the rich general knowledge and the task-specific knowledge, we adopt this paradigm for our image recognition model development.

As illustrated in Fig. 4, we choose the pre-trained ResNet, an image recognition model which has been trained using the raw ResNet network and large-scale crowd-labeled images (Deng et al., 2009). Then, we fine-tune the pre-trained visual model for our specific image recognition task that assigns a label (either “field” or “road”) to each image. The data used for fine-tuning is the images generated by the image generation module for training trajectories, and the task-specific label of an image is the label of the central point in the focused training trajectory segment.

The model training involves optimizing the parameters of the ResNet neural network using the training image data. The parameter optimization in the pre-training phase is shown in Eq. (4), where the parameters of the raw ResNet network (W₁) are initialized in the way used in He et al. (2016). Based on the large-scale crowd-labeled images, (x₁, y₁), W₁ are updated to W₂, producing the pre-trained ResNet model. Then, the parameter optimization in the fine-tuning phase is illustrated in Eq. (5), where the parameters W₂ are updated to W₃ using the task-specific images, (x₂, y₂), producing the fine-tuned ResNet model. (4)${\displaystyle ResNet\left(W_1\right)\underset{}{\overset{\left(x_1,y_1\right)}{\to }}ResNet\left(W_2\right),x_1\in \left\{naturalimages\right\},y_1\in \left\{naturalimagelabels\right\}}$ (5)${\displaystyle ResNet\left(W_2\right)\underset{}{\overset{\left(x_2,y_2\right)}{\to }}ResNet\left(W_3\right),x_2\in \left\{task\text{-}specificimages\right\},y_2\in \left\{task\text{-}specificlabels\right\}}$ where W₁ is the initialized model weights (i.e., weights of the model without any training), W₂ is the pre-trained model weights which were trained in natural image datasets, and the W₃ is the fine-tuned model weights in the task-specific images (i.e., field-road trajectory point image).

Visual feature extraction: As illustrated in Fig. 2, given the fine-tuned ResNet model, visual features are extracted for each input image. Specifically, the image is fed to the fine-tuned ResNet, and the features generated by the backbone of the fine-tuned ResNet are output as the visual features. Although there seems a big difference between our generated images (i.e., task-specific images in Fig. 4) and the crowd-labeled natural images (i.e., crowd-labeled images in Fig. 4) used to train the pre-trained ResNet model, we observe that the visual feature extraction is still effective. This confirms that the task-specific knowledge in our images has been learned through the vision model’s fine-tuning.

Multi-view fusion encoder

Besides the physical and visual features, there are sequential relationships between points of a trajectory, which are also crucial for detecting the motion and shape information of a trajectory. Similar to Chen et al. (2023), BiLSTM is applied to capture such sequential relationships. BiLSTM is a kind of recurrent neural network which can effectively extract short-term and long-term information.

As illustrated in Fig. 2, for each point in a trajectory, its physical feature vector and visual feature vector are concatenated to a new feature vector, and thus the trajectory is represented by a sequence of the catenated feature vectors. Then, the sequence is fed to the BiLSTM network to obtain a new sequence of feature vectors, which provides a multi-view sequential fusion feature representation for the trajectory.

Linear classification

Given the multi-view fusion feature vector F_mv−i for the i-th trajectory point, which is obtained from the multi-view fusion encoder, the softmax layer is used to calculate the probabilities over the categories (‘field’ and ‘road’) for the point (see Eq. (6)), and the category with a higher probability is selected as the final one (see Eq. (7)). (6)${\displaystyle p_i=softmax\left(F_{mv-i}\right)}$ (7)${\displaystyle y_i=softmax\left(p_i\right)}$ where p_i denotes the predicted probability distribution over the two categories (i.e., “field” and “road”) and the y_i denotes the final predicted category for the i-th point, respectively.

Experimental settings

Baseline methods: For a comprehensive evaluation of our proposed method, namely Phi+ResNet(finetune)+BiLSTM, the following four state-of-the-art field-road classification methods were chosen as baselines.

• DBSCAN+Rules (DR) (Chen et al., 2021): It is a method that combines a density clustering method (DBSCAN), and a set of direction-distribution-based rules designed for the field-road classification.

• Decision Tree (DT) (Poteko, Eder & Noack, 2021): It is a method that extracts features with 25 dimensions to represent a trajectory point and then feeds the features to a traditional supervised learning algorithm, a decision tree, to make the field-road classification.

• Graph convolutional network (GCN) (Chen et al., 2022): It is a graph convolutional neural network designed for field-road classification, in which a spatio-temporal feature vector is learned to represent a trajectory point. Given a trajectory, a spatio-temporal graph is constructed, and then a graph convolution process is used to extract the spatio-temporal features, which have aggregated the features of the neighbors of the point.

• SVB (Chen et al., 2023): The approach initially employs a computer vision model to perform semantic segmentation of images, extracting fine-grained pixel-level features of trajectory points. Then, these pixel-level features are combined with the statistical features of each trajectory point and fed into a fully connected network for field-road classification.

Evaluation metrics: following previous work Chen et al. (2021), Poteko, Eder & Noack (2021), we chose four metrics to evaluate the performance of the field-road classification method: the precision (Pre), recall (Rec), F1-score (F1), and accuracy (Acc). The performance on the test set is reported. The experimental results of our models are averaged over 10 runs with different random seeds to ensure the final reported results are statistically stable.

Implementation details: For our multi-view field-road classification method, we used the BiLSTM interface provided by PyTorch (a Python-based deep learning library) (Paszke et al., 2019), and chose the ResNet interface published by He et al. (2016) as our pre-trained ResNet model where the number of layers was set to 50. Moreover, for DBSCAN + Rules, SVB, and GCN, we directly used the implementations in the original paper, and for DT, we used the re-implementation method in Chen et al. (2022).

Training details: The goal of the training process is to learn model parameters in such a way that a loss function is minimized. There are two models to be learned in our method: the fine-tuned ResNet model, and the field-road classification model including the multi-view fusion encoder and the linear classification (Fig. 2). Specifically, the fine-tuned ResNet model was obtained by re-training the pre-trained ResNet model (as shown in Fig. 4). During the re-training process, each batch contained 128 images. On the other hand, the field-road classification model was trained from scratch. For the BiLSTM network, the sequence length of a segment was set to 32, and the hidden state size and layer number of BiLSTM were set to 256 and 2, respectively. During the training process, each batch contained four trajectory segments. Moreover, during either the training or re-training process, the cross-entropy loss function (Hinton, Osindero & Teh, 2006) was used to calculate the error between predicted labels and ground truth labels, the Adam optimizer (Kingma & Ba, 2014) was used to update model parameters, and the learning rate was set to 0.0001. Finally, the configurations used in our experiment are listed in Table 2.

Method comparisons

To compare the effectiveness of the baselines with our proposed method, the five field-road classification methods were evaluated on the four datasets, and the performances are shown in Table 3. From the table, we observe that our method, Phi+ResNet(finetune)+BiLSTM, has achieved the best performances on four trajectory datasets (Wheat 2021, Paddy, Wheat 2023, and Wheat 2024) with accuracy scores of 92.56%, 87.91%, 90.31%, and 94.23%, respectively. The result indicates that our multi-view fusion feature extraction can effectively capture the motion and shape characteristics for the field-road classification task. Moreover, the overall performance improvement is a result of the improvement on the two categories (“field” and “road”), as illustrated in Table 4. For example, on the Wheat 2021 trajectory dataset, compared with the best baseline method (i.e., SVB), the F1-score of our method increases by 19.08% (from 58.56% to 77.64%) on the “road” category and 2.9% (from 92.60% to 95.50%) on the “field” category.

Compared with our method which can simultaneously capture the motion characteristic and shape characteristic, some previous field-road classification methods did not use the shape characteristic, which hurts the classification performance. For example, DBSCAN+Rules utilized the shape characteristic by the rules, which is not an effective way. DT focused on only the motion characteristics. Meanwhile, some existing field-road classification methods attempted to utilize the shape characteristic in different ways. GCN used a graph convolutional neural network to encode the shape characteristic, which is an indirectly way. Although both SVB and our method detected the characteristic through an image, SVB did not utilize the rich general knowledge in the pre-trained model for its visual feature extraction and did not use detailed information on each point, leading to worse performance.

Method analysis

In this section, we provide an in-depth analysis of our proposed method, which consists of three parts: (1) module analysis; (2) pretraining-finetuning paradigm analysis; (3) case study.

Module analysis

As shown in Fig. 2, there are three components to carry out the feature extraction: physical encoder, visual encoder, and multi-view fusion encoder. To investigate the effect of these components, an ablation study was performed and the results are shown in Table 5 for the four datasets, respectively.

First of all, we investigate the impact of the multi-view fusion feature extraction by removing either the physical encoder or the visual encoder from our proposed method, i.e., ResNet(finetune)+BiLSTM and Phi+BiLSTM. In other words, only input features extracted by the physical encoder or by the visual encoder are fed to the multi-view fusion encoder. As we can see from Table 5, compared with our method, the overall performance of the two methods, ResNet(finetune)+BiLSTM and Phi+BiLSTM, dropped significantly. E.g., for Phi+BiLSTM, there are 6.22%, 8.11%, 10.34%, and 5.44% accuracy score decreases on the four trajectory datasets, respectively. The performance drop indicates that both the physical encoder and the visual encoder are important.

Secondly, from Table 5, we observe that after removing the multi-view fusion encoder from our method, i.e., Phi+ResNet(finetune)+Cat, the overall performance degrades significantly, e.g., 8.27%, 6.91%, 7.8%, and 10.43% accuracy score decrease on the four trajectory datasets, respectively. Moreover, by replacing the BiLSTM network in the multi-view fusion encoder with a Transformer Encoder (TE) or gated recurrent unit (GRU) network (i.e., Phi+ResNet(finetune)+TE or Phi+ResNet(finetune)+GRU), the method’s performance in four trajectory datasets exhibits different declines. This confirms that our BiLSTM-based multi-view fusion encoder can effectively fuse the two kinds of input features based on sequential relationships between points of a trajectory.

Finally, regarding the point-based image generation that produces an image for each trajectory point (Sec. Visual Encoder), the balance between visual model performance and inference speed is examined. To evaluate the impact of different visual encoders on performance and the time consumption of the inference process, the ResNet network used in our visual encoder is replaced with one of the following three networks: a popular Vision Transformer (ViT) network, and two lightweight networks (MobileNet and ShuffleNet). As shown in Table 6, compared to the ResNet network, both lightweight networks perform worse on all the datasets, and the Vision Transformer network achieves a slight performance improvement on the Paddy and Wheat 2024 datasets. However, the consumption of the Vision Transformer is approximately 1.5 times longer than that of the ResNet network, which will seriously affect the model’s inference speed. Therefore, this study opts for a ResNet-based visual encoder. Moreover, the different performances of the visual encoder between the Paddy and the other three Wheat datasets may be due to their different recording updated frequencies of GNSS signals (5 s vs. 30 s for the wheat and paddy harvesting trajectories, respectively). Compared with the wheat harvesting trajectory data, the paddy harvesting trajectory data has a much lower recording updated frequency, which results in less effective feature extraction of the physical encoder, and more reliance on the features provided by the visual encoder.

Analysis of pretraining-finetuning paradigm: To show the contribution of the pretraining-finetuning paradigm, the fine-tuned ResNet in our method was replaced by another model, ResNet or ResNet(pre-train), so as to generate two field-road classification models, Phi+ResNet+BiLSTM and Phi+ResNet(pre-train)+BiLSTM, respectively. Specifically, in Phi+ResNet+BiLSTM, a ResNet model is trained from a raw ResNet network with the images used in the fine-tuning process of our visual encoder, and then the trained ResNet model is used to extract visual features. In Phi+ResNet(pre-train)+BiLSTM, the pre-trained ResNet model is used to extract visual features. In other words, the neither the pre-training nor fine-tuning process is used in Phi+ResNet+BiLSTM, and no fine-tuning process is used in Phi+ResNet(pre-train)+BiLSTM. The results of the two methods are shown in Table 7 for the four trajectory datasets, respectively. We observe that after removing the model’s fine-tuning process from our visual encoder, i.e., Phi+ResNet(pre-train)+BiLSTM, the overall performance degrades 6.71%, 1.54%, 1.39%, and 2.91% on F1-score for the four trajectory datasets, respectively. This reflects that the knowledge in our task-specific images has been successfully learned through the fine-tuning process (Fig. 4), and thus visual features specific to the field-road classification can be effectively extracted using the fine-turned ResNet model.

After removing the pretraining-finetuning paradigm from our visual encoder, i.e., Phi+ResNet+BiLSTM, the overall performance drops 2.48%, 9.25%, 2.07%, and 3.25% on accuracy scores for the four trajectory data, respectively. This confirms that both the general knowledge learned by the pre-trained model and the task-specific knowledge learned through the fine-tuning process play an important role in our field-road classification method. Moreover, the paddy harvesting trajectory data can take more advantage of the pre-trained ResNet model, because it relies more on the features provided by the visual encoder.

Case study: To give an intuitive visualization, we show the results of two state-of-the-art field-road classification methods (i.e., GCN and SVB) and our method on two trajectory samples: one is recorded during wheat harvesting and the other is recorded during paddy harvesting. From Fig. 5, we can see that on both samples, the GCN and SVB methods misidentify a lot of road trajectory points as field trajectory points. In contrast, our method can effectively reduce these mistakes. Through effectively extracting visual features from images generated for each trajectory point, our method can effectively distinguish characteristics between field and road trajectory points, leading to accurate classification.