SeedSortNet: a rapid and highly effificient lightweight CNN based on visual attention for seed sorting

Crop identification

Agricultural product assessment and recognition based on machine vision technology have been a research focus in agricultural applications, which is widely used in the detection and sorting of agricultural products such as wheat, corn, fruits, and the identification of plant diseases and insect pests.

 Liu et al. (2015) proposed a novel soybean seed sorting based on neural network. Eight shape features, three-color features, and three texture features are extracted to characterize the soybean seed, and BP neural network is used as the classification model to recognize the different defects. Experiments are conducted on the collected image set which includes 857 images of soybean seeds with insect damage, mildew, and other defects, and the results achieve an average recognition accuracy of 97.25%.  Huang (2012) proposed a neural network-based quality evaluation and classification method for areca nuts. The axis length, secondary axis length, axis number, area, perimeter, compactness, and the average gray level are used as the feature, and a back-propagation neural network classifier is employed to sort the quality of the areca nuts. Aznan et al. (2016) adopted machine vision methods to discriminate the variety of cultivated rice seed, namely M263. They firstly extracted different morphological features and then adopted a stepwise discriminant function analysis (DFA) to classify different types of rice. The classification accuracy for testing and training sets is 96% and 95.8%, respectively. HemaChitra & Suguna (2018) presented a novel sorting method of Indian pulse seeds based on image analysis techniques. In this method, they extracted the colors, shapes, and texture features, and adopted SVM for classification. The accuracy of their method can reach 98.9% accuracy. Li et al. (2019) designed a system to distinguish different damaged types of corn. An image database including normal corn and six different damaged corns is constructed. The features such as color and shape are extracted, then the maximum likelihood classifier is leveraged to discriminate these corns. Experiment results show that the classification accuracy is above 74% for all the classes. However, these methods adopt the handcraft features designed for the specific crops and the traditional classifier for sorting, and suffer from poor adaptability and low accuracy.

Due to the excellent feature representation ability, the deep learning models represented by CNN have achieved good performance in image classification, object detection, and semantic segmentation, and have also been successfully applied in plant disease detection and crop type classification. Sladojevic et al. (2016) proposed a CNN-based system to identify 13 types of plant diseases out of healthy leaves. The performance of this approach exhibited a top-1 success of 96.3%. Veeramani, Raymond & Chanda (2018) studied the effect of the number of convolution kernels in the two layers CNN on the recognition performance of haploid and diploid corn seeds. Veeramani, Raymond & Chanda (2018) adopted VGG19 and GoogleNet to classify corn seed defects and analyzed the influence of the two networks with different depths on the recognition performance. Dolata & Reiner (2018) proposed a method for the classification of barley varieties based on CNN, which is based on two separate convolutional layers to analyze dorsal and ventral sides, respectively. The network is trained on a small sample set of 200-500 cases in 8 categories, and the classification accuracy reaches 97%. Kurtulmuş (2021) adopted AlexNet (Krizhevsky, Sutskever & Hinton, 2012), GoogleNet, and ResNet to identify sunflower seed varieties, and then they were also evaluated in terms of both accuracy and training time, GoogleNet obtained the highest classification accuracy (95%).

The CNN-based crop identification method can achieve the better recognition rate and has higher self-adaptivity. However, the performance of the deep learning method depends on the depth and width of the model, the researchers often boost the depth and width of the model to improve the performance of the detection and recognition system. But this strategy results in slow speed and difficult deployment in industrial applications.

Model lightweight

For the specific application of crop seed sorting, due to the extremely fast production speed, it is necessary to develop the lightweight CNN model while maintaining a higher recognition accuracy. To trade off the model size and performance for deep neural network architectures has been an active research area, the related technologies include model compression, lightweight network design, etc (Liang et al., 2021).

Network construction

Due to the required higher processing speed and recognition accuracy, the representative deep neural network model (eg, ResNet, VGG, GoogLeNet, etc.) cannot efficiently tackle with the seed sorting task because of the lower efficiency and insufficient feature extraction ability. To address these issues, we construct a novel lightweight and efficient network which consists of Root-model, Shield-block, and a novel down-sampling module.

A.Root-model. To effectively improve the feature representation ability while reducing calculation, a dual-branch structure, namely Root-model (Fig. 2A), is designed as the first stage of SeedSortNet. First, the sixteen 3×3 filters are utilized to extract the shallow feature information (such as texture, shape, color, etc.) of the test image. Then, in one branch, the MaxBlurPool which is a non-overlapping 2×2 window is designed for reducing the aliasing effect and improving the shift-invariant of the network. In another branch, we firstly use 3×3 filters with the stride of 2 to convolute the input features and then adopt 1×1 filters to reduce the output dimension of the branch. Finally, the features generated by the two branches are concatenated together as the input of the next layer.

B.Shield-block. The inverted residual block (Sandler et al., 2018) which shifts the identity mapping from high-dimensional representations to low-dimensional ones (i.e., the bottlenecks), has been successfully applied in the design of lightweight networks. However, the connection of identity mapping between thin bottlenecks would inevitably lead to information loss since the residual representations are compressed (Daquan et al., 2020). In addition, this connection would also weaken the propagation capability of gradients across layers due to gradient confusion arising from the narrowed feature dimensions, and hence affect the training convergence and model performance (Sankararaman et al., 2020). To address these issues, we propose a dual-branch feature extraction module by improving the inverted residual block in this article (shown in Fig. 2B).

In the main branch, two 3×3 depthwise convolution layers are utilized to encode richer spatial information to generate a more expressive representation. Then we adopt two pointwise convolutional layers between two 3×3 depthwise convolutional layers, the first point convolution layer reduces the feature channel dimension and the latter increases its dimension, to encode the cross-channel information of the feature maps and reduce the computational complexity. Also, the linear activation function is adopted after the first pointwise convolutional layer and the last depthwise convolutional layer, which can prevent the feature values from being zeroed and hence reduce information loss.

For the other branch, a 3×3 depthwise separable convolution is designed to acquire the spatial representation of different receptive fields, and thus improve the feature representation ability. In the end, the concatenation of the two branches and its shortcut connection with the input feature are combined as the final output.

In the following, we present the detailed data processing operator of Shield-block, and it is shown in Table 1, where H, W and C represents the height, width, and channel number of the feature map, and 1∕t represents the reduction rate of channels. Moreover, to ensure the input channel dimension is consistent with the output channel dimensional, and the hyperparameter 1∕r (referring to the proportion of the input channel of the sub-branch output channel) is adopted in this paper, here we empirically set r to 6.

C.Down-sampling. Down-sampling operator can reduce the feature dimensionality while retaining the valid information. Traditional down-sampling methods (eg. MaxPool, Strided-Convolution, AvgPool, etc.) violate the shift-equivariance and results in small shifts in the input that can drastically change the output (Azulay & Weiss, 2018). And this phenomenon will become more obvious with the increase of the network depth. To solve this problem, a fuzzy sampling method, namely MaxBlurPool, proposed in (Zhang, 2019) is adopted in our method. The specific process of MaxPool and MaxBlurPool is shown in Fig. 3. From this figure, we can see that a blur kernel is inserted between max and subsampling to remove aliased in the MaxBlurPool method, thereby improving the shift-invariant and enhancing the robustness of the CNN model.

Lightweight sub-feature space attention module (SFSAM)

Due to the low discrimination of different types of seeds, the fine-grained spatial features are crucial for seed sorting. Therefore, an extremely lightweight sub-feature space attention module is proposed to selectively emphasize fine-grained features, and suppress the interference of complex backgrounds. It divides the feature maps into different subspaces and infers different attention maps for each subspace, thus can generate the multi-scale feature representation, it is shown in Fig. 4. The detailed process is described as follows.

The input feature map F ∈ R^H×W×C is firstly divided into g mutually exclusive groups $\left[F_1,F_2,F_3,\dots ,F_i,\dots ,F_g\right]$ (i.e., sub-feature spaces),where each sub-feature space F_i contains n intermediate feature maps. Zagoruyko & Komodakis (2016) have proved that pooling operations along the channel axis are effective in highlighting informative regions. Therefore, AvgPool and MaxPool operations are applied to g sub-feature spaces along the channel axis to generate g groups of average-pooled features $F_i^{max}\in R^{1\times H\times W}$ and max-pooled features $F_i^{avg}\in R^{1\times H\times W}$. Then, these features are concatenated separately to generate g efficient feature descriptors $\left[F_i^{max},F_i^{avi}\right]$. Thereafter, the g group’s subspace attention maps are generated using Eq.(1). (1)${\displaystyle M_i=softmax\left(f^{k\times k}\left(\left[MaxPool\left(F_i\right),AvgPool\left(F_i\right)\right]\right)\right)=softmax\left(f^{k\times k}\left(\left[F_i^{max},F_i^{avi}\right]\right)\right)}$

where f^k×k represents a convolution operation with a filter size of k × k. In this paper, k is empirically set to 7. The attention map in each group (subspace) can capture the non-linear dependencies among the feature maps by learning to gather cross-channel information. Meantime, we employ a gating mechanism with a softmax activation to map the attention weighting tensor into [0, 1].

Then, each group of feature maps gets the refined set of feature maps (${\hat{F}}_i$) after the feature redistribution in Eq. (2). (2)${\displaystyle {\hat{F}}_i=\left(M_i\otimes F_i\right)\oplus F_i}$

where ⊗ is element-wise multiplication and ⊕ is element-wise addition.

The final output $\hat{F}$ of SFSAM is obtained by concatenating the feature maps of each group, and it is described as Eq. (3). (3)${\displaystyle \hat{F}=concat\left({\hat{F}}_1,{\hat{F}}_2,{\hat{F}}_3,\dots ,{\hat{F}}_i,\dots ,{\hat{F}}_g\right).}$

Network topology

In this paper, a novel lightweight CNN model with a dual-branch network structure based on visual attention, denoted as SeedSortNet, is proposed for seed sorting with higher efficiency and recognition accuracy. The setting of the proposed SeedSortNet is outlined in Table 2. Each row denotes a sequence of building blocks, which is the repeated times of ‘R’. The reduction ratio of channels is used in each Shield-block is denoted by ‘ 1∕t’, and ‘C’ represents the number of channels in the output feature map.

We first use Root-module to generate 32 feature maps with the size of 112×112. Then, it is followed by the 15 Shield-blocks, four SFSAM attention modules, and four down-sampling layers (i.e., MaxBlurPool) spatial location distributions described in Table 2. At the first Shield-block of stages 2, 5, 8, and 11, the identity mappings do not need to be set because of the increasement of the feature map depth. Besides, we set ‘t’ to 2 to avoid the information loss due to the low-dimensional input. Finally, the output of the fourth down-sampling layer is followed by a global average pooling layer, which can convert 2D feature maps into 1D feature vectors.

Experimental datasets

In this section, two datasets are selected for verifying the effectiveness of the proposed network architecture.

Sunflower seed dataset

To thoroughly evaluate the effectiveness of the proposed method, we constructed our sunflower seed dataset on an industrial production line for the experiments. The image acquisition device equipped with a color line scan camera is established to collect 15834 sunflower seed RGB images with the size of 100×100 pixels. And we divided them into two categories, as shown in Fig. 5. The top row is the abnormal seed images composed of leaves, stones, defective seeds, etc. The bottom row is the normal sunflower seed images. It is worth noting that when the picture contains several seeds and impurities or hybrids, we will classify them as abnormal to ensure a low false alarm. In our experiment, about three-quarters of the dataset are randomly selected as the training set, and the remaining images are used for testing, as shown in Table 3.

Table 3:

Category distribution and proportion of the training set on maize seed dataset and sunflower seed dataset.

Dataset	Category distribution	The proportion of the training set
Maize	1770 (Diploid) 1230 (Haploid)	75%
Sunflower	7837 (Normal) 7997 (Abnormal)	≈75%

DOI: 10.7717/peerjcs.639/table-3

Implementation details and evaluation metric

Result analysis

Results on maize seed dataset

To assess the performance of our network (i.e., SeedSortNet) in the maize dataset. Six representative CNN models (i.e., AlexNet, VGG, ResNet, GoogleNet, DenseNet (Huang et al., 2017), and Resnext (Xie et al., 2017) are selected to conduct comparative experiments. The experimental results are shown in Table 5.

Table 5:

Performance comparison of different network on maize seed dataset.

Model	Parameters	FLOPs	Acc	Precision	Recall	F1-score
AlexNet	57.01M	711.46M	93.33	93.39	92.80	93.07
VGG11	128.77M	7.63G	94.67	94.54	94.43	94.48
VGG13	128.96M	11.33G	94.50	94.51	94.10	94.29
ResNet18	11.18M	1.82G	95.33	95.18	95.18	95.18
ResNet50	23.51M	4.12G	96.00	95.73	96.05	95.88
DenseNet121	6.96M	2.88G	95.83	95.58	95.85	95.71
GoogleNet	5.60M	1.51G	96.67	96.50	96.62	96.56
ResNext101	86.75M	16.48G	96.00	95.77	95.99	95.88
SeedSortNet	0.40M	512.06M	97.33	97.30	97.18	97.24

DOI: 10.7717/peerjcs.639/table-5

From the results in Table 5, we can observe that the adopted network can achieve good classification accuracy, and reach more than 90% under the same experimental environment. SeedSortNet has the best performance, with accuracy, precision, recall, and F1-score of 97.33%, 97.30%, 97.18%, and 97.24%, respectively, with a relatively low computational complexity and model size. These results verify the effectiveness of the proposed method.

Meanwhile, we also compared the performance of mainstream lightweight CNN models (eg, MobileNetv1, MobileNetv2, ShuffleNetv1, ShuffleNetv2, GhostNet) under different calculation benchmarks. The experimental results on the maize dataset in terms of computational complexity, model parameters, classification accuracy, and F1-score are shown in Table 6. The models are typically grouped into two levels of computational complexity for embedded device applications, i.e., ∼300MFLOPs and 500 ∼600MFLOPs. From the results, we can see that the larger FLOPs lead to higher accuracy in these lightweight networks. SeedSortNet outperforms other competitors consistently in classification accuracy and F1-score at various computational complexity levels. Furthermore, the number of parameters has also greatly decreased for the proposed method.

Table 6:

Performance comparison of maize seed dataset in lightweight CNNs.

Model	Parameters	FLOPs	Acc	F1-score
MobileNetv1	3.22M	587.94M	94.00	93.81
MobileNetv2 1.4×	4.06M	566.33M	96.00	95.89
ShuffleNetv1 2×(g=3)	3.53M	537.48M	96.67	96.55
ShuuffleNetv2 2×	5.35M	591.79M	96.00	95.90
GhostNet 2×	12.96M	529.89M	96.50	96.41
SeedSortNet	0.40M	512.06M	97.33	97.24
MobileNetv1 0.75×	1.83M	339.80M	91.83	91.65
MobileNetv2	2.23M	318.96M	95.83	95.71
ShuffleNetv1 1.5×(g=3)	2.00M	301.90M	96.17	96.05
ShuffleNetv2 1.5×	2.48M	302.65M	95.50	95.38
GhostNet 1.5×	7.79M	310.76M	95.83	95.72
SeedSortNet 0.75×	0.23M	338.64M	97.00	96.90

DOI: 10.7717/peerjcs.639/table-6

In order to further demonstrate the effectiveness of the proposed method, ROC curve is adopted to measure the model performance. Figures 6A–6C shows the ROC curves and the calculated area under curve (AUC) scores for using the proposed method and other network models (i.e., the above comparison network) on the maize seed dataset. From AUC scores, it is observed that the method achieve the best result of 99.33% compared with other models, which is superior to the above representative representative CNN models and lightweight networks.

Ablation study

The ablation study is carried on SeedSortNet and the network without SFSAM attention mechanism. The experimental results in Table 9 show that F1-score of 96.33% and 99.37% are obtained without SFSAM on the maize and sunflower seed datasets, respectively, which proves that Root-model and Shield block have better information extraction abilities. Meanwhile, SeedSortNet can get 97.33% and 99.56% F1-score, respectively. These demonstrate that SFSAM can selectively emphasize information features and suppress the interference of complex backgrounds, thereby improving the performance. At the same time, it can also be observed from Table 9 that SFSAM does not introduce too many parameters and calculations.

Effects of g selection in SFSAM

As described in the SFSAM section, the feature maps are divided into g groups and generate g attention maps. Each attention map can capture cross-channel information from the feature maps in its respective group. When g = 1, the cross channel information for the whole feature volume is captured by a single attention map, which is not sufficient to capture the complex relationships in the entire feature space and will result in lower predictive performance. When 1 < g < C, the better exchange of cross-channel information can be obtained. Therefore, we conduct experiments on the different parameters assigned by g (such as g = 1, 4, 8, 16), and the results in Table 10 confirm the correctness of the above analysis. It can also be observed that the maize and sunflower seed datasets have achieved higher performance gains, and the FLOPs and parameters increase with the increase g. Based on the experimental results, we adopt g = 4 to conduct the above series of comparative experiments which provides a reasonable trade-off between preserving good performance and improving computational efficiency.