A hybrid deep learning paradigm integrating segmentation architectures for precise plant disease identification and classification

Image segmentation

The proposed study introduces two innovative architectures, namely Iterative UNet and UNet with ASPP module, as depicted in Fig. 1. These novel approaches are systematically compared against established predefined segmentation models, thereby showcasing the superiority of the proposed methods in the domain of image segmentation. The results underscore the efficacy of the segmentation techniques in accurately delineating and isolating diseased regions within leaf images.

Feature extraction

The feature extraction phase is meticulously executed using the uniquely devised customized architectures, complemented by a hybrid approach that involves the concatenation of both customized and predefined models, as illustrated in Fig. 1. This methodical comparison extends to well-established architectures, offering a comprehensive understanding of the enhanced capabilities of the proposed models in extracting pertinent features. The dual-pronged approach not only underscores the effectiveness of the customization but also accentuates the synergistic benefits derived from the amalgamation of novel and established elements in feature extraction. The proposed models excel in capturing and representing distinctive features critical for accurate plant disease classification. The conducted comparative analyses validate the superiority of the proposed approaches, confirming their potential to substantially improve the accuracy and efficiency of plant disease identification systems.

The results of the experiments undeniably illustrate the efficacy of the segmentation, feature extraction, and concatenation processes employed in the models proposed. To succinctly capture the noteworthy contributions, it is delineated as follows:

• The study introduces two novel segmentation architectures—Iterative UNet and UNet with ASPP module—to enhance the accuracy of diseased region identification in plant leaf images. These architectures are systematically compared with established segmentation models, demonstrating their superior performance in delineating and isolating affected areas.

• A unique feature extraction methodology is proposed, integrating customized architectures with predefined models to enhance feature representation. The hybrid approach ensures the extraction of highly discriminative features, leading to improved classification accuracy in plant disease identification.

• To optimize computational resources, weight-based pruning techniques are incorporated into the proposed architectures. This enhances the efficiency of the models by reducing computational complexity while maintaining segmentation and classification accuracy, making the approach feasible for real-time agricultural applications.

• The study highlights the practical implications of the proposed methodologies by evaluating their performance across various plant disease datasets, ensuring robustness and adaptability in real-world agricultural scenarios.

The following section describes the literature survey of various works handled in this area. ‘Materials & Methods’ describes the materials and methods used. ‘Results’ goes through with the result and discussion part. Whereas ‘Discussion’ compares the proposed work with the existing state-of-the-art methods and finally ‘Conclusions’ discusses the conclusion and future scope.

Literature survey

Soulami et al. (2021) introduced a UNet model designed to identify, segment, and predict breast cancer. The segmentation and classification achieved an intersection over union (IOU) score of 90.5%, and the curve reached 99.88% when applied to Digital Database for Screening Mammography (DDSM) and INbreast datasets. The technique involved classifying every pixel in mammogram images. The utilized datasets encompassed a DDSM, a Curated Breast Imaging Subset of DDSM (CBIS-DDSM), and INbreast (Soulami et al., 2021).

Xiong et al. (2020) proposed an algorithm to introduce a detection technique for cash crop diseases. In order to detach the image background without human intervention an automatic image segmentation algorithm that uses the GrabCut algorithm is used but it retains the disease spots in cash crops. It was able to give a maximum recognition rate of over 80% for the 27 diseases (Xiong et al., 2020). The background of the infected regions is separated by the coefficient-based segmentation which was proposed by Khan et al. (2018). Then VGG16 and caffeAlexNet pre-trained models were used to extract features from chosen disorders. To fuse the extracted features before the max-pooling stage, a parallel features fusion step is integrated. Before using multi-class SVM, the majority of univariant characteristics are selected using genetic algorithms. Openly available datasets such as- plant village and CASC-IFW are used for experiments to reach the 98.60% classification accuracy target.

In a study conducted by Daniya & Vigneshwari (2021) they employed segmentation techniques using the Segmentation Network (SegNet). The integration of statistical, CNN, and texture features was a key facet of their methodology. These distinctive features, when harnessed within the framework of deep recurrent neural networks (RNN), were utilized for predictive modeling of plant diseases. To optimize the Deep RNN’s performance, the researchers employed the RideSpiderWater Wave (RSW) algorithm during the training phase. It gave an accuracy rate of 90.5%. This underscores not only the efficacy of the Deep RNN approach but also highlights the superior performance facilitated by the synergistic integration of SegNet segmentation, statistical, CNN, and Texture features, all guided by the innovative RSW algorithm (Daniya & Vigneshwari, 2021).

Chen et al. (2021) proposed a new method for BLS and rice leaf lesion detection. The segmentation was performed using the UNet network. In order to improve the precision of lesion segmentation, BLSNet used a consideration mechanism and multi-scale extraction integration. The proposed BLSNet model demonstrated higher segmentation and class accuracy, with a score of 98.2% (Chen et al., 2021). Rahman et al. (2019) and his research team executed a sophisticated segmentation approach, incorporating a sequential application of thresholding and morphological operations. To bolster the classification process, they employed a deep neural network. The noteworthy outcome of this research was a remarkable accuracy rate of 99.25%. It is crucial to highlight that their study relied on the comprehensive PlantVillage dataset (Rahman et al., 2019), emphasizing the robustness and reliability of their findings within the context of plant disease diagnosis and classification.

Altuntaş & Kocamaz (2021) introduced a method for deep feature extraction to identify tomato leaf diseases. Established pre-trained models like AlexNet, GoogLeNet, and ResNet-50 were employed as feature extractors. The deep features extracted from these models were combined to enhance prediction performance. The SVM classifier underwent training using 1,000 deep features obtained from the final fully connected layers of the CNN models. The accuracy rating for these combined deep features was 96.99%. The PlantVillage Dataset was used (Altuntaş & Kocamaz, 2021). For feature extraction using PlantVillage, Mohameth, Bingcai & Sada (2020), assessed the CNN models have to do with transfer learning and deep feature extraction. Three different CNN models were utilized, for example, ResNet 50, Google Net and VGG-16 two unique classifiers SVM and KNN (Mohameth, Bingcai & Sada, 2020).

ResNet is a well-known pre-trained CNN architecture, Kumar et al. (2020), worked on an open dataset which consists of 15,200 crop leaves images to perform a task of feature extraction and classification using ResNet34 model. The proposed model achieved a 99.40% accuracy on a test set. Subramanian, Shanmugavadivel & Nandhini (2022) utilized several pre-trained models to extract features and classify corn leaf diseases. The application of these trained models resulted in an accuracy of 93% in the classification of corn leaf diseases. The PlantVillage Dataset (Subramanian, Shanmugavadivel & Nandhini, 2022) was employed for this purpose.

Mobeen-ur-Rehman et al. (2019) implemented the computerized tools to recognize diabetic retinopathy. They have used CNN models that are pre-trained specifically AlexNet, SqueezeNet and VGG16 for classification that gives the accuracy of 93.46%, 91.82% and 94.49% discreetly. Additionally, a modified CNN model with five layers, two convolutional layers and three fully linked layers is suggested. Sensitivity, accuracy, and specificity for this approach were optimistic, with values of 98.94%, 97.87%, and 98.15%, respectively. They used the MESSIDOR Dataset (Mobeen-ur-Rehman et al., 2019). Verma, Chug & Singh (2020), have carried out three pretrained models for detecting severity of tomato late blight disease. The tomato late blight leaf images were taken from Plant Village data set based on three categories of severity namely- early, middle and end. The AlexNet, InceptionV3 and SqeezeNet pre-trained models were used for extracting features and extracted features were used for training multiclass SVM. In the proposed work AlexNet was able to perform well with highest accuracy of 93.4% (Verma, Chug & Singh, 2020).

Wang et al. (2021), suggested a two-stage model for cucumber leaf disease severity classification (DUNet) on complicated backdrops that merges DeepLabV3+ and U-Net. The DeepLabV3+ Model first segments the complicated backgrounds. The images are utilised as the input for the second step after the background has been segmented. UNet is employed for the segmentation in the second stage. It has a Dice coefficient of 0.6914 and a 93.27% accuracy rate. The average disease severity accuracy has been 92.85% (Wang et al., 2021).

Alotaibi & Alotaibi (2020) implemented hybrid architecture by combining Inception and Resnet architectures for hyperspectral image classification (HSI). Four standard HSI datasets were utilised to test the model. The proposed model was expert to achieve the highest of 99.02% accuracy on the Pavia Centre scenes dataset (Alotaibi & Alotaibi, 2020). The classification accuracy largely relies on depth of network and so Li et al. (2019), made a deep feature fusion model which uses pre-trained ResNet50 and VGG16 models to extract features. In the proposed work they made use of the UC Merced land-use dataset with 21 classifications and NWPU-RESISC45 dataset with 45 scene classes. The model was able to attain 99.31% with UCMD dataset and has obtained 94.03% with NWPU45 dataset (Li et al., 2019).

In order to tackle the issue of varying size of maize disease, Yang, Shan & Qu (2020) introduces an enhanced segmentation network model called Deeplabv3+. The proposed model addresses the problem through two main stages. Firstly, in the encoding stage, Resnet101 with atrous convolution is employed for feature extraction, along with the inclusion of the Jump-Connected Atrous Spatial Pyramid Pooling (JCASPP) module. This combination allows for the acquisition of multi-scale semantic information. Secondly, in the decoding stage, the output of JCASPP is merged with the shallow features of the backbone network. This integration facilitates the enrichment of spatial information through the utilization of multilayer and small multiplicative up sampling techniques.

In Mandal (2023) discussed recent advancements and challenges in the field of plant leaf disease detection. Utilizing advanced imaging methods and DL, the survey aims to provide a valuable reference for researchers involved in plant disease detection. It presents cutting-edge models to optimize the detection process, reducing time and costs. Additionally, the article explores existing challenges in the detection process, emphasizing the importance of finding solutions.

The above literature survey clearly portraits the different researches happened towards plant disease detection with various CNN architectures and algorithms. The main drawbacks discussed are the heavy model architectures and the enormous features accumulated for image processing. So in this work the heavy models are replaced with a light weight customized model along with hybrid UNet and DeepLabV3+ segmentation to reduce the total trainable parameters.

Proposed model

The proposed model’s classification architecture diagram is displayed in Fig. 2. in which the images from the dataset are cropped and resized and fed to the pretrained models and proposed hybrid models for classification. The architectural configuration of the proposed model for classification, incorporating segmentation, is visually depicted in Fig. 3. In this schematic representation, images sourced from the dataset undergo a sequence of preprocessing steps, including cropping and resizing.

Subsequently, these preprocessed images are input into a series of models, namely UNet, iterative UNet, UNet with ASPP, and DeepLab V3+, each dedicated to segmentation tasks. The process entails leveraging both pretrained models and the innovative hybrid models introduced in the proposal for extracting features crucial to the classification process. This amalgamation of established architectures and novel, custom-designed models enhances the robustness and versatility of the overall system, thereby contributing to the efficacy of classification and segmentation tasks within the proposed framework.

Dataset description

For the task of plant leaf disease detection, this study primarily focuses on three widely cultivated crops: tomato, potato, and corn. The leaf images utilized are sourced from the well-established and publicly available PlantVillage dataset, known for its high-quality, labeled images. Table 1 provides a detailed summary of the plant leaf diseases considered, along with the corresponding number of images per class included in the dataset.

A comprehensive visual overview is presented in Fig. 4, which displays representative samples encompassing both healthy and diseased leaves from the selected crops. The dataset comprises a substantial collection of 15,079 images, reflecting a diverse array of leaf conditions and disease manifestations across the three crop types. These images are categorized into 10 distinct classes, capturing the variations in disease symptoms and healthy foliage with precision.

To ensure effective model training and evaluation, the dataset is systematically partitioned. A majority portion, 13,554 images, is allocated for training, providing the deep learning model with a rich and varied learning base. An additional 1,505 images are reserved for validation, allowing for fine-tuning of model parameters and performance optimization during training. For model evaluation, a concise yet critical test set of 20 images is designated to assess the model’s generalization ability on unseen data.

This deliberate and structured data allocation reflects a methodical and performance-driven approach, aimed at enhancing the model’s accuracy, robustness, and reliability in detecting plant leaf diseases across multiple crop species under diverse conditions.

Pre-processing

Image pre-processing plays a crucial role in transforming the raw dataset into a format suitable for network analysis. Given that images in the dataset vary in size and come from different sources, including those with noisy backgrounds, the initial step involves resizing them to a standardized dimension. To achieve this, a resizing function is employed, ensuring that 256 × 256-pixel square images are effectively transformed into 224 × 224-pixel square images. This process carefully maintains the aspect ratios of the images while utilizing zero padding, a strategy that aligns with the model’s requirements and contributes to improved results.

Proposed semantic segmentation

Semantic segmentation involves assigning a class label to each pixel in an image, distinguishing it from classification, where a single class is assigned to the entire image. In semantic segmentation, instances of the same class are treated as individual objects, while instance segmentation considers different instances of a similar class as distinct items. Following preprocessing, the image undergoes segmentation, dividing it into segments with similar intensities and similarities. Subsequently, the image enters the Hue Saturation and Intensity (HIS) model. Semantic segmentation is executed using the U-net architecture, an extension of CNN with some modifications. This sophisticated model effectively addresses existing challenges, demonstrating faster image segmentation capabilities, processing images of size 512 × 512 within seconds. In reference to Badrinarayanan, Kendall & Cipolla (2017), images pass through an encoder where spatial information decreases and feature information increases, followed by the same passing through a decoder. Post-segmentation, the images are subjected to a CNN model for feature extraction.

Proposed iterative UNet architecture

The architecture used in this image segmentation model is Iterative U-Net architecture. The fundamental structure of U-Net consists of a contracting and expanding path, giving it a U-shaped configuration and earning it the name U-Net. This architecture is primarily constructed as an end-to-end fully convolutional network (Sivagami et al., 2020). Illustrated in Fig. 5 is the diagram of the Iterative UNet architecture, depicting the iterative application of the U-Net structure in both encoding and decoding. This iterative process results in a U-shaped pattern and provides a comprehensive view of the convolutional and pooling layers involved. The UNet structure is applied iteratively for N-1 iterations, where an increase in N leads to a proportional increase in trainable parameters. This introduces a tradeoff between time, space complexity, and performance.

The encoder part of the iterative UNet architecture consists of downsampling blocks, similar to the traditional UNet. Each downsampling block typically includes convolutional layers followed by pooling or strided convolutions to progressively reduce the spatial resolution while increasing the number of feature channels. This helps in capturing context and extracting high-level features. The output of each convolutional layer is given by Eq. (1),

(1) $$Z_l=\sigma \left(W_1\ast A_{l-1}+b_l\right)$$where Zl is the output, Wl is the weight matrix, Al−1 is the input from the previous layer, bl is the bias term, and σ is the activation function.

Instead of a single decoder, the iterative UNet incorporates multiple decoder blocks, each refining the segmentation results iteratively. Each decoder block takes the output of the previous decoder block and features from the corresponding encoder block. The output of each upsampling layer is given by Eq. (2),

(2) $$U_l=Upsample\left(z_{l-1}\right)$$where Ul is the upsampled output. Concatenation of the upsampled output with the corresponding feature map from the encoder will be denoted in Eq. (3) as,

(3) $$C_l=Concatenate\left(U_l,Z_{encoder},l-1\right).$$

Skip connections are formed by concatenating feature maps from the encoder to the decoder to preserve spatial information. The final layer typically involves a 1 × 1 convolution to produce the segmentation output.

(4) $$Y=\sigma \left(W_f\ast C_{final}+b_f\right)$$where Y is the final output which is shown in Eq. (4). This iterative process helps in refining the segmentation progressively, allowing the model to capture more fine-grained details and improve accuracy.

Proposed UNet architecture with atrous spacial pyramid pooling module

The UNet architecture has been augmented with the ASPP module from the DeepLab V3+ variant. This integration involves placing the ASPP module before the decoder, facilitating the capture of multi-scale contextual information. The ASPP module comprises parallel atrous/dilated convolutions with distinct dilation rates, strategically employed to grasp context at varying resolutions. The resultant feature maps from these parallel convolutions are subsequently merged to form a more comprehensive representation of the input. By incorporating the ASPP module, the architecture effectively captures both local and global contextual information, thereby enhancing segmentation performance.

Expressing the U-Net architecture with ASPP mathematically involves a series of equations. For clarity, a conceptual representation of the U-Net with ASPP is presented here. It is important to note that this representation offers a high-level overview, and actual implementations may entail additional details and considerations. Let X denote the input image and Y represent the output segmentation mask. The U-Net with ASPP can be dissected into encoding and decoding stages, with the ASPP module typically integrated into the encoding stage to ensure the capture of multi-scale contextual information.

Encoding stage: U-Net involves down-sampling and capturing features at different scales. Input Convolution can be denoted as in Eq. (5),

(5) $$F_1=Conv\left(X,W_1\right)+b_1$$where Conv represents the convolution operation, W1 is the weight matrix, and b1 is the bias term. Down sampling can be denoted as in Eq. (6),

(6) $$D_1=MaxPool\left(F_1\right)$$where MaxPool is the max pooling operation.

ASPP module: can be denoted as Eqs. (7) and (8),

(7) $$A_i=Con{v}_{dilated}(D_1,W_{asppi})$$

(8) $$A=Concat(A_1,A_{2,}{A}_{3,\dots }{A}_n)$$where Convdilated is the dilated convolution operation, Wasppi represents the weights for each dilated convolution, and Concat denotes the concatenation operation.

Decoding stage:

The decoding stage involves up-sampling and combining features from the encoding stage. Upsampling will be denoted as in Eq. (9),

(9) $$U_1=UpSample\left(A\right)$$where UpSample is the up-sampling operation. Skip connections will be denoted as in Eq. (10),

(10) $$S_1=Concat\left(U_1,F_1\right).$$

This step combines features from the up-sampled output and the corresponding feature map from the encoding stage. The decoder convolution will be denoted as in Eq. (11),

(11) $$F_2=Conv\left(S_1,W_2\right)+b_2.$$

The output convolution is shown in Eq. (12),

(12) $$Y=Conv\left(F_1,W_{out}\right)+b_{out}.$$

Skip connections, similar to the UNet architecture, are used to bridge the encoder and decoder blocks. These connections help in preserving spatial information and gradients during the upsampling process, aiding in accurate localization of objects and maintaining fine details. Table 2 gives the model summary of the above implemented model with various dilation rates such as 2, 4 and 6.

Proposed pre-trained models for feature extraction

Performance of architectures with and without segmentation

Graphical representation of customized model

Figure 9A shows the training and validation, accuracy and loss curves of customized model. Figure 9B shows the performance improvement of the customized model with segmentation using UNet with ASPP architecture.

Graphical representation of customized model+EfficientNet B7

Figure 10A shows the training and validation, accuracy and loss curves of customized model when concatenated with EfficientNet B7. Figure 10B shows the performance improvement of the concatenated customized model with segmentation using UNet with ASPP architecture.