An efficient parallel DCNN algorithm in big data environment

Non-local means

Non-local means (Zhou et al., 2023) is a filter based on adjacent pixels, denoising data using the image’s global information. For a given data sample U, the grey scale value $\overline{u}\left(x\right)$ Filtered by nonlocal means, it can be expressed as below. (1)${\displaystyle \overline{u}\left(x\right)=\sum _{y\in G}\varphi \left(x,y\right)\ast \theta \left(y\right)}$

where x denotes the search window, y denotes the adjacent window, ϕ(x, y) is the region similarity between x and y, and θ(y) is the noisy image.

Cosine similarity

Cosine similarity (Eminagaoglu, 2022) is a standard measure of similarity between two images. It maps the data of both images into vector space and then calculates the cosine of the inner product space between the vectors to compare the similarity. Assuming that X, Y denote the comparison of individuals, and x, y denote the one-dimensional form of the individuals, the cosine similarity Sim(X, Y) is expressed as follows. (2)${\displaystyle Sim\left(X,Y\right)=cos\theta =\frac{\overline{x}.\overline{y}}{∥x∥+∥y∥}}$

where $∥x∥,∥y∥$ is the mode of the image vectors.

Image to column

Image to column (Im2col) (Li et al., 2023) is a transformation function that converts convolutional operations into matrix multiplication. It reshapes a 3D input data matrix into a 2D matrix and transforms the convolution kernel into a 1D matrix, simplifying the convolution process. Assuming that x is the input feature map, w is the convolution kernel, and i, j are the vertices, the convolution result can be expressed as follows. (3)${\displaystyle a_{i,j}=\sum _{h=0}^{kH-1}\sum _{w=0}^{kW-1}{w}_{h,w}{x}_{i+h,j+w}}$

where kH, kW is the sum of the size of the image.

Mahalanobis distance

Mahalanobis distance (Li et al., 2024) is a commonly used distance metric that effectively measures the similarity between two unknown sample sets. Suppose that x represents the covariance matrix of a multidimensional random variable, the sample mean, and the current single-point data. Then, the martingale distance is as below. (4)${\displaystyle D_M\left(x\right)=\sqrt{{\left(x-\mu \right)}^T{S}^{-1}\left(x-\mu \right)}.}$

Data enhancement

To address the inherent class imbalance in the dataset, data augmentation techniques are employed to enhance the generalization capability of trained models. The methodology consists of three principal transformations:

Image rotation: Given an image sample I(x, y) with spatial coordinates (x, y), the rotated image I′ is computed as: (5)${\displaystyle I^{\prime }\left(x,y\right)=I\left(R\left(x,y\right)\right)}$ (6)${\displaystyle R=\left[\begin{array}{cc}\hfill cos\theta \hfill & \hfill -sin\theta \hfill \\ \hfill sin\theta \hfill & \hfill cos\theta \hfill \end{array}\right]}$

where R is rotation matrix, θ is uniformly sampled from ± 15° to preserve structural integrity while introducing meaningful variations.

Bidirectional flipping: To maximize data utility while maintaining semantic consistency, comprehensive flipping operations are implemented in both spatial dimensions. The transformations are precisely defined as: (7)${\displaystyle \begin{array}{c}{I}_{hflip}\left(x,y\right)\mathrm{}=\mathrm{}I\left(w\mathrm{}-\mathrm{}xy\right)\hfill \\ I_{vflip}\left(x,y\right)\mathrm{}=\mathrm{}I\left(xh-y\right).\hfill \end{array}}$

Among them, I_hflip refers to horizontal flipping, I_vflip refers to vertical flipping, w is the image width, and h is the image height.

Gaussian noise: To improve model robustness against sensor noise and acquisition variations, controlled noise augmentation is performed: (8)${\displaystyle I_{noisy}\left(x,y\right)=I\left(x,y\right)+N\left(0,{\sigma }^2\right).}$

Here, σ is the standard deviation of Gaussian noise, and σ is set to 0.05.

This systematic augmentation approach significantly increases minority class representation while maintaining the statistical properties of the original dataset. Each transformation is carefully designed to preserve label correctness and feature integrity, ensuring the augmented data remains physically plausible and semantically meaningful for training. The combined transformations effectively address class imbalance while enhancing model generalization capabilities.

Feature transformation function

A novel PFE-MHO strategy is proposed to eliminate redundant features in the image data. The strategy consists of two steps: feature extraction and feature selection.

Feature extraction: To obtain high accuracy of image features, we apply a non-local mean filter FT(a, b) based on cosine similarity, to be remove noise from the initial dataset. The process is as follows: (1) the adjacent window matrix and search window matrix centered on the pixel are set up in the target image, and the adjacent window is sliced in the current image. (2) the weighted values of the adjacent window are obtained by comparing the cosine similarity of the matrix where the pixel points a, b are located. (3) the convolution kernel f(x, y) of size is set to 3*3, and Laplace operations on g(x, y) is performed to obtain the equation h(x, y) = ∇²(f(x, y)⋅FT(a, b)). (4) If the second-order derivative of the Laplace equation for the node is cross-zero and the first-order derivative is at a large peak, the node is retained. Otherwise, the pixel is set to zero; (5) the current data nodes are merged to get the image after feature extraction.

Theorem 1. Nonlocal mean filter FT(a, b). Assuming a, b represent the adjacent window centered on a pixel a andthe search window centered on the pixel b respectively. The transformation function FT(a, b) is calculated as follows. (9)${\displaystyle FT\left(a,b\right)=\sum _{b\in G_i}\frac{\overline{a}.\overline{b}}{∥a∥+∥b∥}\cdot \mathrm{}\theta \left(a\right)}$

where θ is the noisy image, and G_i is the current image data.

Proof. If the value of the noise-free pixel block is ω(x, y) and the noise value is ψ(x, y), then the value of the pixel block fused with the noise is ρ(x, y) = ω(x, y) + ψ(x, y), the mean value of the pixel block is $\bar{\rho }\left(x,y\right)=1/k\cdot {\sum }_{i=1}^k{\rho }_i\left(x,y\right)$, and the expected value is $E\left[\bar{\rho }\left(x,y\right)\right]=1/k\cdot {\sum }_{i=1}^k\left(E\left[{\omega }_i\left(x,y\right)\right]+E\left[\psi \left(x,y\right)\right]\right)$. Due to the similarity between pixel blocks, E[ω_i(x, y)] can be simplified to ω(x, y). When the noise is 0, E[ψ(x, y)] = 0, so it follows that $E\left[\bar{\rho }\left(x,y\right)\right]=\omega \left(x,y\right)$. In addition, due to the non-correlation of the noise, the variance of ω(x, y) is $\sigma {\left[\bar{\rho }\left(x,y\right)\right]}^2=\sigma {\left[\omega \left(x,y\right)\right]}^2+1/k^2{\left({\sum }_{i=1}^k{\sigma }_{{\psi }_i}^2\right)}^2$. Since ω(x, y) is noiseless and corresponds to a variance of 0, there is $\sigma {\left[\bar{\rho }\left(x,y\right)\right]}^2=1/k\cdot \sigma {\left[\psi \left(x,y\right)\right]}^2$. This suggests that the noise ψ(x, y) is correlated with the variance and FT(a, b) reduces the data noise by reducing ψ(x, y).

Feature selection: After the feature extraction is completed, the images in the batch are divided into smaller chunks, and redundant features are eliminated by calculating the feature similarity between image blocks. The specific process is as follows: (1) the same class of images is sliced into equal size blocks and the feature correlation index FCI(x, y) to calculate the similarity between two image blocks x, y. (2) the key-value pairs <(x, y), FCI(x, y) > are stored in HDFS and remove redundant features in the image by removing items with FCI(x, y) < ɛ in the key-value pairs according to the correlation coefficient ɛ. (3) the key-value pairs are traversed again, and the key of the remaining key-value pairs in HDFS is read to obtain the number of the image block after filtering.

Theorem 2. Feature correlation index FCI(x, y). Suppose that x, y represent the two feature vectors, μ_x, μ_y represent the expectation of x and y, and σ_x, σ_y represent the variance of x and y. The formula for calculating the correlation index FCI(x, y) is as follows. (10)${\displaystyle FCI\left(x,y\right)=\frac{2{\sigma }_x\cdot {\sigma }_y}{{\sigma }_x^2+{\sigma }_y^2+{\left({\mu }_x-{\mu }_y\right)}^2}.}$

Proof. FCI(x, y) is a measure of the feature similarity between x and y. Set μ_x, μ_y denote the expectation of x and y, and σ_x, σ_y denote the variance of x and y. When σ_x = 0, the convolution operation is a linear superposition, and FCI(x, y) = 0; When σ_x ≠ 0, σ_y ≠ 0 and the features of the feature vectors x and y are similar, FCI(x, y) → 1.

Model parallel training

The PMT-IM strategy, which consists of two main stages, convolution kernel pruning and parallel Im2col convolution, is proposed to eliminate redundant convolutional kernels during the execution of parallel convolutional operations.

Convolution kernel pruning: To reduce the invalid computation caused by redundant convolutional kernels, the MDCV system is designed to sieve out invalid convolutional kernels. The specific process involves: (1) the covariance matrix S and mean µ of all the convolution kernels calculated in the convolution layer to construct the objective function f(x) of the MDCV. (2) the second-order Taylor expansion f(x) = x_k + ∇f(x_k)(x − x_k) + 0.5⋅(x − x_k)^T∇²f(x_k)(x − x_k) of f(x) is computed at its stationary point x_k. (3) if the current second-order derivative is non-singular, the next iteration point is x_k+1 = x_k + ∇²f(x_k)⁻¹∇f(x_k), otherwise the ∇²f(x_k)d =  − ∇f(x_k) is calculated to determine the search direction d_k, and then the next iteration point x_k+1 = x_k + d_k until the optimal MDCV value is found. (4) The distance dist between all the convolutional kernels and the MDCV value is calculated, and the convolutional kernels dist < α are pruned to complete the convolutional kernel pruning process.

Theorem 3. Mahalanobis distance center value MDCV. X₁, X₂, …, X_n represents the convolutional kernels in the network model, S represents the covariance matrix of the convolutional kernels, and µ represents the mean of the convolutional kernels. The equation for the MDCV of the Mahalanobis distance centroid is as follows. (11)${\displaystyle MDCV=x^{\ast }=min\sum _{x\in R^n}\sqrt{{\left(x-\mu \right)}^{\mathrm{T}}{S}^{-1}\left(x-\mu \right)}.}$

Proof. Suppose S is the covariance matrix of the vector group X₁, X₂, …, X_n and µ is the mean of the vector group. S is used to exclude the interference of similarity between variables. MDCV is the minimum distance from the eigenvector x^∗ to the eigenvector group X₁, X₂, …, X_n. When the feature vector x → MDCV, the feature vector x is replaced easily by X₁, X₂, …, X_n. When it means that x is linearly correlated with X₁, X₂, …, X_n. Therefore, MDCV is the minimum distance that indicates x^∗ to X₁, X₂, …, X_n.

Parallel Im2col convolution: The parallel Im2col convolution is combined with the MapReduce computing framework to extract image features. The main components are as follows. (1) Im2col method is used to map the feature map M_i into a convolutional operation matrix I_i, and the key-value pair <I_i, K_z > is stored with the corresponding convolutional kernel; (2) Map() function is used to perform data allocation; (3) perform matrix multiplication operation between I_i and the one-dimensional vector of the corresponding convolutional kernel to obtain the intermediate convolution result; (4) Reduce() function is used to merge the feature maps of the same data to obtain the output feature map.

Parameter parallel updating

To solve the problem of poor loss function convergence caused by abnormal nodes in the training data during gradient descent, an IMBGD strategy is proposed. This consists of gradient construction and parameter parallel updating.

Gradient construction: To eliminate the effect of abnormal node training data on batch gradient, loss of mean weight LAW(g_i) and loss summation gradient LSG(T) are designed. The concrete process is: (1) the mean value of the loss function of batch data is calculated according to the formula of the loss function, and the difference between the loss function and the mean value is calculated to obtain the loss mean weight, which is mapped to the key-value pair <g_i, LAW(g_i) > and stored in HDFS. (2) the partial derivative∇J_δiof the loss function of each data g_i is calculated concerning the current parameter δ_z, and the result <g_i, ∇J_δi > is stored in the HDFS. (3) the <g_i, LAW(g_i) > and <g_i, ∇J_δi > is traversed with g_i as the index, the average gradient LSG(T) is constructed, and the batch gradient of the current parameter is obtained.

Theorem 4. Loss mean weight LAW(g_i). Suppose g_i denotes a piece of data in a batch, J(ω, b)_i denotes the loss function value of the data g_i, batch_size is the batch data size, and LAD(g_i) is the absolute value of the difference between J(ω, b)_i and the average of the loss function value. The formula for calculating the loss-mean weight LAW(g_i) is as follows. (12)${\displaystyle LAW\left(g_i\right)=\left\{\begin{array}{c}\hfill 1\mathrm{}LAD\left(g_i\right)<\tau \hfill \\ \hfill 0\mathrm{}LAD\left(g_i\right)\ge \tau \hfill \end{array}\right.}$ (13)${\displaystyle LAD\left(g_i\right)=\left|\sum _{i=1}^{batch\text{\_}size}J{\left(\omega ,b\right)}_i/batch\text{\_}size-J{\left(\omega ,b\right)}_i\right|.}$

Proof. LAW(g_i) is the weight indicator of the loss function value of data g_i.τ is the threshold value to measure LAD(g_i). When LAD(g_i) < τ, g_i belongs to the regular value, LAW(g_i) = 1. When LAD(g_i) ≥ τ, g_i belongs to the abnormal value, LAW(g_i) = 0. Therefore, different values of LAW(g_i) can be used to decide whether to keep the loss function value of g_i or not.

Theorem 5. Loss summation gradient LSG(T). Assume that T denotes all data in the batch, ∇J_xi denotes the gradient of the loss function of data g_i with respect to parameter x, and batch_size denotes the batch data size. LSG(T) is calculated as follows. (14)${\displaystyle LSG\left(T\right)=\frac{\sum _{i=1}^{batch\text{\_}size}\nabla J_{xi}\times LAW\left(g_i\right)}{batch\text{\_}size}}$

Proof. It is known that ∇J_xi is the gradient of the loss function of g_i for parameter x, and batch_size is the batch data size. When LIW(g_i) = 1, the gradient ∇J_xi of data g_i decreases toward the optimal direction. When LIW(g_i) = 0, the gradient ∇J_xi of data g_i deviates more from the optimal direction and is not counted in LSG(T).

Parameters updating: The parameters are updated using the propagation algorithm in the following error term procedure to obtain the model after parallel update of parameters. The process is as follows: (1) the gradients ${\sum }_{i=1}^{k}LSG{\left(T\right)}^{l-1}$ of all parameters of the convolution kernel $W_k^{l-1}$ at layer l − 1 are computed and the results are mapped as key-value pairs $<W_k^{l-1},{\sum }_{i=1}^{k}LSG{\left(T\right)}^{l-1}>$ into HDFS. (2) the change $\Delta W_k^{l-1}$ of the parameters of the convolution kernel $W_k^{l-1}$ is calculated, and the network parameters of the convolutional kernel at the layer l − 1 are updated. (3) the updated parameters are synchronized to all nodes via HDFS, and then the next update is performed until all parameters in the model are updated.

Time complexity

To effectively analyze the time complexity of the PDCNN-MI, the algorithms MR-LSCNN, ECSO-FRCNN , and N-SAMCN are selected for comparison.

The time complexity of the PDCNN-MI algorithm is primarily composed of three parts: feature extraction in parallel, model training in parallel, and parameter update. The time complexity of each section is calculated as follows.

(1) Feature parallel extraction

The time complexity of this stage consists of two main components: feature extraction and feature screening. Suppose the number of samples is n, the number of cluster nodes is k, and the maximum number of Laplace iterations is m. In the feature extraction stage, two sliding windows are used to traverse the data and the cross-zero of the target window a is calculated, thus the time complexity is $O\left(m\cdot n\cdot logn/k\right)$. In the feature screening stage, the time complexity of calculating the similarity of two data slices is O(n²). Thus, the total time complexity is as follows. (15)${\displaystyle T_1=O\left(m\cdot n\cdot logn/k+n^2\right).}$

(2) Model parallel training

The time complexity of this stage includes two components: the computation of the MDCV value and the parallel convolution operation. Assuming the number of convolution kernels is p, and the size of the convolution kernel is s, the algorithm calculates the standard deviation of the convolution kernels and identifies the maximum linearly correlated kernel in the process of computing the MDCV value. This has a time complexity of $O\left(p\cdot s^2/k\right)$. After the data processing stage in the previous phase, the algorithm only needs to perform matrix multiplication during the parallel convolution operation, which has a time complexity of O(n²). Therefore, the time complexity of model parallel training is given as follows. (16)${\displaystyle T_2=O\left(p\cdot s^2/k+n^2\right).}$

(3) Parameter parallel updating

A gradient construction method for batch data is proposed in this stage. Assuming the number of nodes is k and the size of the fully connected input to the model is c, the time complexity of batch data gradient construction is O(k⋅c²). Therefore, the time complexity of the parameter parallel update is as follows. (17)${\displaystyle T_3=O\left(k\cdot c^2\right).}$

In a word, the time complexity of PDCNN-MI algorithm is as follows. (18)${\displaystyle T_{PDCNN-MI}=T_1+T_2+T_3=O\left(\left(m\cdot n\cdot logn+p\cdot s^2\right)/k+n^2+k\cdot c^2\right).}$

In the MR-LSCNN algorithm, spatial–temporal features are processed sequentially through convolutional operations and LSTM-based temporal modeling. Assuming p is the number of convolutional kernels, q is the kernel size, and h is the LSTM hidden units, the time complexity is as follows: (19)${\displaystyle T_{\left(MR-LSCNN\right)}=O\left(n\cdot p\cdot q+n\cdot h^2\right).}$

In the ECSO-FRCNN algorithm, the Faster R-CNN backbone and ECSO-based optimization are jointly applied. Assuming c is the number of convolutional kernels in the CNN backbone, q is the average kernel size, p is the number of region proposals, and k is the iteration count of ECSO, the time complexity is formulated as: (20)${\displaystyle T_{ECSO-FRCNN}=O\left(n\cdot C\cdot q+p\cdot q+k\cdot \left(p+{log}_p\right)\right).}$

In the N-SAMCN algorithm, the adaptive bilateral filtering, multi-feature extraction and deep hybrid classifier are jointly applied. Assuming n is the pixel count, k is the filter size, r is the LGXP neighborhood radius, d is the GLCM gray-level, c is the feature channels, and h is the hidden units, the time complexity is formulated as: (21)${\displaystyle T_{N-SAMCN}=O\left(n\cdot k^2+n\cdot r^2+d^2+c^2\cdot k^2+c^3+n\cdot h\right).}$

Since the value of n is much larger than other metrics in the big data environment, the time complexity of the PDCNN-MI algorithm is smaller than other algorithms, namely, T_PDCNN−MI < min(T_MR−LSCNN, T_ESCO−FRCN, T_N−SAMCN).

Experimental settings

The experimental setup consists of a computing cluster comprising eight nodes, including one master node and seven slave nodes, with their configurations detailed in Table 1. Each node has an Intel i5-12400F processor, 16GB RAM, and an NVIDIA RTX 3070Ti graphics card with 16GB of memory. To ensure efficient data transmission, the nodes are interconnected via a 1Gb/s Ethernet network. The software environment for the experiments includes Ubuntu 22.01 as the operating system, Python 3.9 for scripting and model implementation, JDK 18.0.1 for Java-based processes, and Hadoop 3.2.1 for distributed computing.

Experimental dataset

To evaluate the effectiveness of the PDCNN-MI algorithm, four datasets from different domains were used in the experiments: EMNIST-Balanced (Baldominos, Saez & Isasi, 2019), CompCars (Xu et al., 2022), ImageNet 1K (Li et al., 2022), and COCO5 (Lin et al., 2014). Table 2 provides details of these datasets. EMNIST-Balanced is a handwritten character dataset derived from the NIST database. CompCars contains car images collected from both network and surveillance scenarios. ImageNet 1K is a large-scale digital image dataset widely used in computer vision tasks. COCO is a large-scale dataset designed for image detection, semantic segmentation, and keypoint detection.

Experimental metrics

Experimental results and analysis

Visualization of feature map

To evaluate the impact of the PFE-MHO strategy on feature map accuracy, the feature representations of the algorithm were visualized to assess the necessity of applying the PEFS-MHO strategy to the VGG-16 model. The corresponding visualization results are presented in the following Figs. 2A and 2B.

Compared to the algorithm without the MHO-PEFS strategy, the version incorporating the PFE-MHO strategy produces feature maps with enhanced feature representation at each convolutional layer. This improvement is clearly illustrated in Fig. 2A, which demonstrates that the PFE-MHO strategy results in more structured and refined feature representations in each feature map. In contrast, Fig. 2B shows that without the PFE-MHO strategy, the boundary delineation of extracted features appears less distinct, particularly in areas of light and dark transitions.

The observed improvements can be attributed to the following factors: The PFE-MHO strategy enhances feature representation by employing non-local mean filtering based on cosine similarity and zero-crossing detection from the Laplacian operation. These techniques enable the extraction of edge features while effectively reducing noise interference. By improving the quality and accuracy of the data fed into the convolutional neural network, the strategy preserves essential information and provides a robust foundation for subsequent model training. This, in turn, contributes to improved model accuracy and stability.

Furthermore, the feature selection process within the PFE-MHO strategy optimizes model performance by eliminating redundant features. This is accomplished through the calculation of the feature correlation index and the application of a correlation coefficient threshold. By reducing data dimensionality, the strategy lowers computational resource consumption and minimizes the risk of overfitting, thereby enhancing the generalization capability of the model.

The benefits of this approach extend to predictive accuracy and classification performance when applied to new data. By ensuring more efficient and accurate model learning in complex data environments, the PFE-MHO strategy proves to be a highly effective method for enhancing feature extraction in the PDCNN-MI algorithm.

Algorithm feasibility

Experiments were conducted on four datasets: Emnist-Balanced, CompCars, ImageNet 1K, and COCO to assess the feasibility of the PDCNN-MI algorithm. Figure 3 illustrates the speedup performance of the algorithm across these datasets.

As illustrated in Fig. 3, the speedup ratio for all four datasets—Emnist-Balanced, CompCars, ImageNet1K, and COCO—demonstrates a consistent increase as the number of computational nodes grows, peaking when the node count reaches eight. Specifically, with two nodes, the speedup ratios are 1.61, 1.54, 1.75, and 1.63, respectively. When the number of nodes is doubled to four, the speedup ratios rise significantly to 2.92, 3.08, 3.3, and 3.17. Further increasing the node count to six results in even higher speedup ratios of 4.26, 4.03, 4.67, and 4.46. Finally, with eight nodes, the speedup ratios reach their maximum values of 5.1, 5.4, 5.85, and 6.2, indicating that the algorithm achieves its highest efficiency at this node count. This trend underscores the algorithm’s ability to effectively leverage additional computational resources to accelerate performance across diverse datasets.

Additionally, the PDCNN-MI algorithm demonstrates a more significant speed improvement when processing large-scale datasets such as ImageNet 1K and COCO. This observation demonstrates the efficiency of the algorithm in handling large volumes of data, confirming its scalability and effectiveness in high-performance computing environments.

Algorithm runtime analysis

To evaluate the runtime efficiency of the proposed algorithms in a large-scale data environment, experiments were conducted to compare the performance of the N-SAMCN, ECSO-FRCNN, and MR-LSCNN algorithms with the PDCNN-MI algorithm. The comparison criterion was the total runtime required for each algorithm to achieve stable training accuracy on the specified dataset. The experimental results are illustrated in Fig. 4.

As shown in Fig. 4, the difference in training time between the PDCNN-MI algorithm and the other algorithms is minimal when processing small datasets such as EMNIST-Balanced and CompCars. Specifically, the PDCNN-MI algorithm reduces training time by 28.76 min, 15.21 min, and 37.84 min for EMNIST-Balanced and by 35.87 min, 8.45 min, and 45.93 min for CompCars, compared to the other algorithms. However, the training time difference becomes more substantial when dealing with large-scale datasets such as ImageNet 1K and COCO. In ImageNet 1K, the PDCNN-MI algorithm reduces 91.64 min, 56.39 min, and 186.12 min, while in COCO, the reductions are 106.73 min, 65.28 min, and 197.51 min. These results demonstrate that the PDCNN-MI algorithm significantly improves computational efficiency, particularly for large-scale data processing.

The computational inefficiency of N-SAMCN, ECSO-FRCNN, and MR-LSCNN on large-scale datasets primarily arises from their algorithmic deficiencies in three key aspects: inadequate handling of feature redundancy in high-dimensional data leading to computational waste, insufficient optimization of convolutional operations for high-resolution inputs, and defective convergence mechanisms due to poorly designed loss functions, suboptimal optimization algorithms, and ineffective handling of data imbalance. These fundamental shortcomings collectively result in significantly prolonged runtime during both training and inference phases.

The reasons why the PDCNN-MI algorithm performs well are: The PMT-IM strategy calculates the covariance matrix and mean of all convolutional kernels in the convolutional layer at each node, constructing the objective function for the Mahalanobis distance center value (MDCV). By computing the second-order Taylor expansion of the objective function at its stationary points, the strategy iteratively identifies the optimal MDCV value. This process removes redundant convolutional kernels that are highly correlated with others in the current convolutional layer, thereby reducing redundant parameters in the network model, decreasing computational load and storage requirements, and ultimately accelerating convolutional layer operations. Additionally, by integrating the Im2col algorithm with the MapReduce computational framework, the strategy implements parallel Im2col convolution, transforming convolution operations into matrix multiplications and executing them in parallel across multiple computational nodes. This approach significantly enhances the computational speed of the convolutional layer. Compared to traditional convolution methods, the PMT-IM strategy can substantially reduce model training time and improve the overall performance of the algorithm.

Speedup ratio analysis

To assess the parallel computing performance of the PDCNN-MI algorithm in a large data environment, 10 tests were conducted on the above four data sets for the N-SAMCN, MR-LSCNN, and ECSO-FRCNN algorithms. The mean values from these tests were used to compute the speedup ratios of each algorithm across different numbers of computational nodes. The experimental results are presented in Fig. 5.

As shown in Fig. 5, the speedup ratios of all algorithms across the four datasets increase as the number of computational nodes grows. The speedup ratio gap between the algorithms is relatively small when processing EMNIST-Balanced, a smaller dataset, as illustrated in Fig. 5A. When the number of nodes is two, the speedup ratio of the PDCNN-MI algorithm is 0.06, 0.01 and 0.21 higher than that of the N-SAMCN, ECSO-FRCNN and MR-LSCNN algorithms, as the number of nodes increases to eight, the speedup ratio of the PDCNN-MI algorithm exceeds those of the other algorithms by 0.42, 0.57, and 0.69, respectively. However, when processing COCO, a dataset with a significantly larger data size, the speedup ratio gap between the algorithms becomes more pronounced, as depicted in Fig. 5D. When the number of nodes is two, the PDCNN-MI algorithm achieves speedup ratios of 0.01, 0.03, and 0.17. higher than the N-SAMCN, ECSO-FRCNN and MR-LSCNN algorithms. As the number of nodes increases to eight, the PDCNN-MI algorithm outperforms the other algorithms with speedup ratios of 0.48, 0.39, and 1.32 higher, respectively.

The reasons for the slowdown in the speedup ratio of N-SAMCN, ECSO-FRCNN, and MR-LSCNN algorithms as the number of parallel nodes increases are as follows: although these algorithms employ parallel frameworks combined with DCNNs for parallel computation, they fail to account for the multi-dimensional characteristics of big data environments. Consequently, the efficiency improvement of convolutional operations becomes less significant as the data scale grows and the number of parallel nodes increases. As a result, when the number of parallel nodes rises, the trend of parallel efficiency growth diminishes.

These results demonstrate that the PDCNN-MI algorithm exhibits superior parallel computing efficiency, mainly when processing large-scale datasets. This improvement can be attributed to two key factors. First, the PMT-IM strategy optimizes convolutional operations by calculating the covariance matrix and mean of all convolutional kernels in the convolutional layer at each computational node. It then constructs an objective function for the MDCV. By computing the second-order Taylor expansion of this objective function at its stationary points, the strategy iteratively determines the optimal MDCV value. This process effectively removes redundant convolutional kernels that correlate highly with others in the current convolutional layer. As a result, the strategy reduces redundant parameters in the network model, lowers computational load and storage requirements, and accelerates convolutional layer operations. Second, integrating the Im2col algorithm with the MapReduce computational framework enables parallel Im2col convolution. This transformation converts convolution operations into matrix multiplications, executed in parallel across multiple computational nodes. By leveraging this approach, the strategy significantly enhances the computational efficiency of the convolutional layer. Compared to traditional convolution methods, the PMT-IM strategy substantially reduces model training time while improving the overall performance of the algorithm. These optimizations collectively demonstrate the effectiveness of PDCNN-MI in handling large-scale datasets with improved speed and efficiency.