MLPruner: pruning convolutional neural networks with automatic mask learning

Network sparsity

By eliminating superfluous network weights (LeCun, Denker & Solla, 1989; Han et al., 2015; He, Zhang & Sun, 2017), network sparsity/pruning has evolved into a contemporary tool for acquiring lightweight sparse models. The practice of discarding individual weights at random positions, fine-grained sparsity, successfully boasts a high sparse ratio whilst maintaining performance assurance (Han et al., 2015; Evci et al., 2020). RigL (Evci et al., 2020) periodically alternates between the removal and reinstatement of weights, determined by magnitudes and gradients. Sparse Momentum (Dettmers & Zettlemoyer, 2019) takes into account the average momentum magnitude within each layer to reallocate weights. Regrettably, the consequent unstructured sparse weights struggle to effectuate acceleration on standard hardware. Unstructured sparsity proved successful in preserving performance, even with a high sparsity exceeding 95% (Liu et al., 2021). However, the ensuing irregular sparse tensors resulted in negligible speed improvements on common hardware (Wang, 2020). Coarse-grained sparsity, which typically removes an entire weight block (Ji et al., 2018; Meng et al., 2020) or convolution filter (Liu et al., 2019; Lin et al., 2020a), bears more hardware compatibility. Contrasted with fine-grained sparsity, the compressed model enjoys a substantial speed increase, albeit at the cost of considerable performance degradation. In this article, our focus lies on filter pruning, an aspect of coarse-grained sparsity, with the objective of simultaneously preserving the performance of DNN models and enabling hardware acceleration.

Filter pruning

Filter pruning eliminates entire convolutional filters from the initial CNNs, yielding a structured pruned model (Liu et al., 2017; Lin et al., 2020a; He et al., 2019; Lin et al., 2021). Filter pruning aims to uphold the original model’s performance while significantly decreasing both the FLOPs and parameter burden. It draws mounting attention for the practical compression effect as the compressed network receives adequate support from conventional hardware and readily accessible basic linear algebra subprograms (BLAS). The mainstram research on filter pruning bifurcates approximately into two distinct classes, as illustrated further.

The first group adheres to a three-step pruning pipeline, which encompasses pre-training the model, removing unimportant filters, and meticulously fine-tuning the resulting pruned model. Predominantly, most methodologies in this sphere tend to underscore the second stage, deploying a diverse array of filter importance estimation strategies. For instance, Li et al. (2017) elected to prune filters exhibiting smaller ${\ell }_1$ norm values. He et al. (2019) considered the next layer’s construction error as a pivotal criterion and proceeded with layer-by-layer pruning. Wang et al. (2019) employed attention modules to derive the scaling factor for each filter and leveraged it as a benchmark for determining filter importance. Despite their innate simplicity, these hand-crafted benchmarks for the quantification of filter importance frequently stumble in accurately eliminating filters integral to the comprehensive performance.

Contrarily, the secondary category catalyzes filter pruning by training the network, supplementing the sparse constraints on individual filters (Huang & Wang, 2018; Luo & Wu, 2020). As a result, the pruned model becomes available through the removal of zero-valued filters or those falling below a specified threshold. For instance, Huang & Wang (2018) presented a scaling factor to augment the output of a defined structure, thereby enhancing sparsity on such factors. Luo & Wu (2020) incorporated an “automatic pruner” layer into the convolutional layer to execute filter pruning autonomously. Xiao, Wang & Rajasekaran (2019) pruned CNN models utilizing gradient-based update rules. Tang et al. (2021) further delved into prevalent information to dynamically identify filter redundancy within CNNs. However, these methods routinely interfere with the standard training procedure of the network due to the penalties imposed on weight norms, leading to sub-optimal outcomes. Our proposed MLPruner aims to execute filter pruning during training without incurring any weight penalty, thus alleviating the restrictions imposed by the mentioned sets of pruning methods.

Besides, an emerging class of methods primarily focuses on the architecture of pruned networks, specifically the pruning rate of each layer (Liu et al., 2019; He et al., 2018b; Lin et al., 2020c). For instance, He et al. (2018b) proposed utilizing reinforcement learning to automate the search process. Similarly, Liu et al. (2019) pre-trained a PruningNet to predict the weights of potential networks and incorporated an evolutionary algorithm to scout for the finest candidate. In addition, several methodologies automatically identify optimal network architectures by designing differentiable operators (Ning et al., 2020; Guo et al., 2020; Li et al., 2023). For instance, DMCP models channel pruning as a differentiable Markov process to learn the significance of each layer. Our proposed MLPruner is orthogonal to these architecture search-based methods. To explain, MLPruner can apply a global pruning based on the learned masks to automatically determine the pruning rate for each layer or directly utilize the structure obtained from existing architecture search-based methods and perform local pruning from each layer.

Recent surveys (Verma et al., 2024) comprehensively analyze the evolution from static to dynamic neural architectures. Gao et al. (2024) proposed a unified dynamic and static channel pruning method based on two-layer optimization (BilevelPruning/UDSP), which evaluates the static subnetwork and jointly learns the channel selection of both from dynamic subnetwork to end-to-end, which retains the storage-saving advantage of static pruning and compensates for the performance constraints of the fixed subnetwork with the help of dynamic pruning, and at the same time, directly sets the parameter and computational constraints without manually fusing the functions. The parameters and computational constraints are set directly, eliminating the need for manual fusion functions. In parallel, some recent works attempt to unify pruning with architectural search or dynamic mask generation. Wu et al. (2024) proposed Auto-Train-Once (ATO), a single-shot framework that uses a controller network to dynamically generate pruning masks. ATO removes the need for fine-tuning and comes with convergence guarantees. Our MLPruner contributes to this direction by incorporating dynamic mask learning into training while maintaining a static, hardware-efficient architecture for deployment. Our MLPruner contributes to this evolving landscape by incorporating dynamic mask learning into the pruning process, while retaining a static and hardware-friendly deployment structure.

Background

We first describe necessary preliminaries for CNNs filter pruning. Denote an L-layer CNN model’s parameters as $\mathbf{W}=\{{\mathbf{W}}^1,{\mathbf{W}}^2,{\mathbf{W}}^3,...,{\mathbf{W}}^L\}$, the convolution kernel weights of a distinct layer $l$ can be represented as ${\mathbf{W}}^l\in {\mathbb{R}}^{C_{out}^l\times C_{in}^l\times K^l\times K^l}$, where $C_{out}^l,C_{in}^l,K^l$ symbolize the number of output filters, input filters, and the kernel size, respectively. Let $I^l\in R^{B\times C_{in}^l\times H^l\times W^l}$ be the input feature map for layer $l$. Here, B acts as the batch size, $H^l$ and $W^l$ represent the dimensions of the input image. The output from layer $l$ is predicated upon:

(1) $${\mathcal{O}}^l={\mathcal{I}}^l⊛{\mathrm{W}}^l,$$where the convolution operator is represented as $⊛$¹ . Considering a batch of training image set X paired with class labels set Y, the training loss manifests as:

(2) $$\underset{\mathbf{W}}{min}\mathcal{L}(X,\mathbf{W};Y),$$where $\mathcal{L}(\cdot )$ operates as the loss function, usually taking the form of cross-entropy loss. For filter pruning, a subset of output filters in ${\mathbf{W}}^l$ is excised to yield the pruned kernel ${\hat{\mathbf{W}}}^l\in {\mathbb{R}}^{{\hat{C}}_{out}^l\times {\hat{C}}_{in}^l\times K^l\times K^l}$, subject to ${\hat{C}}_{out}^l\le C_{out}^l$ and ${\hat{C}}_{in}^l\le C_{in}^l$ constraints. It is worthwhile to highlight that the associated input channels of ${\mathbf{W}}^{l+1}$ are also expelled. We can hence transpose Eqs. (1) and (2) for the pruned network as follows:

(3) $${\hat{\mathcal{O}}}^l={\hat{\mathcal{I}}}^l⊛{\hat{\mathrm{W}}}^l,$$

(4) $$\underset{\hat{\mathbf{W}}}{min}\mathcal{L}(X,\hat{\mathbf{W}};Y).$$

However, restoring the performance of pruned models, i.e., minimizing Eq. (4), is challenging due to the elimination of pre-trained weights. Early research often utilized various criteria to remove insignificant filters, such as ${\ell }_1$-norm (Li et al., 2017), geometric data (He et al., 2019), and activation sparsity (Hu et al., 2016). Recent research have developed training-aware pruning methods have been proposed to automatically identifying non-essential filters (Ding et al., 2018; Zhang et al., 2022). This class of methods typically achieves filter pruning by gradually penalizing some filter weights during the training process via the imposition of ${\ell }_2$ regularization constraints on the original weights. However, such regularization constraint may affect the normal training objective in Eq. (4), thereby influencing the final performance of the pruned network.

MLPruner

In this article, we present MLPruner to mitigate the aforementioned issue, with its core innovation falls into optimizing a learnable mask $\mathbf{M}=\{{\mathbf{M}}^1,{\mathbf{M}}^2,{\mathbf{M}}^3,...,{\mathbf{M}}^L\}$ to automatically locate essential filters without weight penalty. The framework of MLPruner is illustrated in Fig. 1. In the following, we particularly introduce the optimization process for $\mathbf{M}$, as well as validate how the learned mask aptly reflects the corresponding filter’s significance.

Mask forward. During forward propagation within a specific layer $l$, we leverage the mask ${\mathbf{M}}^l$ to obtain a binary indicator ${\mathbf{B}}^l\in \{0,1{\}}^{C_{out}^l\times C_{in}^l\times K^l\times K^l}$ that guides the removal or preserve of corresponding filters. This is achieved using a pre-defined pruning rate $p^l$ to round ${\mathbf{M}}^l$ to 0 or 1 s, where 0 indicate pruning corresponding weights, and vice versa. Formally, we first accumulate ${\mathbf{M}}^l$ in a filter-wise manner as:

(5) $$\widehat{{\mathbf{M}}_i^l}=||{\mathbf{M}}_{i,:,:,:}^l|{|}_1,i=1,2,...,C_{out}^l.$$

Then, the binary indicator is derived as

(6) $${\mathbf{B}}_{i,:,:,:}^l=\{\begin{array}{c}0,\mathrm{i}\mathrm{f}\widehat{{\mathbf{M}}_i^l}\mathrm{i}\mathrm{n}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{t}\mathrm{o}\mathrm{p}\text{-}{p}^l\mathrm{s}\mathrm{m}\mathrm{a}\mathrm{l}\mathrm{l}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{v}\mathrm{a}\mathrm{l}\mathrm{u}\mathrm{e}\mathrm{s}\mathrm{o}\mathrm{f}{\hat{\mathbf{M}}}^l,\hfill \\ 1,\mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}.\hfill \end{array}$$

Finally, we use the binary indicator to prune the dense weights during forward propagation as:

(7) $${\mathcal{O}}^l=\mathcal{L}⊛({\mathbf{B}}^l\odot {\mathbf{W}}^l),$$where $\odot$ denotes the point-wise multiplication.

Mask backward. The above forward phase enables automatic filter pruning by looking at value of mask ${\mathbf{M}}^l$, yet Eq. (7) is non-differentiable due to the round operation, making the mask unable to be updated, i.e., the gradient can not passed from ${\mathbf{B}}^l$ to ${\mathbf{M}}^l$. To address this, we use the straight-through estimator (STE) (Bengio, Léonard & Courville, 2013) as an alternative to approximate the mask gradient of ${\mathbf{M}}^l$ as:

(8) $$\begin{array}{rl}\frac{\mathrm{\partial }\mathcal{L}}{\mathrm{\partial }{\mathbf{M}}^l}=& \frac{\mathrm{\partial }\mathcal{L}}{\mathrm{\partial }({\mathbf{B}}^l\odot {\mathbf{W}}^l)}\frac{\mathrm{\partial }({\mathbf{B}}^l\odot {\mathbf{W}}^l)}{\mathrm{\partial }{\mathbf{B}}^l}\frac{{\mathbf{B}}^l}{{\mathbf{M}}^l}\\ \approx & \frac{\mathrm{\partial }\mathcal{L}}{\mathrm{\partial }({\mathbf{B}}^l\odot {\mathbf{W}}^l)}\frac{\mathrm{\partial }({\mathbf{B}}^l\odot {\mathbf{W}}^l)}{\mathrm{\partial }{\mathbf{B}}^l}\cdot \mathbf{1}\\ =& \frac{\mathrm{\partial }\mathcal{L}}{\mathrm{\partial }({\mathbf{B}}^l\odot {\mathbf{W}}^l)}{\mathbf{W}}^l\end{array}.$$

In this manner, the mask can be jointly optimized with the weight during training. Notably, such mask learning process brings no gradient distortion to the weights as traditional training-aware pruning methods do. Furthermore, such learned masks can well tell the importance of corresponding filters, which we analyze below.

Tractability of optimization. We take a specific filter ${\mathbf{W}}_{i,:,:,:}^l$ to show that our learned mask score can well reflect the relative importance of filter weights.

Omitting the update schedule including learning rate and momentum, we can represented the learned mask ${\mathbf{M}}_{i,:,:,:}^l$ as

(9) $${\mathbf{M}}_{i,:,:,:}^l\approx \sum _{t=1}^T\frac{\mathrm{\partial }{\mathcal{L}}^t}{\mathrm{\partial }{({\mathbf{B}}^l\odot {\mathbf{W}}^l)}_{i,:,:,:}}{\mathbf{W}}_{i,:,:,:}^l,$$where T denotes the total training iterations.

Now, let us turn our focus to the loss variation $\mathrm{\Delta }\mathcal{L}({\mathbf{W}}_{i,:,:,:}^l)$ when removing ${\mathbf{W}}_{i,:,:,:}^l$. It can be derived using the Taylor expansion (Molchanov et al., 2017) as

(10) $$\begin{array}{rl}\mathrm{\Delta }\mathcal{L}({\mathbf{W}}_{i,:,:,:}^l)& =\mathcal{L}({\mathbf{W}}_{i,:,:,:}^l=0)-\mathcal{L}({\mathbf{W}}_{i,:,:,:}^l)\\ & \approx \mathcal{L}({\mathbf{W}}_{i,:,:,:}^l)-{\mathbf{W}}_{i,:,:,:}^l\frac{\mathrm{\partial }\mathcal{L}}{\mathrm{\partial }({\mathbf{W}}_{i,:,:,:}^l}\\ & +R_1({\mathbf{W}}_{i,:,:,:}^l=0)-\mathcal{L}({\mathbf{W}}_{i,:,:,:}^l)\\ & \approx -{\mathbf{W}}_{i,:,:,:}^l\frac{\mathrm{\partial }\mathcal{L}}{{\mathbf{W}}_{i,:,:,:}^l}\end{array},$$which leads Eq. (9) to

(11) $${\mathbf{M}}_{i,:,:,:}^l=\sum _{t=1}^T\mathrm{\Delta }\mathcal{L}({\mathbf{W}}_{i,:,:,:}^l).$$

Here we can observe that the trained mask ${\mathbf{M}}_{i,:,:,:}^l$ is the accumulation of loss change for removing weights ${\mathbf{W}}_{i,:,:,:}^l$. A small score denotes less loss increase thus the corresponding candidate can be safely removed. Thus, our introduced mask learning can be used to measure the relative importance between convolution filters, and preserving filters with larger mask value can well decrease the loss after pruning. The learned masks directly correlate with filter importance through the accumulated loss gradient Eq. (9). Unlike weight-penalty methods that distort gradients via L2 regularization, MLPruner’s masks provide unbiased importance estimates while preserving original weight optimization.

The overall pipeline of MLPruner is outlined in Algorithm 1. In particular, we set a mask training epochs $\mathcal{T}$ to learn the masks for filter pruning. After that, we stop optimizing masks and obtain the pruned weight $\hat{\mathbf{W}}$ by extracting the corresponding filter from $\mathbf{W}$ according to the binary indicator $\mathbf{B}$. Finally, we fine-tune the pruned model to further recover its performance. It can be inferred that MLPruner does not bring any weight penalty during training, but automatically learns masks that can well reflect the importance of filters with respect to the training loss, achieving holistic filter pruning in an end-to-end manner. MLPruner fundamentally differs from weight-penalty approaches in three ways: (1) Our masks provide unbiased importance estimates without distorting weight gradients as ${\mathbf{M}}_i^l$ directly measures ${\mathbf{W}}_i^l$’s true contribution to loss minimization, uncontaminated by auxiliary penalties, (2) The pruning decision is based on accumulated loss impact rather than instantaneous magnitude, and (3) We maintain the original training objective without additional regularization terms, delivering better generalization ability as weight updates follow the original loss gradient ${\mathrm{\nabla }}_{\mathbf{W}}\mathcal{L}$ (Eq. 10), maintaining the model’s intrinsic generalization dynamics.

Remark. In our description of MLPruner, we take the layer-wise pruning rate $p_l$ to be a known quantity. In practical scenarios, we can employ global sorting for the learned mask to acquire the binary indicator for each layer. In such case, the layerwise pruning rate is automatically derived, thereby showcasing the ease-of-use proffered by MLPruner. Furthermore, it is also worth noting that the existing structure-searching pruning methods (Lin et al., 2020c; Liu et al., 2019) hold orthogonality to MLPruner, which means that we could utilize the pruning rates identified from each layer to locally sort the mask and obtain the binary indicator. We manifest the comparative performance achieved through diverse layerwise pruning rates in the experiment section.

Experimental settings

Datasets and networks. We conducted experiments on representative image classification datasets CIFAR-10, CIFAR-100 (Krizhevsky & Hinton, 2009), and ImageNet (Deng et al., 2009) to demonstrate the effectiveness of the proposed MLPruner. CIFAR-10 is comprised of 60,000 distinct 32 × 32 color images, spread evenly across 10 various classes, with each class housing 6,000 images. The CIFAR-100 dataset, designed parallel to CIFAR-10, departs only in its increased class count, featuring 100 distinct classes, each containing 600 images. We elect representative convolutional networks to validate the efficacy of MLPruner, including VGGNet-16 (Simonyan & Zisserman, 2015), GoogLeNet (Szegedy et al., 2015), ResNet (He et al., 2016), and MobileNet-V2 (Sandler et al., 2018).

Implementation details. We set the mask training epochs $\mathcal{T}=30$ for CIFAR-10/100 and $\mathcal{T}=10$ for ImageNet, with all masks initialized to 1s. After training masks, we directly conduct global sort on the masks to obtain layer-wise pruning rates and then prune filters based on the learned masks. All our pruned models are fine-tuned employing the stochastic gradient descent (SGD) optimizer, paired with a momentum of 0.9 and a batch size of 256. On CIFAR-10, Each pruned network is fine-tuned for 300 epochs, incorporating a weight decay factor of $5\times$10⁻³ and an initial learning rate of 0.1. This rate undergoes a subsequent reduction to 0.01 and then to 0.001 after specific intervals of 50 and 100 epochs respectively. On ImageNet, we give 90 epochs for tine-tuning with weight decay of $1\times$ 10⁻⁴. The learning rate is initialized to 0.1 and decayed by cosine annealing schedule. Besides, we employ random crop and horizontal flip to the input images. All experiments is implemented based on PyTorch and executed on NVIDIA 3090 GPUs.

Performance metrics and baselines. MLPruner is juxtaposed with several state-of-the-art pruning methods (He, Zhang & Sun, 2017; Huang & Wang, 2018; Li et al., 2017; Lin et al., 2020a, 2020c, 2019; Yu et al., 2018; Zhao et al., 2019). We report the top-1 accuracy to facilitate a quantitative comparison among various methods. The computing costs and storage demands are reflected in the number of FLOPs and parameters, accompanied by their corresponding pruning rates (PR), forming the basis for assessing the efficacy of our MLPruner and other methods used for comparison.

CIFAR-10

VGGNet-16. We first incorporate our MLPruner to prune the 16-layer VGGNet model, a classic CNN comprising 13 sequential convolutional layers and 3 fully-connected layers. Evident from Table 1, MLPruner markedly surpasses the current state-of-the-art methods across all performance indicators. Our MLPruner accomplishes a parameter compression of $89.1\mathrm{\%}$, thereby enhancing the computation by 71.9% along with a negligible accuracy decrease of 0.03%. This substantial improvement proves instrumental in deploying the VGGNet model on resource-restricted devices.

GoogLeNet. We further demonstrate the effectiveness of MLPruner in pruning GoogLeNet, a prevailing network with inception structure. Table 2 substantiates that notwithstanding marginal drops in top-1 accuracy (94.78% for MLPruner vs. 95.03% for the baseline), MLPruner significantly curtails the FLOPs by 62.5% and parameters by 61.3%. When juxtaposed with the superior state-of-the-art, i.e., HRank, MLPruner not only surpasses in accuracy performance but also substantially reduces the quantity of FLOPs and parameters. Consequently, MLPruner excels in diminishing the superfluity of networks with intricate multi-branch structures.

ResNet. We also evaluate the network pruning performances of various methods on ResNet (He et al., 2016), a predominant deep CNN with residual modules, as shown in Table 3. Notably, our MLPruner enhances the original ResNet-56 performance by 0.05%, successfully eliminating approximately 54.8% of FLOPs. This efficiency contrasts sharply with other methods, which invariably experience an accuracy reduction to varying degrees, despite achieving fewer reductions in FLOPs. Furthermore, our MLPruner exhibits remarkable superiority when applied to ResNet-110. Despite a robust 65.8% reduction in FLOPs, it generates a performance improvement of 0.08%, significantly outpacing competing methods.

MobileNet. MobileNet-v2 (Sandler et al., 2018), an advancing network, exemplifies compact design via its depth-wise separable convolution. Given its minuscule computational cost, pruning MobileNet-v2 proves to be an exceptionally challenging task. Nevertheless, as illustrated in Table 4, in comparison with its competitors, MLPruner impressively maintains a superior top-1 accuracy of 95.31%, whilst achieving a more significant pruning of FLOPs by 31.5%. Therefore, the results underscore the notable preeminence of our MLPruner in CNN pruning, unflinchingly handling even highly compact network models.

CIFAR-100

We further demonstrate the efficacy of MLPruner for pruning VGGNet-19 on the more complex dataset CIFAR-100. Table 5 shows the comparative performance of MLPruner under analogous pruning rates vs. other methodologies. Observable from these comparisons, MLPruner outperforms the state-of-the-art (SOTA) alternatives across varying pruning rates. Emphasizing further, MLPruner successfully minimizes the FLOPs to a fraction of 70.7%, simultaneously achieving a top-1 accuracy of 73.45%. Conversely, the recent SOTA, HRank (Lin et al., 2020a), demands greater computation to yield subpar top-1 accuracy at 72.91%. Undeniably, these results accentuate the distinct advantage of employing our MLPruner in pruning CNNs on more challenging datasets.

ImageNet

In the challenging ImageNet dataset, we compared MLPruner with current state-of-the-art pruning technologies. Key performance indicators, as observed in Table 6, reveal that MLPruner demonstrates a significant advantage in many core metrics. Firstly, MLPruner achieved a Top-1 accuracy of 75.62%, which not only surpasses most of the listed methods such as White-Box, DSA, and CLR-RNF—with accuracies of 75.32%, 75.10%, and 74.85%, respectively—but also shows considerable improvements over ThiNet at 72.04% and GAL at 71.95% Top-1 accuracy. This enhancement is particularly crucial as it demonstrates the capability to balance the reduction of computational resources while maintaining model performance during the pruning process. Further, MLPruner outperforms EagleEye by 1.2% accuracy at similar FLOPs because: (1) Our mask learning adapts to training dynamics rather than using fixed heuristics, (2) Joint optimization prevents the greedy local decisions EagleEye makes, and (3) The gradient-based importance measure better preserves critical filters.

Moreover, in terms of computational efficiency, MLPruner’s performance is equally impressive. It reduced the total number of floating-point operations (FLOPs) by 46.0%, achieving only 2.21G. This figure is not only lower than the baseline model’s 4.10G but also slightly better compared to other methods like DSA’s 2.46G and CLR-RNF’s 2.44G, indicating that MLPruner effectively lessens the computational load while preserving a relatively high accuracy. To sum up, Compared to other pruning methods, MLPruner not only reduces parameters more efficiently but also maintains better performance, even on more challenging dataset.

We conclude with a discussion on the pruning efficiency. Traditional methods, such as ThiNet and HRank, typically determine layer-wise pruning rates through manual design, which incurs significant human effort. On the other hand, search-based approaches, such as EagleEye and ABCPruner, often employ evolutionary algorithms or specific evaluation metrics to explore optimal architectures, which still demand substantial computational time. In contrast, our proposed MLPruner achieves automatic mask training with only one-tenth of the fine-tuning cost, enabling end-to-end optimization of both layer-wise pruning rates and critical filter identification, thereby enhancing pruning efficiency compared to existing methods.

Performance analysis

In this section, we prune ResNet-56 and test its performance on CIFAR-10 as an example to analysis the performance of MLPruner w.r.t. its different components.

Mask learning epochs. Initially, we scrutinize the import of learning epochs $\mathcal{T}$ for masks. Figure 2 delineates the performance nuances when utilizing varying epochs at diverse pruning ratios. Manifestly, enhancing the epochs of mask learning significantly augments the model’s efficacy, particularly under conditions of high sparsity. This phenomenon is intuitively understandable as a higher sparsity demands the removal of multiple filters, thereby necessitating an amplified number of mask training epochs to discern the priority between diverse filters. Concurrently, it should be noted that the performance gains of increasing epochs possess a ceiling effect, and an abundance of epochs can incur inflated training overhead. Therefore, we configure the mask training for 30 epochs.

Layerwise pruning rate. We further investigated the impact of layerwise pruning rate on MLPruner’s performance. Table 7 exemplifies the effect on FLOPs reduction and overall performance when employing different strategies to distribute the layerwise pruning rate. Interestingly, direct global ranking based on the learned mask can even yield better results than a manually set procedure, which underscores MLPruner’s efficiency. Furthermore, the use of a searched layerwise pruning rate can lead to enhanced performance, albeit at increased costs. This demonstrates MLPruner’s scalability, remaining orthogonal to the pruning methods predicated on architecture search.

Weight penalty. At last, we examine the impact of weight penalties on mask training within MLPruner, a widely adopted approach in training-aware filter pruning methods (Huang & Wang, 2018; Xiao, Wang & Rajasekaran, 2019). Specifically, we imposed an L1 penalty on the mask during its training course, with results displayed in Table 8. As can be seen, the introduction of a sparsity regularization term resulted in performance degradation. This can be attributed to the fact that our mask training strategy already accurately identifies critical filters, and the penalty item inevitably hinders the optimization of the mask.