TbsNet: the importance of thin-branch structures in CNNs

The depth and width

Cybenko (1992) proposed that a single-layer neural network with Sigmoid activation can approximately represent any function with arbitrarily small error on the condition that the network must have enough width, which may increase the params and FLOPs greatly. Liang & Srikant (2016) proposed that the non-deep network needs more exponential neurons than the deep network to achieve the accuracy of function approximation, which is one of the main advantages of deep neural networks. With the emergence of ResNet (He et al., 2016b) with up to 1,000 layers, increasing the depth of the network to improve network performance has become a consensus. Surprisingly, a Transformer with more than 1,000 layers has also been designed and performs training stablely (Wang et al., 2022). However, the shallow network ParNet (Goyal et al., 2021) with only 12 layers has achieved success by increasing the width on a large scale, which has once again aroused the discussion of the network structure in the artificial intelligence community.

The depth and width of the network play an important part in scaling the size of params and FLOPs. For any input Features = (B, C_in, H, W), the shape of the convolution kernel tensor is Kernel = ((k_h, k_w), S, C_in, C_out), where B is the size of batch size, H and W are the height and width of the input feature map respectively, (k_h, k_w) represents the height and width of the convolution kernel respectively, S is the step size (stride), C_in and C_out represent the number of input channels and output channels respectively. C_in × k_h × k_w.

Kernel size and receptive field

The receptive field has a powerful influence on the performance of CNN. The success of Transformer in computer vision has inspired the artificial intelligence community to re-examine the shortcomings of convolutional neural networks (Dosovitskiy et al., 2021; Han et al., 2023; Lin et al., 2022; Liu et al., 2021b, 2022a). The self-attention mechanism of the Transformer has the ability of global perception, which exceeds the receptive field determined by the kernel size, which becomes one of the main gaps between the Transformer and CNN (Han et al., 2023). The remarkable advantage of the convolutional operator lies in its outstanding local perception ability, which uses a large convolution kernel to obtain a larger local receptive field, thus narrowing the gap between the convolutional operator and Transformer (Ding et al., 2022; Liu et al., 2022a).

The receptive field represents the range region of a specific CNN feature in the input space, and we can calculate the size by two factors (the center position and the feature map size) (Dang Ha The Hien, 2017). The calculation of receptive field size involves feature numbers (F) in each dimension, the size of the current receptive field (R), the distance between two adjacent features (J), and the center coordinate of the features (Cen). And Cen is taken to be the central coordinate of the receptive field. For any input feature F_in, the kernel size is k, the step size is s, the padding size is p, and the output feature is F_out,

(6) $$F_{out}={\displaystyle \lfloor \frac{F_{in}+2p-k}{s}\rfloor +1}$$

The distance between two adjacent features in the output feature map (J_out) is equal to the product of the distance step size between them,

(7) $$J_{out}=J_{in}\times s$$

The receptive field size RF_out is calculated by the formula as follows,

(8) $$R{F}_{out}=R{F}_{in}+\left(k-1\right)\times J_{in}$$

And the center position of the receptive field of the output features is calculated by the formula as follows,

(9) $$Ce{n}_{out}=Ce{n}_{in}+\left({\displaystyle \frac{k-1}{2}-p}\right)\times J_{in}$$

Formula (8) shows that the receptive field size is closely related to the kernel size. The kernel sizes commonly used are 3×3, 5×5, 7×7, and more large convolution kernels with the size of 9×9, 13×13, 31×31, etc. However, Formulas (2) and (5) also show that kernel size has a powerful impact on params and FLOPs. Even if large and super-large kernel sizes are more effective in some network structures, more applications of a large kernel are bound to cause more computation costs and make the network untrainable. An ingenious alternative is to replace the large kernel with multiple stacked Conv3×3 to obtain the equivalent receptive field size, which has a better performance in terms of computation, such as replacing a Conv5×5 with two Conv3×3 (Ioffe & Szegedy, 2015) and replacing a Conv7×7 with three Conv3×3 (Liu et al., 2021c, 2022a; Simonyan & Zisserman, 2015). There is a tradition of stacked Conv3×3 in classical networks, such as Vgg (Simonyan & Zisserman, 2015), ResNets (He et al., 2016a), ResNeXt (Xie et al., 2017), etc.

Luo et al. (2016) proposed the effective receptive field (ERF) theory at first. Luo et al. (2016) found that ERF is related to the theoretical receptive field, but not all pixels in the receptive field have the same contribution to the output vector. In many cases, the influence of pixels in the receptive field obeys the Gaussian distribution, the ERF is only part of the theoretical receptive field, and the Gaussian distribution attenuates rapidly from the center to the edge. In addition, the size of the ERF is also affected by the network structure. For example, although the residual network structure solves the problem of network degradation well, the Identity Map makes the ERF smaller (Luo et al., 2016).

Multi-branch structure

Researchers have never stopped making a full study of the network structure to improve network performance. The plain structure has the advantages of fast inferring and common cost, while the multi-branch structure can obtain multi-scale feature information. RepVgg (Ding et al., 2021) uses the multi-branch structure to improve the classical Vgg (Simonyan & Zisserman, 2015) network and achieves good performance. ParNet (Goyal et al., 2021) uses the double-branch parallel structure to verify that the shallow network can also play an excellent performance, and can greatly reduce the number of network parameters. Inception is a relatively early successful multi-branch structure, and the residual structure in ResNet (He et al., 2016a) is also a branch structure in nature.

However, adopting more branches in the network will harm performance (Cheng et al., 2020; Xu & McAuley, 2023). The results are mainly about two aspects. (1) The params are too large, and it is easy to produce an overfitting phenomenon. (2) It takes up more memory cost and inference time, which is not conducive to training the model.

As one of the optimization methods of multi-branch structure, reparameterization has attracted more and more attention in recent years. RepVgg (Ding et al., 2021) adopts the reparameterization method to speed up the inference speed. However, the inference model structure needs to be redesigned when using the reparameterization method, so reparameterization is limited and is not suitable for all architectures.

Strategies for constructing CNNs

In summary, we propose several strategies for convolutional neural network design. (1) Network scaling in three dimensions: depth, width, and feature graph size. Scaling the depth and width of the network has a significant impact on FLOPs and params, while choosing the right depth and width can improve network performance without significantly increasing computing costs. In addition, the size of the input feature map also has an important effect on the network performance. (2) To extract multi-scale feature information, branch structure is indispensable. In addition to multi-scale feature sampling, multi-branch structure also has feature alignment function during feature fusion. (3) The overall structure of the network should be kept plain to reduce feature redundancy and inference time. A multi-branch structure increases memory occupancy and inference time, while a plain structure has faster inference time and less memory search. (4) Lightweight operators such as pointwise convolution, depthwise convolution, asymmetric convolution, and group convolution can be adopted to reduce the params and FLOPs of the network. (5) Network design must pay attention to the size of the effective receptive field. The branch structure has a certain effect on the size of the effective receptive field. (6) The reparameterization method is not necessary for any architecture. The goal of the reparameterization method is to reparameterize the learning weight of the multi-branch structure network model into the weight of a plain structure used, but not every network model can be reparameterized.

TBS block

Although the residual structure (He et al., 2016b) solves the problem of network degradation very well, the Identity map makes the effective perception field smaller (Luo et al., 2016). To overcome the problem of receptive field degradation, TbsNet draws lessons from the success of Residual block and Inception in structure, adopts a double-branch structure similar to Residual block, studies and improves the Identity map and Bottleneck units in Residual block (Fig. 1).

Architecture

Overall structure

A thin-branch structure network (TbsNet) was constructed by stacking multiple TBS blocks (Fig. 2). TbsNet consists of one Stem module, four Stage modules, and one Classifier module, Stage = {S₁, S₂, S₃, S₄}. The Stem module contains only 1 Conv3×3+BN+ReLu, and its function is to expand the number of channels of the input feature map to 64, and whether to 2X downsampling is controlled by the step size. The Stage module consists of multiple TBS blocks, and only the last TBS block performs a judgment to expand the channels and 2X downsampling. The total number of TbsNet branches is four, which is much lower than other branch structure networks. The last TBS block of the ${\mathrm{S}}_1$ module controls whether to 2X downsampling by the parameter stride, ${\mathrm{S}}_2$, ${\mathrm{S}}_3$, and ${\mathrm{S}}_4$ complete 2X downsampling in the last TBS block. The Classifier module flattens the input feature information and outputs the result according to the number of classes.

TbsNet structure keeps plain on the whole, and the adjustable downsampling multiple can flexibly adapt to feature images with different resolutions, which enables TbsNet to have good scaling ability. Compared with ResNet (He et al., 2016b), 2X downsampling of TbsNet exists in the last TBS block, which could reduce the number of unnecessary branches.

Compared with RepVgg, Inception and other modules, we design modules with fewer branches, and compared with ResNet, our short-hop connection is downsampled only four times in four stages. The network architecture we designed is thin in general.

Conv3×3 vs Conv1×1

In the following experiments, 16 variant structures were designed (Table 3), and the performance of variant structures decreased overall when replacing Conv3×3 with Conv1×1 as a shortcut. The settings are as layers = [17, 23], “β” = [1, 0], width_list = {w₁, w₂, w₃, w₄}.

Some comparison between Conv3×3 and Conv1×1 is shown as follows. (1) Decrease in accuracy. Under the setting of the depth = 17 and 23, the max values of Acc_conv3×3 − Acc_conv1×1 were 0.46 and 0.34, the averages were 0.26 and 0.17% respectively. (2) The cut of params. The minimum of Params_conv3×3 − Params_conv1×1 is 2.25(M) and the max is 9.31(M). The max cut ratio was 54% and the average was 34% (Fig. 3). (3) The cut ratio of FLOPs. The max cut ratio of FLOPs is 48%, and the average was 29% (Fig. 3). We can see from the comparison of experiments on the CIFAR-10 (Krizhevsky, 2009) dataset that the performance of Conv3×3 is better than that of Conv1×1 (Table 3). However, Conv1×1 has advantages in params and FLOPs.

In general, the number of parameters params and FLOPs will increase as the depth and width increase. Since the number of shortcuts is unchanging in TbsNet, the proportion of params and FLOPs decreasing will show a decreasing trend with increasing network depth and width (Fig. 3). When the asymmetric convolution in the stack unit sets the number of groups (“set_group” = 1) (Fig. 1), the total number of params and FLOPs of TbsNet will be drastically cut, and the parameters of the shortcut unit are still unchanged. Thus, the parameter proportion of the shortcut in the whole network is improved. That is why the cut ratio of params and FLOPs is significantly higher when the asymmetric convolution in stack unit sets the number of groups.

To sum up, two conclusions can be summarized. (1) Conv1×1 has the advantage of decreasing the parameters and floating-point computations, but Conv3×3 outperforms Conv1×1 in general. (2) It is beneficial to use Conv1×1 to reduce the number of parameters and floating-point computations. Therefore, TbsNet retains Conv1×1 as the lightweight option but prefers to use Conv3×3 as the short-hop connection.

Asy vs GAsy

Replacing standard convolution with asymmetric convolution (Ronneberger, Fischer & Brox, 2015) can significantly reduce the number of parameters while maintaining stable performance (Ding et al., 2019). To further reduce the number of params and FLOPs, the groups of the first and second asymmetric convolution (Conv1×3 + Conv3×1) in stack unit (Fig. 1B) are assigned as 4 and 8, respectively. The settings of the network are as follows. shortcut = Conv3×3, Depth = depth_list, Width = width_list, β = [−1, 1], dataset = [CIFAR-10, CIFAR-100] (Krizhevsky, 2009). Selected expansion rate β = 1. Table 4 lists the Asy and GAsy results of experiments, where Asy denotes asymmetric convolution and GAsy denotes group asymmetric convolution.

We can see from the experimental data in Table 4 that compared with the Asy, the Top 1 Accuracy of the GAsy on CIFAR-10 and CIFAR-100 datasets (Krizhevsky, 2009) is reduced by 0.26 and 0.60 points on average, and the maximum reduction is 0.61 and 2.06 points. Of the 20 variants with different depths and widths, four variants improve their accuracy. It mainly appears in variant structures with larger network width, width = w₃ and w₄. Thus, it shows that adopting GAsy maybe decrease the performance compared to Asy overall. But in terms of parameters comparison, Params_GAsy is much lower than Params_Asy, in which the maximum reduction ratio reaches 63%, and the average reduction rate is 58%. As param’s decrease, the FLOPs decrease significantly.

Figure 4 shows the changing trend of Acc_GAsy and Acc_Asy. In Fig. 4, the vertical axis represents the value of GAsy-Asy, and the horizontal axis is the sequence of 20 variant structures for the combination of depth_list and width_list. Under the setting of β = −1, we observed that three values calculated by Acc_GAsy-Acc_Asy are higher than zero, and all of those are experimental results on CIFAR-10 (Krizhevsky, 2009); on the other hand, all values (Acc_GAsy-Acc_Asy) on CIFAR-100 (Krizhevsky, 2009) are less than zero (for the experimental results of β = −1, please refer to Appendix 1). By comparing Figs. 4A and 4B, we found that the value change has a two-point similarity. One is that the values (Acc_GAsy − Acc_Asy) are generally less than zero, and the other is that the values are gradually approaching zero with the increase of the depth and width of the network.

To sum up, two conclusions can be summarized. (1) GAsy will reduce the accuracy of the network, but with the increase of the depth and width of the network, the performance of GAsy will be gradually close to the Asy’s. (2) Although GAsy can decrease the params and FLOPs significantly, it can not be adopted freely because of the depth and width of the network. When the depth and width of the network are small, it is not a good time to set the number of groups for asymmetric convolution, but the opportunity is when the depth or width of the network is high.

According to the above conclusions, the TBS block retains the setting of the number of groups as an option. It is worth mentioning that group convolution and depthwise separable convolution may be affected by cuDNN and lead to increase training and inference time.

Scaling ability

The verification of the scaling ability lies in four aspects. (1) Network depth. The TBS blocks of each Stage are stacked to expand the depth of the network. (2) The width of the network. The widths of every stage are set according to the width_list. (3) Expansion and squeeze mechanism. The expansion coefficient β in the TBS block completes the scaling of the network width (4) Feature map scaling. The downsampling multiple is controlled by the stride, with a minimum of 8X and a maximum of 32X to adapt to feature maps of different sizes. The feature map size plays a crucial role in determining the scaling ability of the network. If the feature maps are too small, it would severely limit the network's ability to scale and tackle large-sized input images. In contrast, while larger feature maps can offer richer information, they can also make the network excessively complex and bloated, resulting in delayed training and prediction speeds, which may not be practical. In this experiment, the params, FLOPs, and accuracy (%) still indicate the scaling ability of TbsNet (Dollár, Singh & Girshick, 2021).

We selected a base model as the baseline, and the settings are as follows. Depth = 17, width = w₁, shortcut = Conv3×3, set_group = 1, β = −1. Depth_list = {17, 23, 29, 35, 44} and width_list = {w₁, w₂, w₃, w₄} are the depth and width lists for the scaling experiment. The number of parameters of the baseline is params = 3.65M, the number of floating-point calculations is PLOPs = 12.42G, the Top 1 Accuracy on CIFAR-10 (Krizhevsky, 2009) is Acc_CIFAR-10 = 95.49, and on CIFAR-100 (Krizhevsky, 2009) is Acc_CIFAR-100 = 78.31.

The maximum of width and depth

Increasing the depth and width of the network at the same time can further improve accuracy, but the increment in terms of params and FLOPs is signal. We find that expanding the width brings instability to the improvement of accuracy. There are four accuracy drops in Table 5 due to the width expansion.

Figures 5B–5F more intuitively shows the impact of expanding width on the three performance indicators of params, FLOPs, and accuracy. Under the setting of width = w₄, there were six accuracy drops and four raises. Four of the six accuracy drops occurred in experiments of CIFAR-10 datasets (Krizhevsky, 2009). A similar phenomenon occurs in the experiment with β = 1 (Table 4). Experiments show that the setting of width = w₄ cannot improve the performance of TbsNet on the CIFAR-10 and CIFAR-100 datasets (Krizhevsky, 2009), especially on the CIFAR-10 datasets (Krizhevsky, 2009).

In terms of network depth selection, Fig. 5A shows the effect of depth on network performance. As the depth increases, the rate of accuracy improvement gradually stabilizes while FLOPs still increase significantly, so layers = 44 is retained as the maximum depth, and no deeper network is constructed.

On CIFAR datasets

The CIFAR-10 dataset (Krizhevsky, 2009) consists of 60,000 32×32 color images in 10 classes, with 50,000 training images and 10,000 test images. Each image is labeled as one of 10 classes, and each class consists of 5,000 training images and 1,000 test images. The CIFAR-100 dataset (Krizhevsky, 2009) consists of 60,000 32×32 color images and has 100 classes.

Table 6 summarizes the performance of various networks on CIFAR-10 and CIFAR-100. On the CIFAR-10 dataset, Top 1 Accuracy exceeds ResNet50 (He et al., 2016b), ResNet152 (He et al., 2016b), ResNeXt101 (32×8d) (Xie et al., 2017), WideResNet (28, 10) (Zagoruyko & Komodakis, 2016), RepVGG-B2g4 (Ding et al., 2021), Parnet (Goyal et al., 2021) by up to 2.48 points, 1.67 points, 1.24 points, 1.02 points, 0.7 points, 0.9 points, respectively. On the CIFAR-100 dataset, TbsNet’s Top 1 Accuracy exceeds ResNet50 (He et al., 2016b), ResNet152 (He et al., 2016b), ResNeXt101 (32×8d) (Xie et al., 2017), WideResNet (28, 10) (Zagoruyko & Komodakis, 2016), RepVGG-B2g4 (Ding et al., 2021), Parnet (Goyal et al., 2021) by up to 7.97 points, 7.67 points, 2.72 points, 2.81 points, 1.84 points, 2.21 points, respectively.

From the above analysis, we sum up that TbsNet has the advantage of lower params and FLOPs, the performance of TbsNet is competitive with the most advanced CNNs, such as ResNet (He et al., 2016b), ResNeXt (Xie et al., 2017), and RepVgg (Ding et al., 2021), etc.

On Tiny-ImageNet dataset

Tiny-ImageNet (Le & Yang, 2015) has 200 classes, each class has 500 training images, 50 validation images, and 50 test images, and the size of the images is 64×64. The experimental environment is the same as the hyperparameter setting of the CIFAR dataset and training with the Vanilla (Abai & Rajmalwar, 2019) method.

We compared some previous state-of-the-art network architectures with TbsNet (Table 7). These architectures employ some training techniques such as AutoMix (Liu et al., 2022c), SAMix (Li et al., 2021), PuzzleMix+DM (Liu et al., 2022b), DCL (Luo et al., 2021), etc. Since no training techniques are employed (vanilla training scheme), the TbsNet results on Tiny-ImageNet (Le & Yang, 2015) are not very nice. Compared with these architectures TbsNet still shows excellent performance. The settings of the network model are β = −1, set_group = 0, and shortcut = Conv3×3.

On ImageNet-1K dataset

The ImageNet-1K dataset (Russakovsky et al., 2015) contains 14,197,122 labeled images. Since 2010, this dataset has been used in the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC), a benchmark for image classification and object detection (Russakovsky et al., 2015). The ImageNet-1K dataset is a sub-dataset used in the ISLVRC2012 competition (Hariharan & Girshick, 2017). The ImageNet-1K dataset (Russakovsky et al., 2015) has a total of 1,000 categories. The training set has 1,281,167 pictures, about 1,300 pictures in each category; the validation set has 50,000 pictures, 50 pictures in each category, and the test set has 100,000 pictures, 100 pictures in each category.

Employing different training schemes will yield different accuracies on the ImageNet-1K dataset (Wightman, Touvron & Jégou, 2021). Table 8 lists the weights obtained from Pytorch official ResNet50, MobileNetV2, and other network architectures under two different training schemes, namely IMAGENET1K_V1 and IMAGENET1K_V2 (Torch Contributors, 2017). The test accuracy of IMAGENET1K_V2 is higher than IMAGENET1K_V1. The accuracy obtained under the training scheme of Goyal et al. (2021) are between the IMAGENET1K_V1 and IMAGENET1K_V2. The training schemes are given by Liu et al. (2022a), Foret et al. (2020) are significantly better than the test accuracy of IMAGENET1K_V1 and even better than IMAGENET1K_V2’s.

We employed distributed multi-GPU and mixed-precision training schemes to train the model. Data enhancement techniques and learning rate adjustment strategies were selected (Wightman, Touvron & Jégou, 2021). The training program we adopted refers to the setting in Pytorch’s official blog “How to Train State-Of-The-Art Models Using TorchVision’s Latest Primitives” (Vryniotis, 2021), which is equivalent to IMAGENET1K_V2, so it is a relatively fair comparison to IMAGENET1K_V2.

The experimental data in Table 8 shows that TbsNet exceeds ResNet50_IMAGENET1K_V2, ResNet152_IMAGENET1K_V2, ResNeXt101_32×8D_IMAGENET1K_V2, by up to 5.31 points, 3.89 points, 3.34 points, respectively. Compared with the ParNet (Goyal et al., 2021) and ConvNeXt (Liu et al., 2022a), TbsNet exceeds ParNet by about four points and exceeds ConvNeXt_Large_IMAGENET1 by up to 1.3 points. Although The Top 1 Accuracy of TbsNet is equivalent to ConvNeXts (Liu et al., 2022a) and EfficientNet_V2_ IMAGENET1K_V1, TbsNet has more advantages in terms of params and FLOPs.