Enhanced mechanisms of pooling and channel attention for deep learning feature maps

FMAPooling

The proposal FMAPooling is introduced in this section. At first, the both pooling functions are deeply studied.

Max pooling is the most widely applied approach. The function takes the maximum pixel value of the batch selected, i.e., a group of pixels determined by the max-pooling kernel size. It recognizes and retains the most prominent characteristics of the feature maps. Average pooling takes the average pixel value of the batch selected, which results in smoothing out the feature maps. The mathematical formulas for the two pooling functions are as follows:

(1) $$P_{Max}=Ma{x}_{i,j=0,0}^{i,j=h,w}\left\{x_{i,j=0,0},x_{i,j=0,1},x_{i,j=1,0},...,x_{i,j=h,w}\right\}$$

(2) $$P_{Avg}={\displaystyle \frac{1}{h\times w}\sum _{i,j=0,0}^{h,w}{x}_{i,j}}$$where $P_{Max}$ and $P_{Avg}$ denote the max pooling and average pooling operation on a single group of data in the feature maps determined by the pooling kernel; $h\times w$ denotes the kernel size for pooling operation, which is up to the case. In general, the kernel size is less than the size of input feature maps $H\times W$ and is set to $2\times 2$; $i\in [0,h]$ and $j\in [0,w]$; $Max\{\ast \}$ denotes the max value of the data group $\left\{x_{i,j=0,0},x_{i,j=0,1},x_{i,j=1,0},...,x_{i,j=h,w}\right\}$; $x_{i,j}$ denotes the pixel element of the $i_{th}$ column and the $j_{th}$ row determined by the pooling kernel $h\times w$; Furthermore, $P_{Max}\{\ast \}$ and $P_{Avg}\{\ast \}$ denote the pooling operation on the whole input feature maps.

It is hard to say which pooling performs better than another. Generally, it is up to the exact tasks (Yu et al., 2014). For clarity, we made an experiment to provide a quite intuitive and concise illustration of the both pooling functions extracting features from two different inputs, as is shown in Fig. 1. The task is to recognize the circle in the images. With the pooling operation:

• Cases with inputs like feature map A: the informative features vanish greatly for the output of max pooling; And the average pooling retains the most of the valuable features;

• Cases with inputs like feature map B: it is obvious the max pooling outperforms average pooling with more clearer features extracted from the original images.

Conclusions could be drawn that the max pooling extracts features better for images in which the background of the images is dark and the features located on the lighter pixels of the images, like MNIST dataset which corresponds to the case of input feature map B. However, for the cases of input like feature map with the background of the images is white the features located on the darker pixels, max pooling is impotent and the average pooling performs better. Actually, the tasks of DNNs are quite complicated rather than black-white or gray images. For example, the datasets like ImageNet, CIFAR10/100, etc, the informational features need to be extracted from rather complex images. Then, a single max pooling or average pooling inevitably damages the useful features in the images. Thus, we propose the fused max-average pooling (FMAPooling) functions to enhance the representations of the feature maps after being down-sampled by pooling operation.

The structure of the proposal FMAPooling is shown in Fig. 2:

The mathematical function of FMAttn is as follows.

(3) $$F_O=Con{v}_{pw}\left(F_{cat}\left(P_{Max}\left\{F_{IN}\right\},P_{Avg}\left\{F_{IN}\right\}\right)\right)$$where $Con{v}_{pw}$ denotes the point-wise convolution; $F_{cat}$ denotes the concatenation operation; $F_{IN}$ and $F_O$ denote the input and output feature maps of FMAttn.

The FMAttn consists of three steps. First, the input feature maps are processed with max pooling and average pooling respectively, with the parameters h and w of function $P_{Max}$ and $P_{Avg}$ both being set to 2 in general, which is less than the size $H\times W$ of input feature maps. Second, the two pooling outputs are concatenated together resulting in channels doubled. Third, the concatenated feature maps are convoluted with pointwise convolution to fuse the both features and reduce the doubled dimensionality by half as well as the introduced computation overhead by concatenation at the same time. In addition, the pointwise convolution is also followed by a batch normalization (BN) layer and a ReLU layer. Then, with the concatenation and the pointwise convolution operations, the both pooling features are merged and then the features are enhanced.

FMAttn

For the channel attention mechanism, Woo et al. (2018) have proposed the convolutional block attention module (CBAM) which adopts the channel attention method CAM by utilizing both the max pooling and average pooling in the stage of channel attention. However, the CAM processes the two outputs generated by the two pooling functions with the shared MLP respectively at first. Second, the both pooling features are simply added together which is then denoted as MP_features. Finally, the original feature maps are multiplied by MP_features and then the channel-attention feature maps are obtained. The method CAM achieves better performance compared with the SENet and ECANet. Furthermore, we propose a novel mechanism for channel attention which is defined as FMAttn by utilizing the max pooling and average pooling. The structure of FMAttn is detailed in Fig. 3.

The mathematical function of FMAttn is as follows.

(4) $$C{A}_{Max}=S\left(MLP\left(P_{Max}\left\{F_{IN}\right\}\right)\right)$$

(5) $$C{A}_{Avg}=S\left(MLP\left(P_{Avg}\left\{F_{IN}\right\}\right)\right)$$

(6) $$F_O=Con{v}_{pw}\left(F_{cat}\left(C{A}_{Max}\otimes F_{IN},C{A}_{Avg}\otimes F_{IN}\right)\right)$$where S denotes the sigmoid function; h and w, the parameters of $P_{Max}$ and $P_{Avg}$ which are presented in Eqs. (1) and (2), equal the feature map size H and W respectively in Eqs. (4) and (5); MLP denotes the multi-layer perceptron with two convolutional layers followed by Hardtanh function respectively; $C{A}_{Max}$ denotes the channel attention with max pooling, and the generated features by are denoted as $C{A}_{Max}:\left\{m_1,m_2,m_3,...,m_k\right\}$; $C{A}_{Avg}$ denotes the channel attention with average pooling, and the generated features are denoted as $C{A}_{Avg}:\left\{a_1,a_2,a_3,...,a_k\right\}$; k equals the channels C of input feature maps; $F_{cat}$ denotes the concatenation operation; $\otimes$ denotes the convolution which is pointwise and depthwise in this study; $Con{v}_{pw}$ denotes the point-wise convolution which reduce the dimensionality by half; And $F_O$ denotes the output feature map of the proposal FMAttn.

Differing from the excitation structure adopted in SE block, the feature dimensions for MLP remains the same as the original input without reducing the dimensions, as is shown in Fig. 4. The channel weights based on max pooling channel attention and average pooling channel attention, i.e., $C{A}_{Max}$ and $C{A}_{Avg}$, are generated respectively at first. And then, $C{A}_{Max}$ and $C{A}_{Avg}$ convolve with the input feature maps respectively to generate the self-attention feature maps. Finally, the two groups of feature maps are concatenated which is followed by the pointwise convolution to fuse and enhance the feature map representations further. The mechanism is basically different from the channel attention proposed in previous methods and has better representation properties.

Preliminary

The hardware platform for the experiments: Intel(R) Core(TM) i9-10900 CPU@2.80 GHz, 4 × 16 GB DDR4 main memory, and GeForce GTX 3080 Ti with CUDA version 11.4. The software environment is the PyTorch of version 1.9.0+cu102 on Ubuntu 20.04.

For the training settings, the optimizer is the momentum-accelerated stochastic gradient descent (SGD) (Sutskever et al., 2013), with the momentum of 0.9, weight $\mathrm{\_}$decay of 0.0001, nesterov. The training epochs are set to 155. It needs to be noted that the cosine annealing schedule is adopted for the learning rate strategy, which is defined as follows in the experiments (Loshchilov & Hutter, 2017).

(7) $$lr(n)=l{r}_{min}+{\displaystyle \frac{1}{2}(l{r}_{max}-l{r}_{min})(1+cos({\displaystyle \frac{T_{cur}}{T_i}\pi ))}}$$where lr(n) denotes the learning rate of the $n^{th}$ training epoch; $l{r}_{max}$ and $l{r}_{min}$ denote the initial and the minimum learning rate which are set to 0.01 and 1e−5 in this article; $T_i$ denotes the number of iterations for the $i_{th}$ restart, and $T_i\in \left\{5,10,20,40,80\right\}$; $T_{cur}$ denotes the account of epochs performed since the $i_{th}$ restart. The experimental results consist of two sections: the validation accuracies of training with 155 epochs once and twice. This provides us with a more comprehensive and profound understanding of the performance of the FMAPooling and FMAttn. As the twice training improves the validation accuracy in general which is also proved in the following experiments, the section provides both the experimental results of the once training and the twice training. For the analysis of the experimental results, we focus on the data of the twice training.

Experiments-VGG

For datasets CIFAR10/100 and ImageNet100, the experiments take two VGG network structures being different in number of layers as well as the input/output sizes. In addition, five strategies are established as follows to conduct comparison experiments in this section:

• VGG_A: the baseline VGG network that without channel-attention mechanism and adopts simply the max pooling layer after each conv-x layer;

• VGG_B: the model that takes the proposed standard FMAPooling module to replace the original max pooling layers of the network;

• VGG_C: the control network especially for VGG_B that replace the avg pooling operation with max pooling to verify the effectiveness of FMAPooling module. For simplicity, it is named as FMMPooling;

• VGG_D: the network that takes both FMAPooling module and FMAttn method, and the FMAttn is added at the end of each conv-x layer;

• VGG_E: the control network especially for VGG_D that takes channel-attention mechanism proposed in CBAM to verify the effectiveness of FMAttn method. And the attention is added as the same with VGG_D.

Experiments-ResNet

In terms of experiments concerning ResNet, two networks are applied for datasets CIFAR10/100 and ImageNet100 respectively being difference in number of layers as well as the Input/Output sizes. In detail, three strategies are designed to conduct comparative experiments.

• ResNet_A: the baseline model that without channel-attention mechanism and adopts simply one max pooling layer after the convolutional layer;

• ResNet_B: the network that takes the proposed FMAttn, and three FMAttn modules are added after each convN_x;

• ResNet_C: the control network that takes the channel-attention mechanism proposed in CBAM to verify the effectiveness of FMAttn.

Experiments-MobileNetV2

For MobileNetV2, the experiments are conducted simply on ImageNet100. Being different from experimental strategies of VGG and ResNet, the proposed FMAttn is applied for MobileNetV2 by replacing the last [ $1\times 1$] convolutional layer within the MobileNetV2-Bottlenecks rather than directly added to the network, as is shown in Fig. 5. In detail, the experimental strategies are established as follows:

• MobilenetV2_A: the baseline model which is the standard MobileNetV2 as is proposed in Sandler et al. (2018);

• MobilenetV2_B: the network with the last [ $1\times 1$] convolutional layer of the MobileNetV2-Bottlenecks replaced by the proposed FMAttn;

• MobilenetV2_C: the control network that utilizes the channel-attention mechanism proposed in CBAM to replace the last [ $1\times 1$] convolutional layer of the MobileNetV2-Bottlenecks.

Table 9 shows the experimental results of MobileNetV2 on ImageNet100: based on the analysis of the twice training, the accuracy of MobileNetV2_B is 0.53% higher than that of the baseline model MobileNetV2_A, and 1.80% higher than that of the control model MobileNetV2_C which takes the channel attention mechanism CAM.

Summary

According to the experimental results and analysis above, conclusions could be drawn as follows:

• In terms of VGG networks on datasets CIFAR10/100 and ImageNet100, the FMAPooling improves the validation by 1.31–1.63% compared with the baseline; the FMAttn together with the FMAPooling improves the performance by 1.56–5.06% compared with the baseline and 0.15–2.21% compared with the channel attention mechanism CAM.

• For ResNet models on datasets CIFAR10/100 and ImageNet100, the FMAttn improves the performance by 0.28–2.37% compared with the baseline and 0.06–0.65% compared with the channel attention mechanism CAM.

• As for MobileNetV2 on dataset ImageNet100, the FMAttn improves the performance by 0.53% compared with the baseline and 1.80% compared with the channel attention mechanism CAM.

In summary, the experiments by conducting VGG, ResNet, and MobileNetV2 on datasets CIFAR10/100 as well as ImageNet100 prove the effectiveness of the proposed FMAPooling and FMAttn. Although the improvement in the DNNs’ performance varies with DNN architectures as well as datasets, the superiority of FMAttn to the previous studies on the channel attention mechanism is verified. As for the application, the proposed FMAttn could be directly integrated into various DNN architectures, as is shown in the experiments of VGG and ResNet; Furthermore, the FMAttn could be also conveniently applied by replacing certain layers of DNNs for utilizing the DNN architectures such as saving computation overhead, as is shown in the experiments of MobileNetV2.