Reconstructed SqueezeNext with C-CBAM for offline handwritten Chinese character recognition

Basic module redefinition

The SqueezeNext network is made up of multiple iterations of the basic module. In this article, multiple basic modules are treated as a stage. Convolutional layers for down-sampling are interspersed between stages. Therefore, the performance of the basic module can directly affect the performance of the overall model. In the original model, the Add function was used for feature fusion. However, the Add function requires not only the same feature resolution but also the same number of channels of the two sets of feature tensors to be fused, i.e., the same number of convolution kernels. This limits the number of model compression methods that can be used to adjust the number of convolutional kernels, such as pruning. The reconstructed basic module proposed in this article is shown in Fig. 1.

The proposed basic module in this article uses two bottleneck layers, BottleNeck1 and BottleNeck2, for channel count compression. The two convolutional layers originally connected in series using K*1 and 1*K convolutional kernels are replaced with parallel ones. The output of Bottleneck 2 will be passed into these layers at the same time. The output of these two convolutional layers will be fused with features by concatenating. The concatenated feature maps will then be concatenated again with the input features. The number of channels of the tensor obtained by concatenation is large, so it needs to go through another bottleneck to adjust the number of channels. The feature concatenating approach increases the number of parameters but retains all the output of the previous convolutional layers and there is no limit to the number of channels. Conditions are created for model compression methods such as pruning.

To adapt the SqNxt structure to the handwritten Chinese character dataset, the filter size of the first convolutional layer in the original network is changed from 7*7 to 3*3 and the stride is reduced to 1. The backbone undergoes three times resolution compression to obtain four sets of feature maps with different resolutions. The number of repetitions for basic modules is two, four, 14, and one. There are 3,755 Chinese characters in the dataset, so the number of channels needs to be adjusted to avoid a sudden expansion in the number of channels leading to a large amount of redundant information.

Enhanced attention module: C-CBAM

Chinese characters are pictographs. Many Chinese characters are so similar that an additional stroke is all that is needed to form another character. In this article, we introduce an enhanced attention module C-CBAM in addition to the optimization basic module. C-CBAM, which is an improved version of CBAM (Woo et al., 2018), consists of two sub-modules, CAM and SAM.

First, referring to the idea of ECA (Wang et al., 2020), the fully connected layer in CAM is replaced by Conv1D and the layer used for channel scaling is removed, which reduces the number of parameters in the CBAM module and improves operational efficiency while maintaining accuracy. The filter size of the Conv1D layer is calculated in the same way as ECA, as shown in Eq. (1), where C is the number of input channels. (1)${\displaystyle filter\text{\_}size=\left({log}_{2}C+1\right)/2}$

With a traditional CBAM attention module, CAM is first applied to the input features to obtain the channel modulation features and then SAM is applied to obtain the final output features. However, as SAM is applied to the channel modulation features, the effect of the SAM is influenced by the CAM. At the shallow end of the model, the number of feature maps is small and the spatial information of the features is not compressed. The spatial information at this point is of greater importance to the overall model. Therefore, the order of the two sub-modules will be the same as the original CBAM, ensuring that the feature map is spatial modulated before output. In the middle layer of the model, the two modules are deployed in parallel. The two sets of modulation features are fused and output. In the deeper layers of the model, the number of channels expands several times, while the spatial information is highly compressed. The channel information at this point then has a greater impact on the classification accuracy of the Softmax function. Moving the SAM module in CBAM before the CAM module ensures that the output features are finally modulated by channel attention. The two sub-modules are shown in Fig. 2.

The attention module is deployed in the method shown in Fig. 3. Where K3 indicates a convolutional filter size of 3, S1 indicates a stride of 1 and C64 indicates a number of output channels of 64. Modules 1, 2, and 3 in the dashed box indicate three different structures of CBAM with CAM in front, CAM and SAM in parallel, and SAM in front respectively. The output of ConvFirst is used as input to C-CBAM1. The output of the MaxPooling layer before Block3 is used as input to C-CBAM2. The output of the MaxPooling layer before Block7 is used as input to C-CBAM3. This residual connectivity across multiple basic modules can improve the interaction between shallow and deep information and services to enhance deep semantic information.

The convolutional layer is initialized with ‘he_normal’. To maintain good classification accuracy at low precision (16 or 8 bits) and to reduce the impact of subsequent quantization operations on the accuracy, the activation function used in this article is ReLU. The h-swish activation function proposed in MobileNetV3 is only advantageous in deep networks so that ReLU will be able to satisfy the requirements of the network we proposed.

The proposed C-CBAM can enhance detection accuracy, but it does not assist with model compression. The module has a small number of parameters, which slightly increases the model’s size and computational requirements.

Pruning

The most straightforward method of model compression is to reduce the number of parameters, that is the number of convolutional filters. In addition to adjusting the number of convolutional filters per layer directly, network pruning can also be done to reduce the number of convolutional filters. Network pruning can be divided into structured pruning and unstructured pruning. Structured pruning refers to the removal of the complete convolutional filter. The structured pruning process is shown in Fig. 4. Unstructured pruning refers to fine-grained pruning, where some weights are removed from the convolutional filter, but not the entire filter. Unstructured pruning is highly flexible and can form sparse networks without sacrificing accuracy. However, pruning the weights can make the matrix too sparse, leading to redundant memory allocations that affect the operational efficiency of the network, and when deployed with the hardware side, it relies on algorithm libraries such as cuSPARSE, making it difficult to accelerate parallel operations through the circuit. Due to the limitations of unstructured pruning, structured pruning is used in this article.

The pruning process uses asymptotic soft filter pruning (ASPF). The technique is characterized by resetting to zero, rather than removing, the filters that satisfy the cropping criteria during training. These filters will continue to participate in subsequent training. The number of filters that are set to zero is then gradually expanded. At the end of the last training epoch, this part of the convolution filter is pruned. Compared to pruning the convolutional filters directly during training, ASPF gradually focuses the information on the more important convolutional filters. The ASFP process is illustrated in Fig. 5.

The standard of pruning is an important factor in the effectiveness of pruning. Pruning operations that are based directly on the magnitude of the sum of the absolute values of the convolutional kernel weights can cause large fluctuations in accuracy. In this article, we introduce the L1 and L2 norms in the proposed importance evaluation formula, as shown in Eq. (2). where α and β are scaling adjustment factors and θ is the weight. (2)${\displaystyle Ipt=\alpha \sum _{i=1}^n\sum _{j=1}^n\left|{\theta }_{ij}\right|+\beta \sqrt{\sum _{i=1}^n\sum _{j=1}^n{\theta }_{ij}^2}}$

When the values of the elements in the matrix change significantly, the fluctuations in the L1 norm of the matrix are usually larger and the fluctuations in the L2 norm are smaller. So the L1 and L2 norm of each matrix will be weighted and then summed. When the L1 norm is too large, the matrix is too sparse or the element values are too small, the model may be under-fitted and the sparse weights are not conducive to subsequent model compression means; when the L1 norm is too small, the feature extraction performance of the weights is reduced and useful information cannot be filtered out. When the L2 norm is too large, it pulls the weights to a small value, resulting in an underfitting of the model; when it is too small, it loses the effect of smoothing the curve. Therefore, α and β need to be set within a reasonable range to ensure the validity of the evaluation formula.

As the convolutional kernel weights may vary significantly between the shallow and deep layers of the model, in order to obtain a standardised wide range of thresholds that are not affected by outliers, the evaluation formula requires the values of all convolutional kernels in the full model by counting them. Compared to global thresholding, a multiple-stage thresholding approach allows a more appropriate threshold to be established based on the specific distribution of weights in each stage between each downsampled layer, ensuring that there will not be a case of cropping only one part of the model. In this article, the Z-score normalisation method is used to indicate the importance of a value in that stage, as shown in Eq. (3), (3)${\displaystyle Ip{t}_{stage}^i=\frac{Ip{t}^i-\overline{Ipt}}{Ip{t}_{\sigma }}}$ where $\overline{Ipt}$ and Ipt_σ represent the mean and standard deviation of all values in a single stage, respectively. The importance of the convolution kernel is determined by ranking the $Ip{t}_{stage}^i$. The i-th convolutional kernel corresponds to importance Iptⁱ and is ordered within the stage $Ip{t}_{stage}^i$. This normalisation method is designed to limit the threshold to a single stage and to facilitate pruning without affecting the original sorting order of $Ip{t}_{stage}^i$. Half of the values after normalisation are less than zero. During the experiment, not all convolution kernels corresponding to a less than zero are pruned.

Pruning is carried out according to the following steps and the flow chart is shown in Fig. 6.

Repeat:

1. The model before this round of pruning is noted as M₀.

2. Zero the weights in X% of the convolution kernels in the model that meet the clipping criteria to obtain the M₁.

3. Compare the accuracy of M₀ and M₁.

   • If Δacc(M₀ − M₁) ≥ 1.0%, it means that the accuracy fluctuates too much after the greedy algorithm has set the weight of that part of the convolution kernel to zero, and this round of pruning needs to be abandoned.

4. M₁ is fine-tuned to recover accuracy after several training sessions to obtain M₂

5. Compare the accuracy of M₀ and M₂.

   • If Δacc(M₀ − M₂) ≥ 0.6%, it means that the pruning has cut the more important convolutional kernels and the accuracy fluctuates a lot, and the accuracy cannot be recovered by training fine-tuning, so this round of pruning needs to be abandoned

6. If Δacc(M₀ − M₁) ≤ 1.0% and Δacc(M₀ − M₂) ≤ 0.6%, then this pruning has minimal impact on the accuracy of the model and can be trained to improve accuracy and is in a position to continue with the next round of pruning. At this point M₂ is used as M₀ for the next round of pruning. Due to the soft pruning, the number of parameters and the structure of the model do not change at this point. At the start of a new pruning round, an additional X% of the convolution kernels are set to zero on top of step 2.

7. If this pruning round is abandoned at step 3 or 5, the M₀ model from step 1 needs to be read and the size of X and the $Ip{t}_{stage}^i$ criteria adjusted downwards to start the next pruning round with a smaller growth ratio.

Following the above steps for structured pruning allows redundant convolutional kernels to be quickly screened out and zeroed at the beginning of the pruning. The increment X is decreasing and the number of convolutional kernels to be zeroed at a time is increasing. The number of convolutional kernels for cropping is shown in Eq. (4), where k_i denotes the number of pruning rounds at increments of X_i. (4)${\displaystyle \delta \mathrm{K}=\sum _{i=1}^N{k}_i{X}_i}$

Quantization

The main quantization methods for TensorFlow models are post-training quantization and quantization-aware training. Post-training quantization is used for floating-point models that have already been trained. Quantization-aware training is the calculation of simulated low-precision numerical types during the training of the model, which can compensate for the decrease of accuracy caused by quantization during the training process and speed up forward propagation. It can approximate the accuracy achieved by the float32 model as closely as possible, enabling deployment on hardware platforms.

To simulate low-precision calculations, a fake-quantization operation needs to be introduced into the training. The simulated low-precision computational flow is shown in Fig. 7. After obtaining the feature map of the upper layer input, the weights of the 32-bit are quantized to low-precision values. The low precision weights will be convolved with the feature map and then the output of this convolution layer will be obtained by ReLU activation. At this time, the weights will be inversely quantized to a 32-bit floating point type.

Fake-quantization nodes are added to the computation process of each layer to count the minimum and maximum values of float32 data for each weight at training time. The fake-quantization nodes are not involved in backpropagation, as gradient updates need to be calculated under the floating-point type. Accurate gradient updates require exact weights. Between the two convolution layers, there will be an inverse quantization and a quantization operation adjacent to each other. After the weights are back-quantized back to the floating point at the output of the upper layer, they are quantized again to low precision values without any operation. In this case, a lot of computational resources are wasted. Therefore, when inverse quantization and quantization are adjacent, they can cancel each other out and no operation is performed on the weights.

As each value in the floating-point type tensor maps one-to-one to a low-precision value, any further computation using the tensor does not introduce additional losses and can accurately simulate low-precision computations. The quantized simulation operations need to be integrated into the training process to be consistent with the quantized computation.

As symmetric quantization is not effective in representing unevenly distributed fractions, this article uses asymmetric quantization, where the mapping of zeros to floating point numbers requires an offset. The mapped scale factor P is shown in Eq. (5), where F_max and F_min denote the maximum value of the 32-bit floating point number counted, respectively, and I_max and I_min denote the maximum value that can be represented by int8, respectively. (5)${\displaystyle P=\frac{F_{\max }-F_{\min }}{I_{\max }-I_{\min }}=\frac{F_{\max }-F_{\min }}{255}}$

Once the proportional relationship is obtained, it can be mapped to the corresponding int8 integer value based on the 32-bit floating point value, as shown in Eq. (6), where I denotes an int value, F denotes a float value, and the round() function denotes rounding. Z represents the integer value to which the floating-point zero value is mapped, as shown in Eq. (7). When symmetric quantization is used, P∗I_max = F_max, the floating-point number Z corresponds to the integer type zero. (6)${\displaystyle I=round\left(\frac{F}{P}+Z\right)}$ (7)${\displaystyle Z=round\left(I_{{}_{\max }}-\frac{F_{\max }}{P}\right)}$

Based on the quantization process of individual values, the quantization needs to be extended to the multiplication and addition operations of the convolution kernel. Unlike the determinant calculation, the convolution operation involves multiplying the corresponding positions of the matrix and finally adding all the products to obtain the final result. The quantization process for multiplying two floating point values is shown in Eq. (8): (8)${\displaystyle P_3\left(I_3^{i,j}-Z_3\right)=P_1\left(I_1^{i,j}-Z_1\right)P_2\left(I_2^{i,j}-Z_2\right)}$ The expression for $I_3^{i,j}$ is obtained by collation, as shown in Eq. (9): (9)${\displaystyle I_3^{i,j}=\frac{P_1{P}_2}{P_3}\left(I_1^{i,j}-Z_1\right)\left(I_2^{i,j}-Z_2\right)+Z_3}$

In Eq. (9), all values are integers, except for the scale factor (P₁P₂)/P₃.

Assuming that the size of the convolution kernel is L*L, the single convolution operation is shown in Eq. (10). (10)${\displaystyle M_F=\sum _{i=0,j=0}^L{m}_3^{i,j}=\sum _{i=0,j=0}^L{m}_1^{i,j}{m}_2^{i,j}}$ where M_F denotes the value obtained from a single convolution operation, m₁, m₂ and m₃ denote the matrix to be convolved and the resultant matrix respectively, all with 32-bit floating-point values in the matrix. Bringing Eq. (8) into the matrix yields a fixed-point to floating-point formula for each value, as shown in Eq. (11). (11)${\displaystyle M_I=\sum _{i=0,j=0}^L{P}_3\left(I_3^{i,j}-Z_3\right)=\sum _{i=0,j=0}^L{P}_1\left(I_1^{i,j}-Z_1\right)P_2\left(I_2^{i,j}-Z_2\right)}$

Dataset and experimental settings

The dataset was selected from the offline handwritten Chinese character dataset of the Institute of Automation, Chinese Academy of Sciences (CASIA-HWDB), with 3,755 commonly used characters at the first level of the GB2312 standard. As shown in Fig. 8, some samples from the dataset are depicted, with the text in the image representing “Offline Handwritten Chinese Character Recognition.”

There are miswriting in the dataset as well as scribbles after the miswriting. In addition, some of the writing fonts are too shallow and the resolution of the parsed individual characters varies, as shown in Fig. 9, so the dataset needs to be pre-processed for normalization. All samples were binarised and stored using the average grey level of the full image as the threshold. Samples after preprocessing and normalization are shown in Fig. 10.

The data from HWDB 1.0 and HWDB 1.1 were mixed and washed to exclude as many useless samples as possible. Finally, 400 samples of each character were selected for training and 100 samples for testing. The details are shown in Table 1.

This experiment is based on Tensorflow-GPU version 2.3.0, using a GPU with NVIDIA RTX2060 6G memory, Intel Core i5-9400@2.9 GHz CPU, and DDR4 2667MHz 16G+16G memory.