FFENet: frequency-spatial feature enhancement network for clothing classification

Application of frequency domain in the field of image classification

Spatial domain images can be classified directly by using trained neural networks, and good classification results can be obtained, but this approach does not fully exploit the information in the image, which is the frequency domain information implies in the image. There are many ways to extract frequency domain information from an image, such as the Fourier transform, discrete cosine transform, wavelet transform, and other methods. Frequency domain information, as an alternative representation of the spatial domain image, may contain information that is not used by the neural network and is useful for classification. Researchers have also conducted research into the use of frequency domain information extracted from images to complement image processing tasks when using deep learning techniques.

First, for DCT, Qin et al. (2021) studied the effect of partially compressed input images using the DCT algorithm on the performance of neural networks. They found that while the DCT algorithm reduces data redundancy, there is a risk of losing valuable features for network learning. In a similar vein, Xu et al. (2020) investigated the use of DCT transformation on original images followed by CNN classification. They demonstrated through experiments that DCT features obtained directly from the JPG format can be processed as effectively as the original image data using the same CNN architecture. The neural network architecture with DCT features performed on par with the one using original image data. Borhanuddin et al. (2019) approached the channel attention mechanism from a different perspective by considering frequency analysis. They showed that regular global average pooling can be seen as a special case of frequency domain feature decomposition and proposed a novel multi-spectral channel attention structure. Liu et al. (2018) proposed methods to compress and accelerate neural network training by focusing on weights and their connections. They explored a data-driven approach to remove redundancy in both spatial and frequency domains, enabling the network to discard more unnecessary weights while maintaining similar accuracy. They achieved this by obtaining an optimal sparse CNN in the frequency domain and reducing the computational burden of convolution operations through linear combinations of DCT basis convolutional responses. On a different note, Gueguen et al. (2018) presented a simple idea where they directly used JPG image processing to generate DCT coefficients. They modified the ResNet 50 (He et al., 2016) network to accommodate DCT coefficients as direct inputs and evaluated the performance of this model on the ImageNet dataset.

In addition to DCT, as a powerful time domain evaluation analysis method, wavelet transform can also provide additional frequency domain information for deep learning techniques. Li et al. (2020) used a nonlinear model and average pooling for wavelet transformation and proposed the wavelet scattering network. The first network layer of this network outputted SIFT-type descriptors, while the next layer provided complementary translation invariant information to improve classification. The network computed a translation invariant image representation that was stable to deformation and preserved high-frequency information for classification. However, the network couldn’t be easily transferred to other tasks due to strict mathematical assumptions. To address the problem of CNN being susceptible to noise interference (i.e., small image noise causing drastic changes in the output), Bruna & Mallat (2013) used discrete wavelet transform to replace max-pooling, step convolution, and average pooling in order to enhance CNN. They proposed a universal discrete wavelet transform and its inverse transform layer suitable for all kinds of wavelets. These layers were utilized to design a wavelet ensemble CNN for image classification.

In addition, a number of variants based on the wavelet transform (e.g., the contour wavelet transform) had also been studied accordingly. Liu et al. (2020) proposed a new network architecture called the contourlet convolutional neural network, which was designed to learn sparse and effective feature representations of images. The contour wave transform was first applied to obtain spectral features from the image. Then, the spatial spectral feature fusion method was used to integrate the spectral features into the CNN architecture, followed by statistical feature fusion to integrate the statistical features into the network. Finally, the fused features were classified to obtain the results.

Mainstream deep neural networks for classification

The image classification task is the task of determining which categories in the category space the input image belongs to. There are two types of mainstream classification networks, one based on convolutional neural networks and the other based on Transformer. Each of these two types has its own advantages and disadvantages. Convolutional neural network-based models work better for both small sample datasets and large datasets, and the network inference is faster. The Transformer-based classification model may not work as well on small datasets as the convolutional neural network-based one, which consumes a lot of memory space. However, it performs well on very large datasets, and the network is less prone to overfitting.

Convolutional neural network based classification models include GoogleNet (Szegedy et al., 2015), ResNet, DenseNet (Huang et al., 2017), EfficientNet (Tan & Le, 2019), ConvNext (Liu et al., 2022), and EfficientNetV2 (Tan & Le, 2021), among which the EfficientNetV2 model has best accuracy and computing speed compared with many classification networks. The focus of this paper is to improve the performance of the network on a small sample dataset, so in this paper, CNN is used to build our clothing classification network. The transformer (Dong et al., 2022; Hua et al., 2022) based classification models, such as ViT and SwinT (Liu et al., 2021), are initially used in the field of natural processing, where the transformer framework based on the attention mechanism achieved good results. Later, Vaswani et al. (2017) introduces the transformer to the field of computer vision, which worked well in mega databases, so the improvement of the Transformer based classification models hung a boom.

Network overview

Our proposed network structure is shown in Fig. 1. When the clothing images are inputted into the DCT frequency domain enhancement (DCT-FDE) module, they are first converted into the YCbCr format. Subsequently, the converted feature map is divided into blocks, and the information of each block is then transformed from the spatial domain to the frequency domain using DCT. The list of spatial domain feature maps is generated based on the frequency domain information at the corresponding position of each block. Finally, the generated feature maps are stitched together to obtain the feature map F. In order to delve deeper into the learned feature map information, we propose the clothing feature extraction (CFE) module to further explore the information within the feature map F.

In the CFE module, we first perform a 3 × 3 convolution operation, followed by the modified fused MBConv block and the modified MBConv block. Subsequently, the feature map output from the CFE module is fed into the classification header for classification. The classification header involves applying a 1 × 1 convolutional layer for channel alignment, followed by global average pooling. Finally, two fully connected layers are utilized. The first fully connected layer is responsible for obtaining the preliminary sequence, while the second fully connected layer is used to generate the final prediction result.

DCT frequency domain enhancement module

The transformation process of the DCT-FDE module is depicted in Fig. 2, which first converts the input RGB image into ycbcr format to obtain the feature map x_ycbcr ∈ R^H×W×3. Subsequently, x_ycbcr is partitioned into a set of 4 × 4 patches to obtain $\left\{p_{i,j}\in R^{4\times 4\times 3}\mid 1\le i\le H//4,1\le j\le W//4\right\}$, where the patches are three channels. A dense DCT transformation is performed on the image window for each one, and each patch is processed in the frequency domain to obtain, where represents the patch corresponding to a particular colour channel in $\left\{d_{i,j}\in R^{4\times 4\times 3}\mid 1\le i\le H//4,1\le j\le W//4\right\}$. Here each value in the patch corresponds to the intensity of a particular frequency band. In order to extract the information of different frequency bands separately, we filter the frequency bands for the number of times of chunk size squared, taking the information of only one frequency band and filtering out the information of other frequency bands each time, and perform a DCT inverse transform to convert the filtered time-frequency domain information into spatial domain information after each filtering operation, and finally get a list of feature maps $x_1^{idct}\in R^{H\times W\times 3},x_2^{idct}\in R^{H\times W\times 3},\dots ,x_n^{idct}\in R^{H\times W\times 3}$, where the value of n is the square of the block size. We have a block size of 4 here, so we end up with 16 feature maps. As Fig. 3 shows an experiment we did, visualising the 16 feature maps obtained after inputting the image to the DCT-FDE module, we can see that the first band of the chunk stores the most colour and texture information, and that the other bands store more shape and detail information, which is what we call low-frequency information and high-frequency information. By this method we do not lose any information in any of the frequency bands, but it allows our subsequent proposed classification network to learn both high-frequency and low-frequency information, which in fact replaces the convolution operation in a sense and has a feature enhancement effect.

Clothing feature extraction module

The MBConv block is a module unit originally introduced by MobileNetV2 (Sandler et al., 2018). It has since been widely adopted due to its portability and good performance. Subsequently, EfficientNet further built upon the MBConv block and achieved notable results in various tasks. EfficientNetV2, a follow-up work, suggests that employing fused MBConv blocks at the shallower layers of the network yields better performance. This improvement was achieved through the application of neural architecture search techniques, which allowed for the identification of more effective network configurations. The module we built is initially a combination of the fused MBConv block and the MBConv block, but in combination with the previous DCT-FDE module, we guess that there is channel compression in the squeeze-and-excitation (SE) module, which might lead to inadequate learning of our frequency domain information, so our DCT-FDE module improves the structure of the fused MBConv block and the MBConv block by replacing the SE module with the ECA module. Our experimental results prove our conjecture, please see the experimental section for details. We use ECA module to enforce the feature map, which is defined as follows: (1)${\displaystyle F_t^{eca}=F_{t-1}\left(\sigma \left(Con{v}^{1\times 1}\left(GAP\left(F_{t-1}\right)\right)\right)\right)}$ where F_t−1 and $F_t^{eca}$ are the input and output feature maps of the ECA module, respectively. σ denotes the sigmoid function. Conv^1×1 denotes the convolution operation with a filter size of 1 × 1.GAP denotes for global average pooling operation.

As shown in Figs. 4A and 4B, the improved fused MBconv block (IFMBConv) and the improved MBConv block (IMBConv) used in this paper are shown schematically. The CFE module specific parameter information can be found in Table 1, which describes each phase of the DCT-FDE module in detail. The parameter Stride indicates whether the first convolution in the first IFMBConv block or IMBConv block in a phase consisting of IFMBConv or IMBConv compresses the feature map, when the step size is 1, the feature map size is unchanged, and when the step size is 2, the compressed feature map size is one half of the input feature map. The parameter Channel indicates the size of the feature map at the time of input to the current stage. The Layers parameter represents the number of times the IFMBConv block or IMBConv block is repeated. Note that the size of the feature map output at stage 7 is 7 × 7 and the number of channels is 384.

Implement details

System configuration information for the experimental platform. The system version is Windows 10, the processor is an Intel(R) Core(TM) i9-12900KF CPU @ 3.20 GHz and the GPU is an NVIDIA GeForce RTX 3090 Ti 24GB. Conda environment relies on python 3.8. The optimizer used is the SGD optimizer, which the initial learning rate is 0.01 and the decay coefficient of the optimizer is 0.0001. The input network has 224 × 224 image pixels, batch size is 8. GoogleNet, ResNet, DenseNet, EfficientNet, ConvNext, EfficientNetV2, ViT and SwinT were chosen to compare the classification of the models.

Dataset

Evaluation criterion

We usually call the prediction is correct and positive as true positive (TP). A false positive (FP) is a prediction that is false and positive. If the prediction os correct and the result is negative, it is called true negative (TN). A false negative (FN) is when the prediction is false and negative. Based on the above theories, the evaluation indexes of the text are as follows. Accuracy is the proportion of correct prediction results in total prediction, which specific calculation method is shown in Eq. (2). Precision is the percentage of positive predictions that are correct, which specific calculation method is shown in Eq. (3). Recall is the percentage of all positive events that correctly predicted the result, which specific calculation method is shown in Eq. (4). Model accuracy is equal to the number of correct predictions (NCP) of all kinds divided by the total number of verified pictures (TNVP), which specific calculation method is shown in Eq. (5). It is the parameter most often used to evaluate the quality of model training. The equations are as follows: (2)${\displaystyle Accuracy=\frac{TP+TN}{TP+TN+FP+FN}}$ (3)${\displaystyle Precision=\frac{TP}{TP+FP}}$ (4)${\displaystyle Recall=\frac{TP}{TP+FN}}$ (5)${\displaystyle ModelAccuracy=\frac{NCP}{TNVP}}$

where accuracy here is calculating the probability that a single category is correct, whereas model accuracy is calculating the probability that the entire model is correct. Accuracy, Precision, and Recall can only reflect the performance of individual categories. To provide a more intuitive assessment of model performance in terms of Accuracy, Precision, and Recall, we introduce mAccuracy, mPrecision, and mRecall to evaluate the effectiveness of each model. By averaging Accuracy, Precision, and Recall, we make the experimental results more comprehensible. Additionally, we incorporate the number of model parameters and model complexity as indicators to analyze our model in the ablation experiment.

Comparison evaluation on Clothing8 dataset

The number of training rounds for our experiments on the Clothing-8 dataset is 100. The robustness of the model is very important, so we verify the performance of our model on the Clothing-8 validation set. The three indicators compared in Table 2 are Precision, Recall and Accuracy, and it can be seen that the effect of our model is better than these models. In this complex scene small sample dataset, it can be seen that the classification model based on convolution is better than the classification model based on tranformer. Firstly, in terms of the prediction mPrecision metric, our model outperforms the best EfficientNetV2 by 3.32%. Secondly, from the average recall, EfficientNetV2 has the best effect, but our model is 3.46% better than ConvNext. Finally, our model is also the best in terms of mAccuracy metric, our mAccuracy is 98.35%, which is 0.82% better than the best existing model EfficientNetV2 in the table.

Table 3 shows the comparison results of Model Accuracy between our model and other existing models on the Clothing-8 validate seen from Table 3 that ResNet, ConvNeXt, and EfficientNet V2 achieve preferably performance in the existing methods, and their accuracy rates are 90.01%, 89.46% and 87.79%, respectively. However, our model EfficientNet V2 achieves an accuracy of 93.4%, 3.39% better than the best model EfficientNetV2. The performance of our model on this small sample dataset of complex scenes is impressive. We verify our code on the DeepFashion dataset and replicate the effects of EfficientNet v2 and EfficientNet-B7 on this dataset, showing that our model FFENet still performs the best.

Comparison evaluation on Fashion-MNIST dataset

The number of training rounds for our conduct experiments on the fashion-mnist dataset is 50. Our model is compared with some existing classification models for the same volume on the fashion-mnist test set, and these experiments are not pre-trained. Among them, ViT and SwinT adopt the largest size model, and ConvNeXt adopts base model, largest model used by EfficientNet and EfficientNetV2. Transformer works well on very large datasets, but the transformer model do not work well on fashion-mnist dataset, so we used the largest volume of Transformer classification model to compare with convolutional classification model.

As can be seen from Table 4, in terms of model accuracy, our model outperforms the best model EfficientNetV2-L by 0.69%. According to the mAccuracy metric, our model is 0.13% better than the best model ConvNext. Then, it also has good effects from the indicators of mRecall and mPrecision. Its mRecall and mPrecision are both 94.62%, 0.69% and 0.64% higher than the current best algorithm respectively. From these comparative experiments, we can see that our model works well even on datasets with simple backgrounds.

Comparison evaluation on deepfashion dataset

In our comparison experiment on the DeepFashion dataset, we conducted 100 training iterations with a batch size of 16. Table 5 presents a comparison of the models used for clothing classification in the DeepFashion test set. Classic models like FashionNet are widely recognized in the field of fashion category classification. The citation of the top-k accuracy rate from the original DeepFashion paper in this study is based on the consistency of dataset division with the DeepFashion paper. Furthermore, we reproduce EfficientNet V2 and EfficientNet-B7 on the DeepFashion dataset, assessing their performance in the clothing classification task.

As shown in Table 5, our method FFENet consistently exhibits superior performance. In comparison to the specialized clothing classification model, FashionNet, our method achieves a 5.87% improvement in Top-3 accuracy and a 3.55% improvement in Top-5 accuracy. Moreover, our method outperforms the commonly used classfication model EfficientNetV2-L by 0.56% in terms of Top-3 accuracy and by 0.3% in terms of Top-5 accuracy.

Ablation study

Our ablation experiments are conducted on the Clothing-8 dataset, and the number of training rounds is set to 100. The other experimental settings are the same as those in subsection ‘Implement Details’. We choose MBConv block and fusion MBConv block to build the network structure, and the specific information can be found in subsection ‘Clothing Feature Extraction Module’. Table 6 shows our improvement process. We add DCT-FDE module and can see a 0.75% improvement in accuracy. We think about why the accuracy is not improved a lot. Considering that the number of channels of our DCT-FDE module output feature map is 48, we adjust the number of input and output channels of the first and second stages of the network to be 48, so that the information of the feature map output by our DCT-FDE module can be fully learned. The number of channels in the previous first and second stage are less than 48, so the learned feature map information must be insufficient. By adjusting the number of channels in the network we get another 1.15% improvement. Because the SE module inside the MBConv block and fused MBConv block has the operation of compression channel, so we replace the SE block with the ECA module. ECA module is also a channel attention mechanism, but the ECA module has no operation to compress the channel. Experiments show that the model accuracy is improved by 1.49% after replacing the SE module with the ECA module. The final model parameters decreased by 22.08M compared with the initial model.

Table 7 shows our other ablation experiment, in which the influence of DCT block size on the accuracy of the final model is discussed. According to the data in the table, we can find that the accuracy rate of 2 × 2 block is lower than that of 4 × 4 block. Meanwhile, compared with 2 × 2 block, the number of parameters and computational complexity in 4 × 4 block are not much improved. From the data in the table, if the 8 × 8 DCT block size is used, the accuracy is also decreased a little compared with the 4 × 4 block size, and the number of parameters and the computational complexity are increased, so we choose 4 × 4 block as our block size.