Foreground separation knowledge distillation for object detection

Focal and global knowledge distillation

The FGD method synergizes the strengths of focal and global KD to enhance the performance and generalization capability of the student model. Focal KD focuses on fine-grained feature transfer, enabling the student model to more efficiently learn and replicate the teacher model’s performance in specific areas or tasks. For example, the FitNets method (Romero et al., 2014) achieves effective distillation of deep networks by aligning intermediate layer features. This method allows the student model to learn the teacher model’s intermediate layer features, thereby better capturing detailed features, especially in image classification tasks. In contrast, global KD emphasizes the extraction and transfer of holistic features, focusing on the overall optimization of the model’s performance. The soft targets distillation method proposed by Hinton, Vinyals & Dean (2015) is a typical example of global KD, which employs softened probability distributions (soft targets) to guide the training of the student model, enabling the student model to comprehensively learn the decision boundaries of the teacher model. Another example is a self-distillation framework proposed by Wang et al. (2024b). This framework introduces additional logit output branches for each shallow block to ensure consistent feature map dimensions between the shallow classifiers and the deep classifier, mitigating potential information loss and its adverse effects.

FGD amalgamates these approaches, significantly improving the performance of the student model through multi-level knowledge transfer. Focal distillation in FGD separates foreground and background, forcing students to focus their attention on the teacher’s key pixels and channels. Global distillation in FGD reconstructs the relationships between different pixels and passes them from the teacher to the students to compensate for the lost global information in local distillation. This combined approach not only enhances the accuracy of the student model but also improves its adaptability to different tasks and data distributions.

YOLOX

YOLOX (Ge et al., 2021) significantly advances object detection by building on the YOLO (You Only Look Once) series’ foundations and incorporating key innovations. Unlike its predecessors, YOLOX adopts an anchor-free mechanism, which simplifies the detection pipeline, reduces computational overhead, and improves performance, especially for small and irregularly shaped objects. Additionally, YOLOX employs a decoupled head architecture, separating classification and localization tasks into distinct branches, facilitating independent optimization and better precision. Empirical results illustrate that YOLOX outperforms most previous YOLO models and other contemporary detectors in both speed and accuracy, making it highly suitable for real-time applications such as autonomous driving and video surveillance.

Given the aforementioned advantages of YOLOX, applying our proposed KD method to YOLOX would serve as an exemplary case study. Moreover, the baseline method used in this paper, FGD, has also been implemented on YOLOX (Yang et al., 2022b). Therefore, although FSD is applicable to various object detection models, this paper focuses on its implementation with the YOLOX detector to effectively demonstrate the superiority of FSD.

Overall framework of the foreground separation distillation method

The design of the KD algorithm critically influences the knowledge transformation capability of the teacher model, which is directly related to the final performance of the student model. To enhance the knowledge transfer capability of the model, the proposed FSD method first manipulates the feature space of the model to enhance the learning and comprehension abilities of the student. Subsequently, an attention mechanism is introduced to extract essential task-related information from attention maps of the teacher and transfer it to the student. This process ensures that the student network focuses more on foreground information and minimizes the impact of background noise on the model.

The overall framework of the FSD method is depicted in Fig. 1. The input image is fed into both the teacher network and the student network for feature extraction, and after which the feature maps are generated. The network differentiates between the foreground and background based on their feature differences. Then, the spatial features of the foreground and background from both the teacher and student are converted into corresponding channel features. Finally, these features are transformed into probabilistic forms, and the KL divergence is utilized to measure the differences between the feature maps.

Foreground separation based on Gaussian mask

Considering the significant difference in model complexity between the teacher and the student, the FSD method incorporates a foreground feature extraction module to separate and individually distill the foreground and background. This module interprets and adapts the knowledge of the teacher model before transferring it to the student, thereby streamlining the training process and reducing the learning difficulty for the student model.

Most currently object detectors employ a Feature Pyramid Network (FPN) to fuse features at different scales. Since FPN can effectively fuse feature information from different levels of the backbone during KD on a detection model, the teacher network aggregates multi-level features through the FPN and subsequently transfer the learned feature knowledge to the student model, enhancing the student model’s performance. Generally, the feature distillation can be formulated as follows: (1)${\displaystyle L_{fea}=\frac{1}{CHW}\sum _{k=1}^C\sum _{i=1}^H\sum _{j=1}^{W}f\left(F_{k,i,j}^T,F_{k,i,j}^S\right)}$ where F^T represents the output features of the teacher model, and F^S denotes the output features of the student model. f(⋅) is a loss function to measure the discrepancy between the outputs of the student and teacher models. H and W represent the height and width of the feature maps, respectively. C denotes the number of channels in the feature maps.

Specifically, the FGD method uses a binary mask M to separate the foreground and background, as illustrated in Eq. (2): (2)${\displaystyle M_{i,j}=\left\{\begin{array}{c}1,if\left(i,j\right)\in d\hfill \\ 0,otherwise\hfill \end{array}\right.}$ where d denotes the bounding box region, and i and j are the horizontal and vertical coordinates of the feature map. If the point (i, j) falls within the bounding box, then M_i,j = 1, otherwise it is 0.

In order to highlight the features of the foreground more prominently, different from FGD, we propose a novel foreground separation approach based on a Gaussian mask in this paper to separate foreground and background from the feature map obtained by the FPN, which can be expressed as follows: (3)${\displaystyle M_{i,j}=\left\{\begin{array}{c}1\mathrm{ },if\left(i,j\right)\in d\hfill \\ e^{-\frac{1}{2}\left({\left(\frac{x-\overline{x}}{\overline{x}}\right)}^2-{\left(\frac{y-\overline{y}}{\overline{y}}\right)}^2\right)},if\left(i,j\right)\in \hat{d}\hfill \\ 0\mathrm{ },otherwise\hfill \end{array}\right.}$ where a threshold region $\hat{d}$ is introduced, which represents a region that is twice the size of the bounding box around d. If the point (i, j) is within the threshold region $\hat{d}$, then M_i,j satisfies Gaussian distribution. In other words, the area around the bounding box is diffused by using the Gaussian heatmaps according to Eq. (3).

Furthermore, to better separate the foreground and background, it is necessary to extract features from both. Therefore, we apply a binary mask to the background, which is defined as follows: (4)${\displaystyle M_{bg}=\left\{\begin{array}{c}1,if\left(i,j\right)⁄\in d\cup \hat{d}\hfill \\ 0,otherwise\hfill \end{array}\right.}$

In different regions, pixels need to have different scores; pixels within the bounding box should have the highest scores, and the farther pixels are from the bounding box, the lower their scores. To achieve this, we design a mask S with varying scales: (5)${\displaystyle S_{i,j}=\left\{\begin{array}{c}\frac{1}{H_d{W}_d},if\left(i,j\right)\in d\hfill \\ \frac{1}{H_d{W}_d},if\left(i,j\right)\in \hat{d}\hfill \\ \frac{1}{N_{bg}},otherwise\hfill \end{array}\right.}$

According to Eq. (5), the proportional masks within the range of d and $\hat{d}$ are both set to $\frac{1}{H_d{W}_d}$, while the mask for the background region is set to $\frac{1}{N_{bg}}$.

In order to have a more intuitive experience of our proposed foreground separation approach, we illustrate the mask images of the same input by FGD and our approach, as shown in Figs. 2 and 3, respectively. Moreover, Fig. 4 demonstrates the heatmaps obtained by FGD and FSD. From Fig. 4, it can be seen that after applying the Gaussian heatmap diffusion, the heatmaps highlight the foreground features more prominently, facilitating easier learning for the student model.

Channel feature extraction with probability distribution

Generally, KD methods calculate the total loss 𝔏_KD by aggregating the loss of each pixel in the spatial dimension: (6)${\displaystyle {\mathfrak{L}}_{\mathrm{KD}}=l\left(y,y^S\right)+\alpha \cdot l\left(\varphi \left(y^T\right),\varphi \left(y^S\right)\right)}$ where y represents the ground truth, with y^T and y^S denoting the activation maps from the teacher and the student models, respectively. l(⋅) represents the loss function. ϕ(⋅) is a function that performs a transformation on features. Thus, for Eq. (6), the first term in the right part is the loss used to measure the difference between the output of the student model and the ground truth, and the second term in the right part represents the distillation loss between the student model and the teacher model. The factor α serves as an equilibrium factor between these two terms.

Activations in different channels encode various characteristics of different categories within the input image. Additionally, well-trained teacher networks for object detection can typically display activation maps of specific masks with clearly defined categories on each channel. Therefore, in this paper, we propose a channel feature extraction approach based on foreground and background separation. This approach transforms the corresponding spatial feature maps into probabilistic forms, in order to fully utilize the knowledge in each channel of a well-trained teacher. Specifically, we convert the activation of the spatial feature maps into a probability distribution, so that we can use probability distance metrics (such as KL divergence) to measure the differences in feature maps. To this end, the distillation loss on the channels can be specifically expressed as follows: (7)${\displaystyle l\left(\varphi \left(y^T\right),\varphi \left(y^S\right)\right)=l\left(\varphi \left(y_c^T\right),\varphi \left(y_c^S\right)\right)}$ where c = 1, 2, …, C represents different channels, with y_c being the feature map in channel form. If the features in a certain channel are determined to be foreground features, ϕ(⋅) should convert the activation values of the features in that channel into a probability distribution: (8)${\displaystyle \Phi \left(y_c\right)=\frac{exp\left(\frac{y_{c,i}}{\tau }\right)}{\sum _{i=1}^{W\cdot H}exp\left(\frac{y_{c,i}}{\tau }\right)}}$ where i denotes different spatial positions within each channel, and τ is a hyperparameter indicating the temperature. A higher τ leads to softer probabilities, resulting in a smoother probability distribution. This means that each channel of the feature map can attend to a wider spatial area and provide more contextual information, thereby effectively enhancing detection performance. Moreover, to mitigate the impact of scale differences between large and small network models, we apply softmax normalization to the feature maps. Softmax normalization is advantageous for KD since it makes the loss function of the model easier to optimize, thereby accelerating the convergence rate of the FSD. Additionally, this paper utilizes the KL divergence to evaluate the differences between the student and teacher networks according to the following equation: (9)${\displaystyle l\left(y^T,y^S\right)=\frac{{\tau }^2}{C}\sum _{c=1}^C\sum _{i=1}^{WH}\varphi \left(y_{c,i}^T\right)log\left[\frac{\varphi \left(y_{c,i}^T\right)}{\varphi \left(y_{c,i}^S\right)}\right]}$

Attention mechanism

Incorporating an attention mechanism during model training can enhance the model’s focus on the information critical to the current task. The effectiveness of this strategy is demonstrated by the SENet (Hu, Shen & Sun, 2018) and CBAM (Woo et al., 2018), which shows that assigning higher weights to some important channels can significantly improve model performance. In this paper, we adopt an attention mechanism similar to that in the study by Zagoruyko & Komodakis (2016) to select critical regions and channels. Specifically, we first calculated the absolute mean values for different pixels and channels: (10)${\displaystyle G^S\left(F\right)=\frac{1}{C}\sum _{c=1}^C\left|F_c\right|}$ (11)${\displaystyle G^C\left(F\right)=\frac{1}{H\cdot W}\sum _{i=1}^H\sum _{j=1}^W\left|F_{i,j}\right|}$ where G^S and G^C are the spatial and channel attention maps on the feature map. Then, the attention mask can be calculated as follows: (12)${\displaystyle A^S\left(F\right)=H\cdot W\cdot \text{softmax}\left(\frac{G^S\left(F\right)}{{\tau }^{\prime }}\right)}$ (13)${\displaystyle A^C\left(F\right)=C\cdot \text{softmax}\left(G^C\left(F\right)/{\tau }^{\prime }\right)}$ where τ′ is the temperature hyperparameter introduced by Hinton et al. to adjust the distribution, with A^S and A^C representing the spatial attention mask and channel attention mask of the teacher detector, respectively.

Since there are significant differences between the masks of the student and the teacher models, we utilized the mask of the teacher model to guide the training process of the student. Specifically, the loss for foreground and background of the student L_Stufg and L_stubg, and those for the teacher L_teafg and L_teabg can be calculated by Eqs. (14) to (17), respectively. (14)${\displaystyle L_{stufg}=\sum _{k=1}^C\sum _{i=1}^H\sum _{j=1}^W{M}_{i,j}{S}_{i,j}{A}_{i,j}^S{A}_k^C{F}_{k,i,j}^S}$ (15)${\displaystyle L_{stubg}=\sum _{k=1}^C\sum _{i=1}^H\sum _{j=1}^W\left(1-M_{i,j}\right)S_{i,j}{A}_{i,j}^S{A}_k^C{F}_{k,i,j}^S}$ (16)${\displaystyle L_{teafg}=\sum _{k=1}^C\sum _{i=1}^H\sum _{j=1}^W{M}_{i,j}{S}_{i,j}{A}_{i,j}^S{A}_k^C{F}_{k,i,j}^T}$ (17)${\displaystyle L_{teabg}=\sum _{k=1}^C\sum _{i=1}^H\sum _{j=1}^W\left(1-M_{i,j}\right)S_{i,j}{A}_{i,j}^S{A}_k^C{F}_{k,i,j}^T}$ Subsequently, the feature loss L_fea can be calculated as follows: (18)${\displaystyle L_{fea}=l\left(L_{stufg},L_{teafg}\right)+l\left(L_{stubg},L_{teabg}\right)}$ Finally, the total loss of the entire framework is (19)${\displaystyle L=L_{fea}+L_{at}+L_{global}}$ where L_at is the attention loss and L_global is the global loss of the FSD method.

Dataset

In this study, the Fall Detection dataset and the Visual Object Classes (VOC) 2007 dataset were employed in our experiments. The dataset is available at the following link: https://zenodo.org/doi/10.5281/zenodo.12752189. Figures 5 and 6 display some examples from the Fall Detection dataset and VOC2007 dataset, respectively.

The Fall Detection dataset comprises the UR Fall Detection Dataset (Kwolek & Kepski, 2014) and the indoor Fall Detection dataset (Adhikari, Bouchachia & Nait-Charif, 2017). This dataset is formatted in COCO style and includes two categories: standing and falling. The dataset contains over 4,000 images, with 80% of the entire dataset randomly selected as the training set and 20% as the test set.

The PASCAL VOC 2007, originally a project initiated by the European Conference on Computer Vision (ECCV) and reported by Everingham et al. (2010), mainly used for object detection, image classification, and semantic segmentation tasks. In this study, we leverage the dataset to evaluate the object detection performance of the compared algorithms. The PASCAL VOC 2007 dataset contains a total of 9,963 images, which are re-divided into a 7:3 ratio to distinguish between the training set and the test set.

Experimental environment and parameter settings

All the experiments presented in this paper were conducted using the mmdetection framework (Chen et al., 2019), implemented in PyTorch running on a system with a 22-core AMD EPYC 7T83 CPU, NVIDIA GeForce RTX 4090 GPU and 90 Giga-bytes memory. The operating system is Ubuntu 20.04, with CUDA version 11.3 and Python version 3.8.

The parameter settings for all the experiments are as follows: the input image size is 640 × 640 pixels, the training epochs for the Fall Detection dataset are 120 while the training epochs for VOC2007 dataset are 40. Since all compared KD methods were applied to the YOLOX detector in this study, all the other parameters align with the default parameters used for YOLOX training, with an initial learning rate of 0.1, momentum set to 0.9, weight decay to 0.0005, and a batch size of 16 images per GPU. YOLOX-L was chosen as the teacher model and YOLOX-S served as the student model.

Evaluation metrics

In this paper, for both two dataset formats (COCO and VOC), the mean Average Precision (mAP) is adopted as the evaluation metric, where Average Precision (AP) represents the area under the Precision-Recall curve for a specific class. The formulas for calculating AP and mAP are shown as follows: (20)${\displaystyle \mathrm{AP}={\int }_0^{1}P\left(r\right)\mathrm{d}r}$ (21)${\displaystyle \mathrm{mAP}=\frac{\sum \mathrm{AP}}{N}}$ where P is the average precision value for the current class, and N is the number of sample categories in the dataset.

For the COCO-format Fall Detection dataset, we also report the mean Average Recall (mAR) and other AP-related metrics supported by the COCO format, including AP50, AP75, AP_S, AP_M, and AP_L. mAR is utilized to measure the whether the model can detect all positive samples and to identify any missed detections. AP50 and AP75 are special cases of calculating AP where the Intersection over Union (IoU) thresholds are set to 0.5 and 0.75, respectively. IoU is the ratio of the intersection over the union between the predicted bounding box and the ground truth box. A prediction is considered correct when the IoU is greater than or equal to a certain threshold (e.g., 0.5 or 0.75). The last three metrics measure the performance of detecting objects of different scales: small, medium, and large.

In addition, a new metric tailored for object detection, the optimal Localization Recall Precision (oLRP) error (Oksuz et al., 2018), has been introduced as an evaluation metric for the COCO-format Fall Detection dataset. The Localization Recall Precision (LRP) error consists of three components: localization, false negative (FN) rate and false positive (FP) rate, as detailed in Eq. (22). Compared to AP, LRP provides more detailed and discriminative information. (22)${\displaystyle LRP\left(Y,Z_s\right):=\frac{1}{N_{TP}+N_{FP}+N_{FN}}\left(\sum _{i=1}^{N_{TP}}\frac{1-IoU\left(y_i,z_{y_i}\right)}{1-\tau }+N_{FP}+N_{FN}\right)}$ where Y is the set of the ground truth boxes, Z denotes the set of the predicted boxes returned by an object detector, and Z_s represents the set of detections with confidence score larger than s.N_TP, N_FP and N_FN are the number of true positives (TPs), FPs and FNs, respectively. LPR penalizes each TP for its erroneous localization normalized by 1 − τ to the [0,1] interval, each FP and FN by 1 (the maximum penalty). The sum of these errors is normalized by N_TP + N_FP + N_FN, yielding a value that quantifies the average error per bounding box, ranging from 0 to 1. The lower the LRP error, the better the performance of the model.

Based on LRP, the oLRP is defined as the minimum achievable LRP error when τ = 0.5, representing the best balance between precision, recall, and IoU. Formally, oLRP can be expressed as: (23)${\displaystyle oLRP:=\underset{s}{min}LRP\left(X,Y_s\right).}$

Comparison of FSD and some state-of-the-art KD methods

In this section, the proposed FSD method is compared against several canonical and state-of-the-art KD methods on the Fall Detection dataset and VOC2007 dataset. The methods compared, as selected from Table 1, comprise three feature-based KD methods that solely rely on feature maps, two feature-based KD methods that utilize both feature maps and masks, and a state-of-the-art non-feature-based KD method. The results of comparisons are detailed in Tables 2 and 3.

The experimental results in Table 2 indicate that the proposed FSD method achieved the highest performance in both mAP and mAR, demonstrating superior overall precision and recall compared to other methods. Specifically, for AP50 and AP75, our algorithm outperformed other KD methods, with a noticeable advantage in AP75, indicating that FSD remains highly effective when more precise object localization is required. This suggests that the proposed foreground separation approach using the Gaussian mask facilitated more efficient learning of foreground features by the student model compared to the approaches using binary masks. In terms of detecting objects of different sizes, FSD achieved the best results for medium and large objects, particularly excelling in large-object detection. This success is likely attributed to the spatial-to-channel transformation module, which effectively filtered out irrelevant features that could impair object detection, enabling the student model to focus more on critical foreground features. For small-object detection, FSD also performed well, ranking second only to MGD. Moreover, in terms of the comprehensive evaluation metric, oLRP, FSD achieved the lowest oLRP score of 0.311 among all methods, highlighting its superior balance between localization accuracy, recall, and precision. Overall, our method outperformed all the other compared KD methods across almost all performance metrics, especially demonstrating strong robustness across different IoU thresholds and object sizes. Additionally, FSD significantly outperformed the baseline FGD across all metrics, further proving the effectiveness of the proposed modules in this paper.

The results on the VOC2007 dataset (Table 3) also demonstrate that the FSD method is better than most of the compared methods, with a mAP improvement of nearly 6% than the baseline FGD. The exception is the MGD method. Unlike the other mask-based KD methods shown in Table 1, MGD applies masks to the student features, compelling the student to generate more robust teacher-like features using only a subset of its own features. However, the comparative and matching requirements of the hidden layer features between the teacher and student models may necessitate additional computational resources and extended training times. In contrast, our proposed method requires fewer computational resources yet still achieves performance similar to that of the MGD.

Moreover, to provide a more intuitive understanding of the performance differences among the compared algorithms, we plotted the changes of mAP in terms of the number of iterations obtained by each comparative algorithm on both two datasets, as illustrated in Fig. 7. From Fig. 7A, it can be seen that our proposed method has a noticeable increase in mAP at the beginning of the training, followed by a slower growth rate. Around 20,000 iterations, our method surpasses the other comparative algorithms and maintains the lead thereafter. For the results on the VOC2007 dataset, although our method does not surpass MGD, it demonstrates a more stable training process. Specifically, the MGD method experiences a noticeable decline in mAP after reaching its peak, while the FSD maintains a more consistent performance.

Ablation experiments

To further validate and analyze the effectiveness and robustness of each module in the FSD method, several ablation experiments were conducted based on the Fall Detection dataset. Given that our work builds upon the FGD method, we adopted the FGD as the baseline. The tested modules include the Gaussian mask foreground separation (GMFS) approach and the channel feature extraction (CFE) approach, as discussed in ‘Foreground separation based on Gaussian mask’ and ‘Channel feature extraction with probability distribution’, respectively. The results on the Fall Detection dataset and the VOC2007 dataset are recorded in Tables 4 and 5, respectively. Moreover, similar to ‘Comparison of FSD and some state-of-the-art KD methods’, we also plotted in Fig. 8 the changes of mAP during the training process by each compared method.

From Table 4, it can be observed that integrating each module individually enhances the performance of the method. Specifically, when the GMFS module is included in FGD, the mAP and mAR of the model increase by 0.2% and 1.2%, respectively, and oLRP decreases by 0.003; when the CFE module is integrated, the improvements are even more pronounced, with mAP and mAR increasing by 1.4% and 2.3%, respectively, and oLRP decreasing by 0.016. On the contrary, almost every result on each performance metric for model 1 or model 2 is worse than that of model 3 (i.e., FSD), verifying the effectiveness of the incorporation of both the GMFS and CWD modules. This similar phenomenon can also be observed in Table 5, demonstrating the validity of each module again.

The changes of mAP by each compared algorithm on two tested datasets indicate that both the two modules can improve the performance of FGD through the whole training process, especially the CFE module. Additionally, comparing the two lines of “FGD+CFE” and “FGD+GMFS+CFE” in Fig. 8B shows that the GMFS module can further enhance the stability of the changes of mAP.

Visual analysis

The feature maps enhanced with attention by all the compared KD methods are presented in Fig. 9, to provide a more intuitive understanding of the effectiveness of our proposed method. These images are sourced from the UR Fall Detection dataset. The maps obtained by the AT method may incorrectly extract non-target objects. The FT method is prone to background interference, which can degrade the visualization effects. The remaining compared KD methods, including FGD, MGD, CWD and BKCD, perform well from a global perspective, but compared to FSD, they show less distinct foreground features. Take the results obtained by CWD as examples. In the feature map produced by CWD for the first image, two holes appear in the middle of the human body, which are absent in FSD’s feature map. For the second image, the features extracted by FSD from the person’s feet are noticeably brighter compared to those extracted by CWD. In summary, it is evident that the FSD can better highlight target regions, thereby extracting target features more effectively and enhancing model performance.