Dual-mode knowledge distillation: Optimizing soft labels and feature alignment for efficient vision models
Abstract
Knowledge Distillation (KD) presents an exciting way to implement deep learning models in environments where resources are limited. It does this by transferring the knowledge gained from large teacher networks to smaller, more efficient student models. In this study, we take a closer look at two KD strategies, soft label distillation and concentrated layer distillation, using a mixed dataset that includes CelebA and CIFAR-10, which cover a variety of image types and classification tasks. A CNN-based teacher, trained on this hybrid dataset, helps guide a lightweight MobileNetV3 student model through various configurations. The results show that soft label distillation is particularly effective for semantic transfer, achieving an impressive test accuracy of up to 96.46\%, especially in fine-grained gender classification. On the other hand, concentrated layer distillation slightly edges out in overall accuracy (96.52\%) and F1-score for minority classes, while also enhancing computational efficiency. This study uncovers important trade-offs between maintaining semantic integrity and achieving representational compression, and it emphasizes how factors like batch size and data augmentation can influence distillation performance. These insights are valuable for choosing the right KD strategies for creating robust and deployable vision systems.