1. Introduction

2. Data

Image 1. Source: https://www.cs.toronto.edu/~kriz/cifar.html, Cifar10 [6].

Image 1. Source: https://www.cs.toronto.edu/~kriz/cifar.html, Cifar10 [6].

3. Methodology

3.2. Building Blocks of Model Architecture

Introduced in 2015, Residual Networks (ResNet) significantly improved the training of deep neural networks by addressing the vanishing gradient problem through the introduction of residual connections. These connections, which act as shortcuts, allow gradients to bypass one or more layers. This mechanism depicted in Image 2, prevents gradient vanishing and facilitates the training of substantially deeper networks.

Image 2. ResNet Block [1].

Image 2. ResNet Block [1].

ResNet architectures are constructed using two primary types of blocks: basic blocks or bottleneck blocks. The basic block, typically employed in ResNet-18 and ResNet-34, consists of two 3x3 convolutional layers. This structure simplifies network design and reduces computational demands while still benefiting from residual connections that add inputs directly to outputs, thus facilitating training and enhancing performance.

In contrast, the bottleneck block, utilized in deeper ResNets such as ResNet-50, ResNet-101, and ResNet-152, comprises three layers (1x1, 3x3, and 1x1 convolutions). The 1x1 layers are responsible for dimension reduction and projection, effectively managing the network’s complexity and computational load. This configuration enables the significant extension of network depth without a substantial increase in resource requirements. The proposed model uses Basic Block because of its smaller size and Squeeze Excitation blocks because of their ability to generate channel wise attention to create the Residual neural network.

3.2.1. Basic Convolutional Block

The Basic Convolutional Block, also referred to as the BasicBlock, forms the fundamental component of the ResNet architecture. It comprises two convolutional layers, each succeeded by batch normalization to stabilize and accelerate training. The first convolutional layer employs a 3x3 kernel with a specified stride, while the second convolutional layer maintains the input size. An integral shortcut connection bypasses these layers, facilitating gradient flow during training.

3.2.2. Squeeze-and-Excitation (SE) Block

The Squeeze-and-Excitation (SE) Block [8] improves feature recalibration by adaptively weighting channel-wise feature responses within the network. This block integrates two convolutional layers with a global average pooling operation to capture channel-wise statistics. The first convolutional layer reduces the number of channels, thereby improving computational efficiency, while the subsequent activation function introduces non-linearity. The final convolutional layer restores the original channel dimensionality, and a sigmoid activation function scales the features, emphasizing the most informative channels.

Image 3. SE-ResNet Module [8].

Image 3. SE-ResNet Module [8].

3.2.3. Residual Layers

The Residual Layers encapsulate multiple instances of the Convolutional Block, forming the core structure of the ResNet architecture. Each Residual Layer comprises a sequence of BasicBlocks, with the number of blocks determined by the specified architecture configuration. The stride parameter controls the downsampling of feature maps within each layer, facilitating the extraction of features at multiple scales. The inclusion of skip connections mitigates the vanishing gradient problem, enabling effective gradient flow during training. This hierarchical arrangement of Residual Layers enables the network to learn increasingly abstract representations of the input data, leading to improved classification performance.

3.3. Loss Function

3.3.1. Mixup Augmentation Loss

Mix-up augmentation technique [9] is designed to improve the generalization of image classification models by training on linear combinations of image pairs and their labels. The images are mixed, and each image is assigned the corresponding set of mixed labels, as seen in Image 4. Hence, promoting the model to learn more robust features that are not tied to specific training examples.

Image 4. Image showcasing mixup augmentation between images of a horse and car; labels (horse, car).

Image 4. Image showcasing mixup augmentation between images of a horse and car; labels (horse, car).

The primary benefits of using Mix-up include reducing the risk of overfitting by smoothing the label space and enhancing model robustness against noisy data. It also stabilizes the training process, leading to more reliable convergence.

A mixUp loss function which is a loss function consisting of the sum of losses of mixed images on both of its true labels. Described in equation (i), is used to train the model containing datasets with mixup augmentation.

$$mixupLoss\left(\widehat{y}\right)=\lambda Loss(\widehat{y}, label A)+(1-\lambda )Loss(\widehat{y}, label B)$$

(i)

Loss is any loss function chosen for training. Lambda is the weight of each label in the image. $\widehat{y}$ is predicted label, and label A and label B are true labels.

3.3.2. Knowledge Distillation Loss

Knowledge distillation [10] is a technique used for transferring knowledge from a large, complex teacher model to a smaller, more efficient one known as a student model. It involves training the student model to mimic the soft output (class probabilities) of the teacher model rather than the hard targets (actual class labels). The soft probabilities contain richer information about the input space as they reflect the confidence of the teacher model across all classes.

There are several ways to perform knowledge distillation, a loss based distillation was used for simplicity and ease of addition to the model. Image 5 depicts the working of a knowledge distillation architecture based on this technique.

The knowledge distillation loss combines the Kullback-Leibler divergence and any relevant loss function, for consideration, the cross-entropy loss, to train a student network effectively using a teacher network’s predictions. The first term measures the divergence between the log-softmax of the student network’s outputs Zs and the softmax of the teacher network’s outputs Zt, both scaled by a temperature parameter T. The second term measures the cross-entropy between the student network’s outputs and the soft targets provided by the teacher network. This term is weighted by 1−α balancing the influence of the distillation loss and the traditional loss. This approach ensures that the student network learns from both the softened outputs of the teacher network and the ground truth labels, improving generalization and performance. The equation for this knowledge distillation loss is given below in equation (ii):

$$Knowledge Distillation Loss=\alpha T^{2}Kld \left(log\right(\sigma (Zs/T)), \sigma (Zt/T\left)\right)+(1-\alpha )Loss(Zs, \sigma (Zt\left)\right)$$

(ii)

$\alpha$ is a weight for each function, Kld represents a function of KL divergence, $\sigma$ represents softmax function, Zs and Zt represent logits of Student and Teacher models respectively, Loss is any loss function that can be used relevant to the task at hand.

3.3.3. CrossEntropy Loss

At the core of the loss functions used in neural networks is the Cross-Entropy Loss. This loss function measures the performance of a classification model whose output is a probability value between 0 and 1. The Cross-Entropy Loss increases as the predicted probability diverges from the actual label. Mathematically, it is defined as Equation (iii),

$$Cross-Entropy Loss \left({\widehat{y}}_i\right)=-{\sum }_i{y}_i log\left({\widehat{y}}_i\right)$$

(iii)

In this formula, $y_i$ denotes the true labels, and ${\widehat{y}}_i$ signifies the predicted probabilities. This loss function effectively penalizes incorrect classifications more heavily, hence training models to make better predictions.

3.3.5. Model Training Loss

The training loss used for the model is a knowledge distillation loss combined with Mixup loss. The Cross-entropy loss is used as the model loss to build upon our derived loss functions.

$$mixupLoss\left(\widehat{y}\right)=\lambda · CrossEntropyLoss(\widehat{y}, ya)+(1-\lambda ) · CrossEntropyLoss(\widehat{y}, yb)$$

$$Knowledge Distillation Loss=\alpha T^{2}Kld \left(log\right(\sigma (Zs/T)), \sigma (Zt/T\left)\right)+(1-\alpha )Loss(Zs, \sigma (Zt\left)\right)$$

Using equations (i) and (ii), the derived loss can be written as,

$$Training Loss=\alpha T^{2}Kld \left(log\right(\sigma (Zs/T)), \sigma (Zt/T\left)\right)+(1-\alpha )mixupLoss\left(\widehat{y}\right)$$

Hence, the equation for the training loss is given in Equation 4 as,

$$Loss=\alpha T^{2}Kld \left(log\right(\sigma (Zs/T)),\sigma (Zt/T\left)\right)+(1-a)(\lambda · CELoss(\widehat{y}, ya)+(1-\lambda ) · CELoss(\widehat{y}, yb\left)\right)$$

(iv)

Note: CrossEntropy Loss is written as CELoss for convenient representation.

Hence, the proposed training loss function effectively integrates knowledge distillation and Mixup loss to enhance model performance. The knowledge distillation loss allows for efficient transfer of knowledge from a teacher model to a student model, while the Mixup loss improves generalization by interpolating between data samples. The derived loss function, as presented in Equation (iv), balances these components to optimize the training process, combining their strengths to improve accuracy and robustness. This approach enhances the model’s learning capacity and ensures it is well-prepared to handle diverse and challenging data scenarios, providing a solid framework for advanced model training.

4. Results and Discussion

5. Conclusions

References

1. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
2. Andrew, G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications, 2017. [CrossRef]
3. Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, Saining Xie; A ConvNet for the 2020s, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 11976-11986.
4. Tan, Mingxing, and Quoc Le. “Efficientnet: Rethinking model scaling for convolutional neural networks.” International conference on machine learning. PMLR, 2019.
5. Gaurav Menghani. 2023. Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better. ACM Comput. Surv. 55, 12, Article 259 (December 2023), 37 pages. [CrossRef]
6. Alex Krizhevsky. Learning multiple layers of features from tiny images. pages 32–33, 2009.
7. Aditya Thakur, Harish Chauhan, and Nikunj Gupta. Efficient resnets: Residual network design. arXiv preprint. arXiv:2306.12100, 2023.
8. Jie Hu, Li Shen, Gang Sun; Squeeze-and-Excitation Networks, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 7132-7141.
9. Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint. arXiv:1710.09412, 2017.
10. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015.
11. Assiri, Yahia. “Stochastic optimization of plain convolutional neural networks with simple methods. arXiv:2001.08856 (2020).
12. Hassani, A., Walton, S., Shah, N., Abuduweili, A., Li, J., & Shi, H. (2021). Escaping the big data paradigm with compact transformers. arXiv preprint. arXiv:2104.05704.
13. Yang, C., An, Z., Zhu, H., Hu, X., Zhang, K., Xu, K., … & Xu, Y. (2020, April). Gated convolutional networks with hybrid connectivity for image classification. In Proceedings of the AAAI conference on artificial intelligence (Vol. 34, No. 07, pp. 12581-12588).