Part 1. General problems of the intellectual systems theory

The technology of knowledge distillation for training neural networks on example of binary classification

V. A. Biryukova

MIREA - Russian Technological University, Institute of Cybernetics

Abstract: Using the technology of training neural networks, the knowledge distillation, models were obtained that solve the binary classification problem with the productivity that is about five times higher than the performance of the teacher network with an insignificant drop in quality. The convolutional neural network ResNet-18 was trained in two ways by this technology (using the pre-trained network ResNet-50) and by the classical method. The concept of the degree of uncertainty of the model on objects' set is introduced as the quantity of the deviation of the neural network predictions from the values accepted for the answer. The experiments on the recursive application of the knowledge distillation technology were also conducted.

Keywords: knowledge distillation, binary classification, residual neural network, convolutional neural network, degree of uncertainty of the model on objects' set, recursive training of neural networks.