What is pre-training in deep learning?
Pre-training in AI refers to training a model with one task to help it form parameters that can be used in other tasks. In AI, pre-training imitates the way human beings process new knowledge. That is: using model parameters of tasks that have been learned before to initialize the model parameters of new tasks.
What is meant by pre-training for weight initialization in deep neural networks?
You start training by initializing the weights randomly. As soon as you start training, the weights are changed in order to perform the task with less mistakes (i.e. optimization). Once you’re satisfied with the training results you save the weights of your network somewhere.
What can pretraining be used for in deep learning?
Pretraining can be used to iteratively deepen a supervised model or an unsupervised model that can be repurposed as a supervised model. Pretraining may be useful for problems with small amounts labeled data and large amounts of unlabeled data.
How is supervised learning used in deep learning?
The supervised learning phase may involve training a simple classifier on top of the features learned in the pretraining phase, or it may involve supervised fine-tuning of the entire network learned in the pretraining phase. — Page 529, Deep Learning, 2016.
How are feature hierarchies used in deep learning?
Deep learning methods aim at learning feature hierarchies, where features at higher levels of the hierarchy are formed using the features at lower levels (Dean et al., 2012). In 2006, Hinton et al. (2006) proved that much better results could be achieved in deeper architectures when each layer is pretrained with an unsupervised learning algorithm.
How is transfer learning used in deep learning?
The concept of transfer learning/pre-training came into being when some researchers found that a deep neural network, after training on a particular recognition task (eg. Object Recognition ), can be applied on another domain (eg. Bird Subcategorization) giving state-of-the-art results.