Impact of Learning Rate

This section provides a tutorial example to demonstrate the impact of the learning rate of a neural network model. If a neural network model starts to generate an oscillating training loss, the learning rate can be reduced to help to a better solution.

In the classical neural network model, the learning rate is important when error function has rough surface near the bottom. A learning rate that is too large will cause the model to oscillate over a possible solution.

If you look at tests we did in the previous tutorial, we will see that the model with 3 layers and 4 neurons per layer failed converge to a better solution. A learning rate of 0.1 was used and it was probable too large, because training loss history is an oscillating curve.

Deep Playground - Complex Model with Learning Rate of 0.1
Deep Playground - Complex Model with Learning Rate of 0.1

If you reduce the learning rate to 0.03 and play it again. The model will converge further and stops with a test loss of 0.056.

Deep Playground - Complex Model with Learning Rate of 0.03
Deep Playground - Complex Model with Learning Rate of 0.03

If you reduce the learning rate to 0.01 and play it again. The model will converge further and stops with a test loss of 0.019.

Deep Playground - Complex Model with Learning Rate of 0.01
Deep Playground - Complex Model with Learning Rate of 0.01

Conclusion, if a neural network model starts to generate an oscillating training loss, the learning rate can be reduced to help to a better solution.

Table of Contents

 About This Book

Deep Playground for Classical Neural Networks

 What Is Deep Playground

 Simple Model in Playground

 Impact of Extra Input Features

 Impact of Additional Hidden Layers and Neurons

 Complex Model in Playground

 Impact of Training Set Size

 Impact of Neural Network Configuration

Impact of Learning Rate

 Impact of Activation Functions

 Building Neural Networks with Python

 Simple Example of Neural Networks

 TensorFlow - Machine Learning Platform

 PyTorch - Machine Learning Platform

 CNN (Convolutional Neural Network)

 RNN (Recurrent Neural Network)

 GNN (Graph Neural Network)

 References

 Full Version in PDF/EPUB