There are three facts that i want you to know before we start Residual NN. — (it is not RNN btw.)
1. AlexNet has 8 layers
2. VGG has 18 layers
3. Performance of AlexNet <<< VGG.
But but but, a very interesting thing happens when we increase more layers. –
**performance get decreases.** Why??
Let’s start Residual Neural Network and see how can we solve this using it.
So let me give you the idea behind Residual Neural Network.
##### What happens in traditional neural Network??
We have a input
We have NN layer
NN layer transform input into output.
Output can completely different than input. Like a complete transformation.
##### What we do in Residual Neural Network
We have a input
We have a NN layer
NN layer computes residual (which means a slight change) from the input
We add the residual to the input which becomes our output.
i.e output = input+that residual.
##### So what problem does it solves?
We can now train deeper NN with this because we are allowing models to updates the incremental updates instead of getting the transformative change at once.
Also this address the problem of vanishing gradient or exploding gradients which in traditional NN makes deep model harder to train.
##### What problem it brings?
Since we are adding changes to each time to input, it can be possible that we have very large activation magnitude during initialisation. We use batch normalisation to solve this.
So we can say the, Residual Network with batch normalisation allow much deeper network to be trained that improve performance across variety of task.