ResNet, which stands for Residual Networks, transformed deep learning by solving the vanishing gradient issue, thus facilitating the training of extremely deep networks. Unveiled in 2015 by Kaiming He and colleagues, ResNet presented residual learning, which permits the construction of deeper architectures without loss in performance.
Conventional deep learning models often experienced a decline in performance as the layer count grew. Nevertheless, ResNet’s residual connections enable gradients to travel directly across the network, enhancing stability and convergence.
Traditional machine learning depends on manually crafted feature extraction methods, necessitating spe- cialized knowledge. Algorithms such as Support Vector Machines (SVM), Decision Trees, and k-Nearest Neighbors (k-NN) utilize features that are engineered by hand to classify images. Nevertheless, this approach is time-consuming and frequently does not generalize effectively. Deep Learning, especially Convolutional Neural Networks (CNNs), streamlines feature extraction by learning hierarchical repre- sentations. The initial layers detect edges and textures, whereas the deeper layers recognize intricate patterns and objects, greatly enhancing accuracy.
- Image Classification
- Object Detection
- Image Segmentation
- Image Enhancement
- Image Reconstruction
- Image Generation
and many more…
CNNs are made up of several essential elements that facilitate automatic feature extraction and pattern
recognition.
- Input Layer: Accepts unprocessed pixel data and normalizes it for further processing.
- Convolutional Layer: Applies filters (kernels) to draw out spatial features from images, including edges, textures, and patterns.
- Pooling Layer: Minimizes the spatial dimensions while retaining significant information, enhancing computational efficiency and lessening the risk of overfitting.
- Activation Layer: Activation layers employ non-linear functions like ReLU to incorporate non- linearity. This allows the network to grasp intricate patterns.
- Fully Connected Layer: Maps extracted features to a final output, such as a classification label.
- Output Layer: The output layer delivers the ultimate prediction utilizing a Softmax function for classification purposes.
Shallow networks find it difficult to learn intricate patterns, which restricts their performance on high- dimensional image datasets. Deep CNNs utilize several layers to extract hierarchical features, resulting in enhanced accuracy, robustness, and generalization.
Nevertheless, augmenting depth presents challenges like vanishing gradients, higher computational costs, and optimization difficulties. ResNet tackles these problems through the use of residual learning.
Challenges in CNNs:
- Weight Initialization: Incorrect initialization may result in prolonged convergence or issues with vanishing/exploding gradients. Approaches such as Xavier and He initialization assist in alleviating this problem.
- Normalization: Batch Normalization standardizes activations over mini batches, enhancing learning stability and speeding up training.
- Regularization: Methods such as Dropout and L2 regularization help to avoid overfitting, ensuring improved generalization to new data.
- Intuitive CNN Architectures: CNN Architectures: Initial architectures like AlexNet, VGG, and Inception presented significant innovations including deeper networks, consistent kernel sizes, and extraction of features at multiple scales.
As networks become deeper, gradients may decrease (vanish) or increase significantly (explode), causing instability in training. This leads to ineffective weight updates, which can hinder or stop learning. Conventional approaches include precise weight initialization, activation functions such as ReLU, and batch normalization. Nevertheless, ResNet offers a more sophisticated solution through the use of residual connections.
Introduction of Residual Blocks
ResNet incorporates skip (shortcut) connections that skip one or more layers, enabling the gradient to flow directly to preceding layers. This avoids deterioration in deep networks and enhances training efficiency.
Historical Impact of ResNet
The introduction of ResNet represented a critical turning point in deep learning. This architecture showcased that extremely deep networks could be trained efficiently while also reaching state-of-the-art performance on various benchmarks. For example: On ImageNet, ResNet-152 reached a top-5 error rate of 3.57%, surpassing earlier architectures such as VGG. The ensemble model of ResNet secured the first position in the ILSVRC 2015 classification task and the COCO object detection and segmentation contests.
Mathematical Formulation of Residual Learning
Residual learning reformulates the mapping problem:
H(x) = F (x) + x
Here, H(x): Desired Mapping, F(x): Residual function learned by stacked layers.
This formulation streamlines optimization by enabling layers to concentrate on learning residuals instead of the complete mapping, thereby effectively tackling the degradation issue.
Model Architecture
ResNet is made up of several residual blocks that are arranged in a sequential manner. The design adheres to a structure of:
- Beginning convolutional layers and pooling.
- Accumulating residual blocks with identity connections.
- Completely connected layers for the ultimate classification.
Variants of ResNet
ResNet is available in multiple versions which vary in their depth and complexity.
- ResNet-18 and ResNet-34: Appropriate for lightweight applications because of reduced computational complexity.
- ResNet-50, ResNet-101, and ResNet-152: Intended for high-performance tasks with greater depth utilizing bottleneck layers for improved efficiency.
Key Innovations in Residual Blocks
Residual blocks utilize identity shortcut connections to enable gradients to circumvent specific layers. This approach guarantees reliable gradient flow even in networks with more than 100 layers. Furthermore:
- When the dimensions of input vary from those of output, linear projections are applied to align the dimensions.
- Bottleneck blocks decrease computational demand by condensing feature maps prior to enlarging them.