Study Note 38 Transformers in Keras
Introduction to Transformers
Transformers have significantly impacted natural language processing and are now applied to various tasks, including image processing and time series prediction.
Introduced in the paper “Attention is All You Need” by Vaswani et al., Transformers use self-attention mechanisms to process data in parallel.
Transformers are the foundation of state-of-the-art models like BERT and GPT.
Transformer Architecture
The Transformer model consists of two main parts: the encoder and the decoder.
Both encoder and decoder are composed of layers including self-attention mechanisms and feed-forward neural networks.
Self-attention allows the model to weigh the importance of different words in a sentence when encoding a particular word.
Feed-forward neural network layers transform the input data after the self-attention mechanism.
Self-Attention Mechanism
Self-attention is the core component of the Transformer architecture.
It allows each word in the input to attend to every other word, capturing context and relationships effectively.
Each word is represented by three vectors: query, key, and value.
Attention scores are computed as dot products of query and key vectors, then used to weigh value vectors.
Transformer Encoder
The Transformer encoder consists of multiple layers with self-attention mechanisms and feed-forward neural networks.
Each layer includes residual connections and layer normalization for stable training.
Input is first embedded and then passed through positional encoding to add information about word order.
Transformer Decoder
The decoder is similar to the encoder but includes an additional cross-attention mechanism to attend to the encoder’s output.
It generates sequences based on the context provided by the encoder.
The decoder takes the target sequence as input, applies self-attention and cross-attention with the encoder’s output, and passes through a feed-forward neural network.