Transformer Architecture

Transformer Architecture

The foundational shift that enabled the development of modern LLMs was the invention of the Transformer architecture. A groundbreaking neural network architecture introduced in the 2017 paper “Attention Is All You Need.”

Key Innovation: Relies entirely on a mechanism called Self-Attention to process data. This allows for parallel processing, making it significantly faster and more powerful.

The Core Technology: The Transformer

Purpose: Excels at handling sequential data like text, but without using traditional sequential models (RNNs).

The Transformer is the neural network architecture that powers most modern LLMs.

Analogy: Instead of reading a sentence one word at a time, a Transformer can “see” all the words at once, understanding their relationships and context instantly.

The Self-Attention Mechanism: This is the most critical concept. It allows a model to understand the relationship between all words in a sequence simultaneously, regardless of their position. This is the innovation that enabled the parallelization and scale of modern models.

Key Innovation: The Self-Attention Mechanism. This allows the model to weigh the importance of different words in a sentence, regardless of their position.

Positional Encoding: As self-attention processes words in parallel, a method is needed to preserve their order. Positional Encoding provides this by adding information about each word’s position in the sequence.

Here are some of the key papers that laid the groundwork for this field:

  • “Attention Is All You Need” (2017) by Vaswani et al. This paper from Google researchers introduced the Transformer architecture and its key innovation, the self-attention mechanism. This work revolutionized natural language processing by showing that sequence-to-sequence tasks could be handled without recurrent neural networks (RNNs), allowing for parallel processing and the scaling of models..
  • “Improving Language Understanding by Generative Pre-Training” (2018) by Radford et al. This paper from OpenAI introduced the first Generative Pre-trained Transformer (GPT-1). It demonstrated a semi-supervised approach where a Transformer model was pre-trained on a large, unlabeled text corpus and then fine-tuned for various specific tasks, proving the power of this method for transfer learning.
  • “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2018) by Devlin et al. from Google introduced BERT (Bidirectional Encoder Representations from Transformers). Unlike GPT, which is a decoder-only model, BERT is an encoder-only model that was trained to understand context from both the left and right sides of a word in a sentence simultaneously. This made it highly effective for tasks like question answering and sentiment analysis.

The Overall Architecture: Encoder & Decoder

The Core Architecture

The original Transformer model consists of two main parts:

  • The Encoder Stack: Processes the input sequence (e.g., a sentence in English). It creates a contextualized representation of the entire input.
  • The Decoder Stack: Uses the encoder’s output to generate the output sequence (e.g., a translated sentence in French) one word at a time.

The Core Mechanism: Self-Attention

The Magic of Self-Attention

  • What it does: Allows the model to weigh the importance of every word in a sentence when processing any single word.
  • Analogy: If a human reads “The animal didn’t cross the street because it was too tired,” they know “it” refers to “the animal.” Self-attention gives the model this ability by focusing on the most relevant words.
  • How it works (Simplified): For each word, the model creates three vectors:
    • Query (Q): Represents the current word being processed.
    • Key (K): Represents all other words in the sentence.
    • Value (V): Contains the information of all other words.
    • The model calculates how well the Query matches each Key to determine which Values to focus on.

Slide 5: Multi-Head Attention

Multi-Head Attention

  • What it is: Instead of just one attention mechanism, the Transformer uses multiple “heads” in parallel.
  • Benefit: Each head can learn to focus on a different aspect of the data, such as:
    • Head 1: Grammatical relationships (e.g., “was” and “tired”).
    • Head 2: Semantic relationships (e.g., “animal” and “it”).
  • Result: This parallel processing provides a richer, more nuanced understanding of the input sequence, capturing a wider range of dependencies.

Positional Encoding

  • The Challenge: The self-attention mechanism processes all words at once, so it loses information about their original order.
  • The Solution: Before processing, the model adds a Positional Encoding vector to each word’s embedding.
  • How it works: This vector contains information about the word’s position in the sentence. This allows the model to understand the sequence and structure of the input, which is critical for language.
  • Method: The original paper used a combination of sine and cosine functions to create a unique, deterministic encoding for each position.

Slide 7: Inside the Blocks

Inside the Encoder & Decoder Blocks

In addition to self-attention, each block contains:

  • Feed-Forward Network: A standard neural network that further processes the output of the attention layer.
  • Residual Connections: “Skip” connections that help information flow directly through the network, preventing the vanishing gradient problem during training.
  • Layer Normalization: A technique to stabilize the training process by normalizing the outputs of each sub-layer.

Why Transformers are So Powerful

Key Advantages

  • Parallelization: The attention mechanism allows the model to process all tokens simultaneously, drastically reducing training time.
  • Long-Range Dependencies: It can easily capture relationships between words that are far apart in a sentence, a major weakness of previous models like RNNs.
  • Flexibility: The architecture is highly adaptable and has led to the creation of various models, including encoder-only (e.g., BERT) and decoder-only (e.g., GPT) variants.