The Transformer Architecture — SophiArch

Why transformers replaced sequence models

Before transformers, the dominant approach to sequence tasks was recurrent networks — LSTMs and GRUs that processed tokens one at a time, left to right. This sequential constraint created two problems that were hard to solve simultaneously.

First, parallelism: because each step depends on the previous hidden state, you can't process tokens in parallel during training. A sequence of 512 tokens requires 512 sequential steps. Transformers eliminated this bottleneck entirely by processing all tokens simultaneously.

Second, long-range dependencies: recurrent networks pass information through a fixed-size hidden state. By the time the model reaches token 200, the signal from token 1 has been diluted through 199 state updates. Transformers don't have this problem. Every token can attend directly to every other token regardless of distance, in a single operation.

These aren't incremental improvements — they're the reason transformers scaled so dramatically. Larger models trained on larger datasets are only feasible because the compute can be parallelised across hardware. RNNs couldn't scale the same way.

Comparison of RNN and Transformer approaches

Three architecture families

Transformers come in three configurations, and choosing the wrong one for a task is one of the most common errors in AI-generated code.

Encoder-only (e.g., BERT, RoBERTa): all tokens attend to all other tokens in both directions. The model builds a rich contextual representation of the entire input. Best for tasks where you need to understand the full input at once: classification, named entity recognition, question answering where the answer is a span of the input.

Decoder-only (e.g., GPT-2, LLaMA, Mistral): each token can only attend to tokens that precede it (causal masking). The model generates one token at a time, conditioned on everything before it. Best for generation tasks: text completion, chat, code generation, summarisation framed as generation.

Encoder-decoder (e.g., T5, BART): a full encoder builds a contextual representation of the input; a decoder generates the output one token at a time, attending both to its own prior outputs and to the encoder's representation. Best for tasks with distinct input and output sequences: translation, abstractive summarisation, structured generation from an input document.

You'll encounter this choice constantly. An encoder-only model cannot generate open-ended text — it has no autoregressive decoder. A decoder-only model used for classification needs a carefully chosen token position to read the label from. Getting this wrong doesn't always produce an error; it often produces silently bad output that looks plausible.

Inside a transformer block

Every transformer is built by stacking N copies of the same block. Understanding one block lets you understand every transformer, regardless of size.

One transformer block. The residual connections (the arrows that bypass attention and FFN) are what make very deep stacks trainable — without them, gradients vanish before reaching early layers.

A single block contains four operations in sequence:

Multi-head self-attention — each token computes attention weights over all other tokens (or, with causal masking, all preceding tokens). The output is a weighted sum of value vectors, one per token.
Add & Norm (first) — the attention output is added to the original input (residual connection), then layer-normalised. The residual connection ensures the block is learning a residual update, not the full transformation from scratch.
Feed-forward network — a two-layer MLP applied independently to each token position. This is where a large fraction of a model's parameters live. In GPT-style models the FFN dimension is typically 4× the model dimension.
Add & Norm (second) — the FFN output is added back to its input, then normalised again.

The residual connections are not optional ergonomics. Without them, a 96-layer GPT-3-scale model cannot train. Gradients must flow cleanly from the loss all the way to the first layer. The residual path provides a direct gradient highway that bypasses the learned transformations.

Embeddings and the feed-forward dimension

Before the first block, tokens are converted to vectors. An embedding table maps each token ID to a fixed-size vector of dimension d_model. These embeddings are learned jointly with the rest of the model.

After embedding, positional information is added — either as a fixed sinusoidal signal or as learned position embeddings. Without this, the model has no way to distinguish "the dog bit the man" from "the man bit the dog". Lesson 3 covers positional encoding in detail.

The feed-forward network inside each block expands the representation to a higher dimension and then projects it back. For a model with d_model=768 (BERT-base), the intermediate FFN dimension is 3072. This expansion creates capacity for the model to learn richer per-token features than attention alone provides.

What you need to understand to review AI-generated transformer code

You won't construct a transformer from scratch in production. But you need to understand the architecture to catch the mistakes AI tools introduce when you ask them to fine-tune, configure, or extend one.

from transformers import AutoModelForSequenceClassification

# Review this code — is it correct?
model = AutoModelForSequenceClassification.from_pretrained(
    "meta-llama/Llama-3.2-1B",
    num_labels=3,
)

This code runs without error, but the architecture choice is wrong for many classification tasks. LLaMA is a decoder-only model. AutoModelForSequenceClassification adds a classification head on top of the last token's hidden state — which for decoder-only models is the last input token, not a dedicated [CLS] token. For most classification tasks, an encoder-only model like bert-base-uncased or roberta-base will produce better results with less compute, because its bidirectional attention gives every token full context from the start.

Decoder-only models work for classification in specific scenarios — very long documents where autoregressive pretraining is relevant, or when you're already using a decoder-only model for generation and want to add classification as a secondary task. For a standalone classifier on text shorter than 512 tokens, they're the wrong default.

Summary

Transformers process all tokens in parallel through stacked blocks of multi-head attention and feed-forward networks. Residual connections and layer normalisation make deep stacks trainable. Three architecture families serve different tasks: encoder-only for understanding, decoder-only for generation, encoder-decoder for sequence-to-sequence. The most consequential design decision in any transformer-based project is which family you choose — AI tools default to decoder-only regardless of task fit.

Next: Lesson 2 opens the attention mechanism itself, tracing exactly how Q, K, and V matrices are computed and why multi-head attention gives the model multiple independent views of each token.