Transformers

Overview

A word on its own leaves a lot unresolved. A Transformer builds a representation of that word using the information around it. Its central mechanism, self-attention, lets each position draw information directly from other positions in the sequence.

This ability to connect distant positions without stepping through every intervening word made Transformers a strong fit for parallel training. The original 2017 architecture addressed translation; the same family now extends well beyond text, including models that process images as sequences of patches. Google Research · Vision Transformer paper

The explanation below follows a decoder-only Transformer used for text generation. Other members of the family share the core operations but allow information to flow differently.

Architecture

Text is split into tokens: words, fragments, or punctuation. Each token starts as an embedding, a vector of learned numbers. Position information gives the model a way to distinguish order. A stack of Transformer blocks then updates the representations before an output layer scores possible next tokens.

From tokens to a next-token prediction

Fig. 01

1. 01 · Encode

   Represent tokens

   Learned embeddings, with position information supplied to the model.

2. 02 · Mix

   Self-attention

   Each position combines information from positions the mask permits.

3. 03 · Transform

   Feed-forward

   Apply a learned transformation at each position separately.

4. 04 · Predict

   Score the vocabulary

   Project the final representation to scores, then probabilities for the next token.

↻ Steps 02 and 03 repeat through the stack

Two operations do different jobs inside each block. Attention mixes information between positions. The feed-forward network applies a learned transformation to each position separately. Residual connections preserve a path for information through the stack, while normalization helps control the scale of activations. Their exact arrangement varies between architectures. Original architecture, sections 3.1–3.5

Attention

For each position, learned projections produce three vectors: a query, a key, and a value. Comparing a query with the available keys gives compatibility scores. Softmax turns those scores into positive weights that sum to one; attention uses the weights to combine the value vectors.

Multiple heads perform different learned comparisons in parallel. Their outputs are combined before the next stage. A head is not assigned a fixed human-readable job such as “grammar” or “facts.” Scaled dot-product and multi-head attention

A decoder uses a causal mask to block future positions. In the example, select “its”: the position can use “robot,” but cannot yet use “battery.” Switch to full context to see what removing that restriction changes.

See what the mask changes

Interactive

Select a query position, then compare the two attention patterns.

Attention pattern

CausalFull context

The01robot02checked03its04battery05

5%

76%

8%

10%

Masked

its is the query at position 4. It can use positions 1–4. Later positions receive zero attention weight.

Attention weights: The, 5 percent; robot, 76 percent; checked, 8 percent; its, 10 percent; battery, 0 percent.

How these weights are calculated

The example keeps the same toy scores when you switch modes. A causal mask excludes later positions, then softmax normalizes the remaining scores. Removing the mask changes the available positions and the distribution of weights.

weightᵢ = exp(scoreᵢ) / ∑ exp(available scores)

A real attention head obtains the scores from query–key dot products scaled by the square root of the key dimension. It then mixes the value vectors; this example shows only the weighting step.

Training & inference

During training, a causal language model sees examples with the answers already available. For “The robot checked its battery,” it can learn to predict “robot” after “The,” “checked” after “The robot,” and so on. The loss measures how well the predictions match the targets; backpropagation updates the model’s parameters.

The sequence’s positions can be processed in parallel within each training layer, because the input is known. The causal mask prevents a position from using its answer. During ordinary autoregressive generation, the next input depends on the token just selected, so successive output tokens are produced sequentially. Training a causal language model

At inference time, a key–value cache reuses earlier attention keys and values instead of recomputing them. That saves work but consumes memory as the sequence grows. Prompt length, output length, batching, and cache precision all affect the cost of serving a model. Prefill latency and the speed of subsequent token generation are separate things to measure. Cache strategies

For a standard fixed-parameter model, sending a prompt changes the current activations and cache; it does not retrain the model. Fine-tuning is a separate process that changes parameters.

Model families

The name describes an architecture family, not a single training objective or product. The main distinction is which positions each part of the model can use. Hugging Face’s architecture overview

An encoder’s access to both directions is useful when the complete input is available. BERT, for example, learns from masked tokens using surrounding context. That is a different constraint from generating a continuation whose future tokens do not yet exist. BERT paper

Limits & trade-offs

Longer context increases work. Conventional dense attention has quadratically many token-to-token comparisons: doubling sequence length produces roughly four times as many comparisons. This describes the attention operation, not a universal fourfold increase in end-to-end latency. FlashAttention improves how exact attention uses GPU memory and compute; it does not remove the underlying dense pairwise computation. FlashAttention-2

Available context is not guaranteed recall. “Lost in the Middle” found that the models it evaluated could perform worse when relevant information appeared in the middle of a long input. That result motivates testing information placement and retrieval on your own workload; it is not a fixed performance claim about every Transformer. Study and evaluation

Architecture alone does not establish reliability. A model’s training data, objectives, post-training, and surrounding tools matter. A plausible answer still needs to be checked against the evidence and the task. An attention visualization shows a particular information mixture, not a complete explanation of why an answer was produced.

Further study

• The original paper — follow the equations and full encoder–decoder architecture.
• Google’s illustrated introduction — see how attention connects words in context.
• Train a causal language model — turn next-token prediction into a working training example.
• Explore cache strategies — connect the architecture to inference memory and latency.