LLM Fundamentals

01

What is an LLM? — The Mental Model

🧠 The gist

Strip away all the hype and an LLM is doing one thing: predicting the next word (technically, token) given everything that came before it. That's the entire game. Reasoning, summarisation, code generation — all of it falls out of doing this one job really, really well at massive scale.

At its core, an LLM is a probability machine. You feed it a sequence of tokens, and it tells you what's likely to come next. During training, it sees trillions of tokens and tweaks billions of knobs (parameters) to get better at this guessing game.

Why does "large" matter? Because language is absurdly complex — idioms, grammar rules, facts about the world, reasoning patterns, cultural references. You need a lot of parameters to hold all of that, and a lot of training data to teach it. That's the whole insight behind the scaling revolution.

📦

Parameters

Weights in the neural network. GPT-4 ~1.7T estimated. LLaMA-3 70B has 70B. Claude 3 Opus — undisclosed. Each param is a float32 or float16 value.

🔁

Pretraining

Next-token prediction on massive corpora (Common Crawl, GitHub, books, etc). Self-supervised — the data itself provides the labels.

🎯

Fine-Tuning / RLHF

After pretraining, models are fine-tuned on instruction data + human feedback (RLHF or RLAIF) to become helpful assistants rather than raw text predictors.

🎓 Interview Questions

What is the training objective of a language model?
What is the difference between pretraining and fine-tuning?
Why does scale matter — what does a larger model learn that a smaller one can't?
What is RLHF and why is it needed after pretraining?

02

The Transformer Architecture

🧠 The gist

Before 2017, we processed text word-by-word with RNNs — imagine reading a novel but you start forgetting the first chapter by the time you hit chapter 5. Transformers blew this up. They look at the entire sequence at once and figure out which words matter for which other words. That parallelism is why we went from LSTM-sized models to GPT-4 in about 5 years.

The paper that started everything: "Attention Is All You Need" (Vaswani et al., 2017). The key insight was ditching recurrence completely. Instead of passing hidden state forward one step at a time, they let every position in the sequence talk to every other position directly through self-attention. Training became embarrassingly parallel, and long-range dependencies became trivial.

Figure: Transformer Block (Decoder-Only variant used in GPT/Claude/LLaMA)

A Transformer block has two main pieces stacked together:

1. Multi-Head Self-Attention — this is where tokens look at each other and decide what's relevant. "The cat sat on the mat" — when processing "sat," the model needs to know that "cat" is the subject. Attention handles that.

2. Feed-Forward Network (FFN) — a simple two-layer neural net (with 4× expansion) applied independently at each position. Think of this as the model's "knowledge storage." Recent research suggests facts and skills are largely stored here, not in the attention layers.

Both pieces have residual connections (skip connections) and layer norm wrapped around them. Without residuals, gradients vanish in deep networks and nothing trains. Without layer norm, activations explode. Simple additions that make 96-layer models actually trainable.

Deep Dive — Layer Norm, Residuals, and Why They Matter

Layer Normalisation normalises activations across the feature dimension (not the batch), making training stable. Unlike Batch Norm, it works per-sample — critical for variable-length sequences.

Pre-LN vs Post-LN: Original Transformer used Post-LN (norm after attention). Modern models (GPT-2 onwards) use Pre-LN (norm before attention), which gives more stable gradients and allows training without warmup.

RMSNorm: LLaMA and modern open-source models use RMSNorm (Root Mean Square Layer Norm) — simpler, faster, no mean subtraction step. Formula: x / RMS(x) · γ

LayerNorm(x) = γ · (x − μ) / √(σ² + ε) + β
// μ = mean, σ² = variance, γ, β = learned scale/shift

RMSNorm(x) = γ · x / √(mean(x²) + ε)
// simpler — no mean subtraction, used in LLaMA

🎓 Deep Dive Questions

Why did researchers switch from Post-LN to Pre-LN in GPT-2+?
What is RMSNorm and which models use it?
What problem do residual connections solve in deep networks?

🎓 Interview Questions

What are the two main sub-layers in a Transformer block?
Why did Transformers replace RNNs/LSTMs for language modelling?
What is the role of the FFN layer? Why 4× expansion?
What is a residual (skip) connection and why is it essential?
How many Transformer layers does GPT-3 have? (96 layers)

03

Self-Attention — The Core Mechanism

🧠 The gist

"The animal didn't cross the street because it was tired." What does "it" refer to? You instantly look back at "animal." That's exactly what self-attention does mechanically — every token asks "what should I pay attention to?" (that's the Query), every other token says "here's what I am" (Key), and if they match well, the actual information flows through (Value). The whole Q/K/V thing clicked for me once I thought of it as a lookup table where the lookup is soft — you don't pick one match, you take a weighted blend of everything.

Concretely: every token embedding gets multiplied by three learned weight matrices to produce Q, K, and V vectors. Then we compute dot products between Q and all the K's — that gives us "how relevant is each other token?" Divide by √d_k (to keep numbers from getting too big), softmax it into weights, and use those weights to blend all the V vectors together. That's one attention head.

Attention(Q, K, V) = softmax(Q·Kᵀ / √d_k) · V
// d_k = dimension of key vectors (e.g. 64)
// √d_k scaling prevents softmax from saturating for large d_k

Figure: Self-Attention Step-by-Step for a Single Head

That √d_k scaling isn't arbitrary. Without it, when d_k is 64 (typical head dimension), the dot products get large enough that softmax saturates — everything becomes 0 or 1, gradients die. Dividing by √d_k keeps the variance around 1. Small thing, makes training actually work.

In GPT, Claude, LLaMA (decoder-only models), there's also a causal mask — each token can only see tokens before it, never what comes after. Mechanically this just means setting future positions to −∞ before softmax, which drives their attention weight to zero. Simple trick, but it's what makes autoregressive generation possible.

Deep Dive — Causal Masking, KV Cache, and Attention Complexity

Causal Masking: The attention mask is an upper-triangular matrix of −∞ values, ensuring autoregressive generation. During inference, this enables the KV cache optimisation.

KV Cache: During generation, for each new token, we only need to compute Q for the new token (since we know all previous tokens). Keys and Values for all prior tokens are cached. This reduces per-step cost from O(n²) to O(n), which is why inference is much faster than training.

Attention Complexity: Standard attention is O(n²·d) where n is sequence length. For n=32K tokens, this is 1 billion operations per layer — hence research into sparse attention (Longformer), linear attention (Performers), and Flash Attention.

Attention per layer: O(n² · d)
// n = sequence length, d = model dim
// 4096 context → 16M ops/layer; 128K context → 16B ops/layer

🎓 Deep Dive Questions

What is the KV cache and how does it speed up inference?
Why is attention O(n²) and what are the alternatives?
What is Flash Attention and what problem does it solve?
Explain the causal mask in decoder models.

🎓 Interview Questions

Explain Q, K, V in self-attention. What does each represent?
Why do we divide by √d_k in the attention formula?
What is causal (masked) attention and why is it needed in decoders?
What is the time complexity of self-attention?
How does self-attention handle long-range dependencies better than RNNs?

04

Multi-Head Attention

🧠 The gist

One attention head can only learn one type of relationship. That's limiting — language has syntax, coreference, semantic similarity, positional patterns all happening at once. Multi-head attention just runs multiple attention ops in parallel, each with its own learned projections, so different heads can specialise in different things. Then you concat the outputs and project back down. It's elegant because it costs the same as one big head (same total params) but learns richer representations.

MultiHead(Q, K, V) = Concat(head₁, ..., headₕ) · W_O
where headᵢ = Attention(Q·W_Qᵢ, K·W_Kᵢ, V·W_Vᵢ)
// Each head has its own W_Q, W_K, W_V projection (d_model/h dims each)
// W_O: output projection matrix d_model × d_model

Figure: Multi-Head Attention (h=3 heads shown)

The key insight: since each head's Q/K/V matrices are d_model/h dimensions, the total computation is the same cost as a single full-dimension attention head, but you get h different learned perspectives.

Deep Dive — GQA, MQA, and Modern Variants (LLaMA-3, GPT-4)

Standard MHA has a memory problem at inference: the KV cache grows as O(n·h·d_k) per layer. For large h and long contexts, this is enormous. Two variants reduce this:

Multi-Query Attention (MQA) — introduced by Google, used in Falcon, PaLM, Mistral. All h query heads share a single K and V head. KV cache is h× smaller. Small quality loss, huge inference speedup.

Grouped-Query Attention (GQA) — used in LLaMA-3, Mistral, Gemma. A middle ground: G groups of query heads each share one K/V head (G < h). LLaMA-3 70B uses 8 KV heads for 64 query heads.

MHA: h_Q = h_K = h_V = h // full, most expensive
GQA: h_Q = h, h_K = h_V = G // LLaMA-3: h=64, G=8
MQA: h_Q = h, h_K = h_V = 1 // most efficient

🎓 Deep Dive Questions

What is MQA and why does it help inference but not training?
What is GQA and which models use it?
How does increasing h (heads) affect KV cache size?

🎓 Interview Questions

Why do we use multi-head instead of single-head attention?
If d_model=512 and h=8, what is d_k per head?
What does W_O (the output projection) do in MHA?
What is GQA and why does LLaMA-3 use it?
How does MHA capture different types of linguistic relationships?

05

Tokenization & BPE — What GPT-4, Claude & LLaMA Use

🧠 The gist

This tripped me up for a while: LLMs don't read words or characters. They read tokens — chunks of roughly 2-4 characters. Why not just use characters? Too granular, too many steps to learn anything meaningful. Why not whole words? You'd need millions of vocabulary entries and still couldn't handle "unhappiness" if you only saw "unhappy" in training. BPE splits the difference: "unhappiness" becomes ["un", "happiness"] — recognisable pieces that the model can compose.

Byte-Pair Encoding (BPE) started life as a compression algorithm, then got adapted for NLP. The idea: start with individual characters, count which pairs appear most often, merge the top pair into a new token, repeat. After enough iterations you get a vocabulary where common words are single tokens and rare words get broken into meaningful subwords.

Figure: BPE Algorithm — Building a Vocabulary from "aaabdaaabac"

STEP 01

Start: character vocab

a a a b d a a a b a c
Each character is a symbol.

→

STEP 02

Count pairs, merge top

"aa" appears 4× → merge.
aa aa b d aa b a c

→

STEP 03

Next most frequent

"aaa" appears → merge.
aaa b d aaa b a c

→

STEP 04

Repeat until vocab size

Run k iterations.
Result: efficient, lossless vocabulary.

In practice, BPE is trained on a large text corpus. After training, the tokeniser splits any new text greedily using the learned merge rules. The resulting vocabulary maps token IDs (integers) to strings.

Real-world example: "unhappiness" → tokens

Word: un happiness

Or more granularly:

un hap pi ness

GPT-4 example (cl100k_base):

"transformers" → transform ers

"ChatGPT" → Chat G PT

Model	Tokeniser	Vocab Size	Details
GPT-4 / GPT-3.5	tiktoken (cl100k_base)	100,277	Byte-level BPE. Efficient on code, multilingual. Used by OpenAI since GPT-4. Encodes at ~4 chars/token average.
GPT-2 / GPT-3	tiktoken (p50k / r50k)	50,257	Older BPE vocab. Byte-level. Less efficient on non-English.
Claude (Anthropic)	Proprietary BPE	~100K (est.)	Anthropic hasn't published exact tokeniser details. Believed to be similar byte-level BPE with extended multilingual coverage.
LLaMA 1 / LLaMA 2	SentencePiece (BPE)	32,000	Uses Google's SentencePiece library. Unigram + BPE hybrid. Smaller vocab, efficient for Latin scripts.
LLaMA 3	tiktoken (custom BPE)	128,256	Meta switched to tiktoken-style BPE for LLaMA-3 with 4× larger vocab than LLaMA-2. Much better multilingual + code coverage.
BERT / RoBERTa	WordPiece	30,000	WordPiece is similar to BPE but uses a likelihood-based merge criterion instead of frequency. Uses ## prefix for subword continuations.

Deep Dive — Byte-Level BPE vs WordPiece vs SentencePiece vs Unigram LM

Standard BPE — starts at the word level, needs pre-tokenisation (whitespace/punctuation splits). Can't handle unknown characters.

Byte-Level BPE (GPT-2+) — operates on raw bytes (256 base vocabulary). Can represent any UTF-8 text without UNK tokens. Used by GPT-2, GPT-3, GPT-4, LLaMA-3. Trades slightly more tokens for robustness.

WordPiece (BERT) — similar to BPE but merges pairs that maximise likelihood of the training data rather than raw frequency. Uses "##" to mark continuation tokens (e.g. "playing" → "play" + "##ing").

SentencePiece (LLaMA-2, T5) — library that trains either BPE or Unigram LM. Key feature: treats whitespace as a regular character (uses ▁ prefix for word-starting tokens), enabling truly language-agnostic tokenisation without pre-tokenisation steps.

Unigram LM (T5, XLNet) — probabilistic approach: start with a large vocabulary, iteratively remove tokens that minimise likelihood loss. More principled, produces more linguistically intuitive splits.

🎓 Deep Dive Questions

What is byte-level BPE and why is it superior to word-level BPE for multilingual models?
How does WordPiece differ from BPE in its merge criterion?
Why did LLaMA-3 increase vocab size from 32K to 128K?
What is SentencePiece and what advantage does it offer?

🎓 Interview Questions

What is BPE and how is the vocabulary built?
Why don't we just use characters or full words as tokens?
What tokeniser does GPT-4 use? What vocab size?
What tokeniser does LLaMA-3 use? How does it differ from LLaMA-2?
What is a "token" — approximately how many characters/words is it?
Why does "1+1=2" take more tokens than the same semantic content in English?

06

Word Embeddings & Positional Encodings

🧠 The gist

Tokens are just integers. But neural networks need vectors. Embeddings are the lookup table — token 4521 maps to a 4096-dimensional vector of floats. The trick: during training, tokens with similar meanings naturally end up with similar vectors (they appear in similar contexts). But here's the problem — Transformers process everything in parallel, so they have no idea what order the tokens are in. Position encodings fix that by adding a unique position-dependent vector to each embedding. Without them, "dog bites man" and "man bites dog" would look identical to the model.

The embedding table (also called the embedding matrix) is a learnable lookup: shape [vocab_size × d_model]. Token ID 4271 → row 4271 of the matrix. These weights are trained jointly with the rest of the model.

Figure: Token Embedding + Positional Encoding

〰️

Sinusoidal PE (Original)

Fixed (not learned). Uses sin/cos at different frequencies. Advantage: generalises to sequences longer than training. Used in original Transformer paper.

📐

Learned Absolute PE (GPT-2/3)

A second embedding table of shape [max_seq_len × d_model], fully learned during training. Simple, effective but doesn't generalise beyond training length.

🔄

RoPE (LLaMA, GPT-NeoX)

Rotary Position Embedding. Encodes relative positions by rotating Q/K vectors. Used in LLaMA-2/3, Mistral, Falcon. Naturally handles long contexts and enables context extension via scaling tricks (YaRN, etc.).

⚡

ALiBi (MPT, BLOOM)

Attention with Linear Biases. No positional embedding at all — instead, adds a fixed bias to attention scores penalising distance. Very strong at generalising to longer contexts.

Deep Dive — RoPE in Detail (Used in LLaMA-3)

RoPE encodes position not by adding a vector but by rotating the Q and K vectors. For a query vector at position m and key vector at position n, the attention score depends only on their relative position (m−n) — this is why RoPE naturally supports longer contexts.

Implementation: split Q and K into pairs (d_model/2 pairs), then apply a 2D rotation to each pair using angle θ = m / 10000^(2i/d). This is equivalent to multiplying complex embeddings by e^(imθ).

RoPE enables context length extension at inference: by scaling θ (e.g. linear scaling or the more effective YaRN technique), models trained on 4K context can run up to 128K context. This is how LLaMA-3 was extended from 8K to 128K.

RoPE: rotate Q at pos m by angle m·θ_i
Attention: Q_m · K_n = f(x_m, m)ᵀ · f(x_n, n)
// Key property: depends only on (m−n), not absolute positions

🎓 Deep Dive Questions

Why does RoPE enable better context length generalisation than learned PE?
What is YaRN and how does it extend LLaMA-3's context to 128K?
What is the key mathematical property of RoPE? (relative position encoding)

🎓 Interview Questions

What is a word embedding and how is it learned?
Why does a Transformer need positional encodings?
What is the shape of the embedding table? (vocab_size × d_model)
Compare sinusoidal, learned, and RoPE positional encodings.
What is RoPE and which models use it?
Why do RNNs not need positional encodings but Transformers do?

07

Encoder-Decoder vs Decoder-Only Models

🧠 The gist

The classic example: machine translation. The encoder reads the full French sentence — both directions, full context. The decoder generates English one word at a time, peeking at the encoder's representation plus what it's already written. But here's what happened: researchers realised that decoder-only models (just the right half, no encoder) can do translation, summarisation, Q&A, coding — basically everything — if you just make them big enough and train them on enough data. That's why GPT, Claude, and LLaMA are all decoder-only. Simpler architecture, scales better, surprisingly general.

Figure: Architecture Families

Property	Encoder-Only	Encoder-Decoder	Decoder-Only
Attention direction	Bidirectional	Enc: Bidirectional Dec: Causal	Causal (left-to-right)
Training objective	Masked LM (MLM) & NSP	Text-to-Text (Span denoising in T5)	Next-token prediction (CLM)
Cross-attention	None	Yes — decoder attends to encoder output	None (single stack)
Parameter efficiency	Efficient for understanding	~2× parameters for same seq length	Single stack — scales efficiently
Strengths	Classification, NER, semantic search, embeddings	Translation, summarisation, structured output	Generation, instruction following, reasoning, code
Key models	BERT, RoBERTa, DeBERTa, ELECTRA	T5, BART, Flan-T5, mT5, mBART	GPT-4, Claude 3/3.5, LLaMA-3, Mistral, Falcon
Why dominates today	Still used for embeddings/retrieval	Specialised tasks but declining for LLMs	Emergent abilities at scale, generalises to all tasks via prompting

Why did Decoder-Only win? At scale (100B+ parameters), decoder-only models show remarkable emergent abilities — in-context learning, chain-of-thought reasoning, instruction following — that enc-dec models struggle to match. A single next-token prediction loss turns out to be sufficient to learn nearly every language task, especially when combined with RLHF fine-tuning. Encoder-decoder models have slightly better parameter efficiency for specific seq2seq tasks, but the ecosystem has consolidated around decoder-only.

Deep Dive — Cross-Attention in Encoder-Decoder (T5/BART)

In encoder-decoder models, the decoder has an additional cross-attention layer between the masked self-attention and the FFN. In cross-attention:

— Q comes from the decoder's current state (what am I generating?)

— K and V come from the encoder's output (what did the source say?)

This allows the decoder to "look at" any part of the source sequence at every generation step, which is why encoder-decoder models are particularly good at translation — the decoder can focus on different source words at each step.

Cross-attention adds ~33% more parameters to each decoder layer (the extra W_Q, W_K, W_V matrices for cross-attention) and makes inference more expensive than decoder-only architectures.

CrossAttention(Q_dec, K_enc, V_enc)
= softmax(Q_dec · K_encᵀ / √d_k) · V_enc
// Q from decoder, K and V from encoder output

🎓 Deep Dive Questions

How does cross-attention work in T5/BART?
Why does cross-attention make encoder-decoder models better at translation?
How many attention sub-layers does a decoder block have in enc-dec models? (3: self-attn, cross-attn, FFN)

🎓 Interview Questions

What is the key difference between encoder-only, encoder-decoder, and decoder-only architectures?
Why does BERT use bidirectional attention but GPT does not?
What is cross-attention in encoder-decoder models?
Why have decoder-only models become dominant for LLMs?
GPT-4, Claude, LLaMA — which architecture family? (Decoder-only)
T5, BART — which architecture family? (Encoder-decoder)
What training objective does BERT use vs GPT? (MLM vs CLM)

08

Training Objectives, Loss & Perplexity

🧠 The gist

Training is dead simple conceptually: show the model text, ask it "what comes next?", penalise it when it's wrong. The loss function measures how far off the prediction was. Perplexity is just e^(loss) — and it has a nice interpretation: if perplexity is 10, the model is basically as confused as if it were randomly choosing from 10 equally likely options at every single step. Lower = better. GPT-4 probably has perplexity around 4-6 on general text.

🔮

CLM — Causal Language Modelling

Predict the next token given all previous tokens. Left-to-right only. Used by GPT-4 Claude LLaMA. Training and test objectives are identical — no separate head needed.

🎭

MLM — Masked Language Modelling

Randomly mask 15% of tokens; predict them using bidirectional context. Used by BERT. Learns rich contextual representations but can't generate text natively.

🧩

Span Denoising (T5)

Mask random contiguous spans (not single tokens); predict the spans as a sequence. Used by T5. A text-to-text framing — every task becomes "input text → output text".

CLM Loss (Cross-Entropy): L = − (1/N) · Σ log P(x_t | x_<t)
// N = number of tokens, P = model's predicted probability for the true token

Perplexity: PPL = exp(L) = exp(−(1/N) · Σ log P(x_t | x_<t))
// Lower is better. GPT-3 on Penn Treebank: ~20. Random baseline: vocab_size (e.g. 50000)

The cross-entropy loss during training is computed token-by-token. The model's final layer outputs logits — one raw score per vocabulary token — and a softmax converts them to probabilities. The loss penalises the log-probability assigned to the correct next token.

Why perplexity is useful: It's interpretable — PPL=5 means the model was effectively choosing among 5 tokens. It's also comparable across models on the same dataset. However: it's computed on specific test sets (WikiText-103, Penn Treebank) — a model with lower PPL on one domain may be worse on another.

Deep Dive — Label Smoothing, Teacher Forcing, and Exposure Bias

Teacher Forcing: During training, the ground-truth previous tokens are always fed as input (not the model's own predictions). This makes training stable and fast but creates a gap with inference — the model never learns to recover from its own mistakes during training.

Exposure Bias: The mismatch between teacher-forced training (sees real tokens) and autoregressive inference (sees own predictions). Errors compound at inference — a wrong token leads to worse subsequent tokens. RLHF partly addresses this.

Label Smoothing: Instead of a hard one-hot target (probability 1 for correct token), use a soft target (e.g. 0.9 for correct, 0.1 spread over others). Prevents overconfidence and improves calibration. Used in the original Transformer paper (ε=0.1).

Label Smoothing: y_smooth = (1−ε)·y_hard + ε/V
// ε = smoothing factor (0.1 typical), V = vocab size

🎓 Deep Dive Questions

What is teacher forcing and what problem does it cause?
What is exposure bias and how does RLHF mitigate it?
What is label smoothing and why does it help calibration?

🎓 Interview Questions

What loss function is used to train a language model?
What is perplexity and how is it related to cross-entropy loss?
What is the difference between CLM (GPT) and MLM (BERT) objectives?
Why can't BERT generate text natively?
What is teacher forcing and what is exposure bias?
If a model has PPL=10, what does that intuitively mean?

09

Decoding Strategies — Temperature, Top-k, Top-p (Nucleus)

🧠 The gist

At each step, the model gives you probabilities over ~100K tokens. The question is: how do you pick one? Greedy (always take the top) is fast but produces robotic, repetitive text. Temperature reshapes the distribution — low temperature makes the model more confident (sharper peaks), high temperature makes it more exploratory (flatter). Top-p is the clever one: instead of picking from the top-k tokens, you pick from however many tokens it takes to cover p% of the probability mass. So for an easy prediction (one obvious next word), you might only consider 3 tokens. For a creative junction, you might consider 200.

Figure: Decoding Strategy Comparison

Strategy	How it works	Best for	Drawback
Greedy	Always pick argmax(logits)	Factual QA, deterministic output	Repetitive, misses better multi-step paths
Beam Search	Keep top-k beams at each step, return best complete sequence	Translation, summarisation (T5, BART)	Computationally expensive; produces generic text
Top-k sampling	Sample from top k tokens by probability	Creative text, chat	k is fixed — bad when distribution is flat or peaked
Top-p (Nucleus)	Sample from smallest set with cumulative P ≥ p	Best general-purpose sampling (default in most APIs)	Slightly more complex; p needs tuning per use case
Temperature	Divide logits by T before softmax. T<1 → sharper; T>1 → flatter	Combined with top-p/k; T=0 ≈ greedy; T=1 = unmodified	Not a standalone strategy — modifies other sampling
Min-p (2024)	Filter tokens below p × max_prob threshold	Better coherence than top-p at high temperatures	Less widely supported

Temperature scaling: logits_scaled = logits / T
P(token_i) = softmax(logits_scaled)_i = exp(l_i/T) / Σ exp(l_j/T)
// T→0: argmax (greedy). T=1: original distribution. T→∞: uniform

Top-p: select S = argmin{|S|} such that Σ_{i∈S} P(i) ≥ p, sample from S
// p=0.9 → sample from tokens covering 90% probability mass

🎓 Interview Questions

What is temperature in LLM sampling and what does T=0 mean?
What is top-p (nucleus) sampling? How does it differ from top-k?
Why is top-p preferred over top-k for general-purpose generation?
What is beam search and why is it less common in chat models?
What temperature would you use for a coding assistant vs a creative writing tool?
What happens to the output if you set temperature very high (> 2)?

10

Fine-Tuning Paradigms — LoRA, QLoRA, Adapters, PEFT

🧠 The gist

Fine-tuning a 70B model from scratch? You'd need ~280GB just for the weights in fp32. Nobody has that kind of GPU lying around. LoRA is the hack that makes it tractable: freeze the entire pretrained model, then add two tiny matrices (A and B) at specific layers. Instead of updating the original weight W, you learn ΔW = A·B — where A and B together might be 0.1% of the original parameter count. You get 90% of the quality of full fine-tuning at a fraction of the memory. QLoRA takes it further: quantise the frozen model to 4-bit, then do LoRA on top. Suddenly you can fine-tune a 65B model on a single 48GB GPU.

Figure: LoRA — Low-Rank Adaptation

Method	Trainable Params	Memory	Key Idea	Used By
Full Fine-Tune	All (100%)	Highest	Update every weight	When compute available; GPT-3 instruct
LoRA	~0.1–1%	Low (frozen base)	Add low-rank ΔW=B·A to frozen W_q, W_v	LLaMA fine-tunes, Mistral variants, FLAN
QLoRA	~0.1–1%	Extremely low	LoRA on 4-bit quantised base model (NF4)	Consumer GPU fine-tuning (Guanaco, WizardLM)
Adapter Layers	~2–4%	Low	Insert small FFN bottleneck inside each Transformer block	BERT adapters (Houlsby 2019)
Prefix Tuning	<0.1%	Very low	Prepend learnable "virtual tokens" to K/V in each layer	Li & Liang 2021; good for NLG tasks
Prompt Tuning	<0.01%	Minimal	Learn soft prompt tokens in input embedding space only	Google T5 (Lester 2021); scales well at 10B+
IA³	<0.1%	Minimal	Learn rescaling vectors for K, V, and FFN activations	T-Few; few-shot task adaptation

Deep Dive — LoRA Mathematics & Which Layers to Apply It To

LoRA is applied to the weight matrices of the attention mechanism, most commonly W_q and W_v (sometimes W_k and W_o as well). The FFN layers are typically left frozen in standard LoRA but included in "full LoRA" configurations.

The rank r is the most important hyperparameter. r=4 or r=8 is standard for domain adaptation; r=64+ for instruction tuning. The LoRA scaling factor α controls the magnitude: the effective update is (α/r)·B·A. Setting α=2r gives stable gradients regardless of r.

QLoRA (Dettmers 2023) combines LoRA with 4-bit NF4 (Normal Float 4) quantisation of the frozen base model. It also introduces double quantisation (quantising the quantisation constants) and paged optimisers (offloading optimiser states to CPU RAM). This allows fine-tuning a 65B model on a single 48GB GPU.

Forward pass with LoRA:
h = W₀x + (α/r) · BAx
// W₀: frozen pretrained weight B: d×r (init zeros) A: r×d (init random Gaussian)
// α: scaling (typically = r, so α/r = 1)

QLoRA memory: 65B model × 4 bits/param = ~32GB (vs ~130GB in fp16)

🎓 Deep Dive Questions

Why is B initialised to zeros and A to random in LoRA?
What is NF4 quantisation in QLoRA and why NF4 specifically?
How does LoRA rank r affect the expressiveness of the adaptation?
What is the difference between LoRA and Adapter layers architecturally?

🎓 Interview Questions

What is LoRA and why is it more memory-efficient than full fine-tuning?
What does "low rank" mean in LoRA — why is rank-8 sufficient?
What is QLoRA and how does it differ from LoRA?
What is PEFT and name three PEFT methods with their tradeoffs.
Which attention weight matrices does LoRA typically target?
When would you choose full fine-tuning over LoRA?

11

Alignment — RLHF, SFT, Reward Models, PPO, DPO

🧠 The gist

Here's the thing that took me a while to internalise: a pretrained LLM has no moral compass. Ask it how to make a bomb or how to explain quantum mechanics — same energy, same willingness. RLHF is how we fix that. Humans rate model outputs ("this one's helpful," "this one's harmful"), a reward model learns those preferences, and then RL (PPO) nudges the base model toward the good stuff. DPO (2023) figured out you can skip the entire RL circus — just show the model pairs of responses (one good, one bad) and train it with a classification loss. Way simpler, competitive results.

Figure: Full RLHF Pipeline (InstructGPT / ChatGPT method)

PHASE 1

Supervised Fine-Tuning (SFT)

Fine-tune pretrained model on curated (prompt, ideal-response) pairs written by contractors. Creates a capable instruction follower as starting point.

→

PHASE 2

Reward Model Training

SFT model generates multiple responses per prompt. Humans rank them. Train a separate reward model (RM) to predict human preference scores.

→

PHASE 3

RL with PPO

Use PPO to optimise the SFT policy to maximise reward model score, with a KL penalty against the original SFT model to prevent reward hacking.

→

RESULT

Aligned Assistant

Model that follows instructions, refuses harmful requests, and produces human-preferred outputs. GPT-4, Claude, LLaMA-3-Instruct all use this pipeline.

PPO Objective: max_π E[r_θ(x,y)] − β · KL[π(y|x) ‖ π_SFT(y|x)]
// r_θ = reward model score β = KL penalty weight (prevents reward hacking)

DPO Objective: L_DPO = −E[log σ(β · log(π(y_w|x)/π_ref(y_w|x)) − β · log(π(y_l|x)/π_ref(y_l|x)))]
// y_w = preferred response y_l = rejected response no explicit reward model needed

🏆

RLHF / PPO

Full 3-phase pipeline. Requires training a separate reward model and a complex RL loop. Used by OpenAI (InstructGPT, ChatGPT), Anthropic (Claude RLHF). Powerful but expensive and unstable.

⚡

DPO (Direct Preference Optimisation)

Reformulates RLHF as a classification loss directly on preference pairs — no RL loop, no separate reward model. LLaMA-3-Instruct and many open models use DPO. Much simpler, competitive quality.

🤖

RLAIF (Constitutional AI)

Replace human raters with an AI critic (Claude, GPT-4) to generate preference data. Anthropic's Constitutional AI approach. More scalable than human annotation.

Deep Dive — Reward Hacking, KL Penalty, and Why PPO Is Hard

Reward Hacking: The model learns to maximise the reward model's score without actually being better — e.g. producing very long, verbose outputs that the RM scores highly, or generating safe-sounding but uninformative answers. The KL penalty (β·KL[π‖π_SFT]) acts as a regulariser, preventing the model from drifting too far from the SFT policy.

Why PPO is hard: PPO involves a main model, a frozen reference model (SFT), a reward model, and a value function — all loaded simultaneously. This is 4× the memory of inference. Training is sensitive to hyperparameters (clip ε, KL β, learning rate), and reward collapse (where the policy degenerates) is common. This motivated the development of DPO.

DPO insight: Rafailov et al. (2023) showed that the optimal policy under the PPO objective has a closed-form relationship to the preference data, allowing the reward to be implicitly expressed in terms of the policy ratio — eliminating the need for an explicit reward model.

SimPO (2024): Further simplified DPO by removing the reference model entirely, using length-normalised reward. Used in many latest open models.

🎓 Deep Dive Questions

What is reward hacking and how does the KL penalty prevent it?
Why does DPO not need a reward model?
What is the Constitutional AI approach used by Anthropic?
What are the 4 models loaded simultaneously during RLHF-PPO training?

🎓 Interview Questions

What are the three phases of RLHF? Explain each briefly.
What is a reward model and how is it trained?
What is DPO and how does it improve on RLHF?
What is reward hacking and how is it mitigated?
What is RLAIF and how does Constitutional AI work?
Why is PPO used in RLHF rather than simpler RL algorithms?
What alignment approach does Claude use? (Constitutional AI / RLAIF)

12

Training at Scale — Parallelism, Mixed Precision & Gradient Checkpointing

🧠 The gist

GPT-3 at fp32 is ~700GB of weights. No single GPU exists that can hold that. So you split: data parallelism (same model copied across GPUs, different data batches), pipeline parallelism (different layers on different GPUs), tensor parallelism (split individual weight matrices across GPUs), or ZeRO (DeepSpeed's trick where you shard optimizer states, gradients, and even parameters themselves). Mixed precision (bf16 or fp16 with fp32 master weights) halves memory and doubles throughput on modern GPUs. Getting all this to work together without deadlocks or load imbalance is genuinely hard engineering.

📊

Data Parallelism (DP)

Each GPU holds a full model copy; different micro-batches per GPU. Gradients are averaged via AllReduce after each step. Simplest but requires full model per GPU. Used in DDP (PyTorch).

🔧

Tensor Parallelism (TP)

Split weight matrices across GPUs column/row-wise. Each GPU holds a shard of each layer. Megatron-LM style. Requires fast intra-node NVLink — latency-sensitive.

🔗

Pipeline Parallelism (PP)

Different Transformer layers on different GPUs. GPUs work in sequence — GPU1 processes layers 1–10, GPU2 processes layers 11–20, etc. Micro-batching (GPipe) hides bubble overhead.

⚡

ZeRO (DeepSpeed)

Shard optimiser states (ZeRO-1), gradients (ZeRO-2), and model parameters (ZeRO-3) across DP ranks. Eliminates memory redundancy entirely. Can train trillion-parameter models on commodity clusters.

Technique	What it saves	Communication overhead	Used for
fp16 / bf16	2× memory vs fp32	None	Always on for LLM training. bf16 preferred (wider range, avoids overflow)
Mixed Precision	~2× memory + faster CUDA ops	None	Forward/backward in fp16/bf16; master weights in fp32 for precision
Gradient Checkpointing	Activations (can be 10–20× memory)	Extra ~33% compute (recompute on backward)	When GPU memory is the bottleneck
Gradient Accumulation	Simulates large batch without memory	Fewer AllReduce calls	When effective batch size > what fits in GPU memory
Flash Attention	Attention activation memory O(n) → O(n²) avoided	None	Long-context training & inference; 2–4× faster, same output

Deep Dive — Flash Attention: IO-Aware Exact Attention

Standard attention materialises the full n×n attention matrix in HBM (GPU high-bandwidth memory) — O(n²) memory. For n=8192, that's 67M floats per head per layer. Flash Attention (Dao et al., 2022) avoids this by computing attention in blocks, keeping intermediate results in fast SRAM (on-chip cache) and never writing the full n×n matrix to HBM.

Key insight: The attention computation is memory-bound (most time is spent on memory reads/writes), not compute-bound. By fusing the softmax + attention into a single kernel pass with tiling, Flash Attention achieves 2–4× wallclock speedup and O(n) memory (storing only the final output per row).

Flash Attention 2 improved parallelism across sequence dimension. Flash Attention 3 (2024) adds support for FP8 and async scheduling on H100 GPUs. Used by LLaMA-3, Mistral, and virtually all modern LLM training stacks.

Standard Attention memory: O(n²) — stores full Q·Kᵀ matrix
Flash Attention memory: O(n) — uses tiled block computation in SRAM
// Mathematically identical output — it's a pure implementation optimisation

🎓 Deep Dive Questions

What is Flash Attention and why is standard attention memory-bound?
What is the memory complexity of Flash Attention vs standard attention?
What is gradient checkpointing and what is the compute/memory tradeoff?
What is bf16 and why is it preferred over fp16 for LLM training?

🎓 Interview Questions

Name the four types of parallelism used in large-scale LLM training.
What is ZeRO and what does each stage (1/2/3) shard?
What is mixed precision training and why use bf16 over fp16?
What is gradient checkpointing — what is the tradeoff?
What is Flash Attention and what does it optimise?
How does 3D parallelism (DP+TP+PP) combine all three strategies?

13

Inference Optimisation — Quantisation, Speculative Decoding, Batching

🧠 The gist

You train once; you serve billions of times. And the bottleneck at inference isn't compute — it's memory bandwidth. You're constantly loading weight matrices from HBM into the compute cores, and that transfer is the slow part. Quantisation helps: INT8 halves the model size (and the bandwidth needed), INT4 quarters it, with surprisingly small quality loss. Speculative decoding is the other big win: run a tiny "draft" model that proposes say 5 tokens ahead, then have the real model verify them all in one parallel forward pass. If 4 out of 5 are correct, you just got 4x speedup for free.

🗜️

Quantisation — INT8 / INT4

Represent weights in 8-bit or 4-bit integers instead of 16/32-bit floats. 4-bit quantisation = 4× smaller model. GPTQ, AWQ, GGUF are common formats. Quality loss is minimal at INT8; modest at INT4 with calibration.

🚀

Speculative Decoding

A small "draft" model (e.g. 7B) auto-regressively generates k tokens. The large "oracle" model verifies all k tokens in a single forward pass. If accepted, k tokens are generated for ~1 forward pass cost. 2–3× speedup.

📦

Continuous Batching

Traditional batching waits for all requests to finish. Continuous batching (vLLM, TGI) inserts new requests into the batch as soon as slots free up, maximising GPU utilisation. Fundamental for serving LLMs at scale.

🧱

PagedAttention (vLLM)

KV cache is fragmented and dynamically allocated in "pages" (like OS virtual memory), preventing memory fragmentation. Enables much higher throughput than pre-allocated KV cache. Foundation of vLLM.

Method	Speedup	Memory reduction	Quality impact
INT8 (LLM.int8)	1.2–1.5×	2× vs fp16	Negligible (<0.5% perplexity)
INT4 / GPTQ	2–3×	4× vs fp16	Small (<2% perplexity), depends on calibration
AWQ (Activation-aware)	2–3×	4× vs fp16	Better than GPTQ at same bitwidth
GGUF / llama.cpp	CPU-runnable	2–8× depending on bits	Variable — Q8_0 near-lossless, Q4_K_M small loss
Speculative Decoding	2–4× tokens/s	None (needs draft model)	Zero — mathematically equivalent
Continuous Batching	3–10× throughput	None	Zero

🎓 Interview Questions

What is quantisation and what are the tradeoffs between INT8 and INT4?
How does speculative decoding achieve speedup without quality loss?
What is continuous batching and why does it outperform static batching?
What is PagedAttention (vLLM) and what OS concept does it borrow from?
Why is LLM inference memory-bandwidth-bound rather than compute-bound?
What is the difference between GPTQ and AWQ quantisation?

14

Scaling Laws, Chinchilla & Emergent Abilities

🧠 The gist

The million-dollar question: if I double my model or double my data, which helps more? Early research (Kaplan 2020) said "always go bigger on parameters." Then Chinchilla (Hoffmann 2022) dropped and showed the opposite: most models were undertrained. You should have roughly 20 tokens per parameter — so a 7B model needs 140B tokens. GPT-3 (175B params, 300B tokens) was actually compute-inefficient. LLaMA proved this by training a 7B model on 1 trillion tokens — it beat GPT-3 on many benchmarks despite being 25x smaller. The implication: inference cost matters more than training cost, so you want the smallest model that hits your quality bar, trained on as much data as possible.

Figure: Scaling Laws — Loss vs Compute Budget

Paper	Key Finding	Implication
Kaplan et al. 2020 (OpenAI)	Loss follows power law in N (params), D (data), C (compute). Suggested scaling params faster than data.	Led to GPT-3 (175B on 300B tokens) — now seen as under-trained
Chinchilla (Hoffmann 2022, DeepMind)	For fixed compute C, optimal: N ∝ √C and D ∝ √C. Chinchilla (70B, 1.4T tokens) matched Gopher (280B, 300B tokens)	The dominant guidance today: 20 tokens per parameter for compute-optimal training
LLaMA-1 (Meta 2023)	7B model trained on 1T tokens outperforms 175B model on many benchmarks	Inference-optimal ≠ compute-optimal; for deployment, train smaller models longer
Emergent Abilities (Wei 2022)	Certain capabilities (arithmetic, chain-of-thought, multi-step reasoning) appear abruptly above a threshold model size	Emergent ≠ smooth scaling — some tasks require crossing a threshold (debated in literature)

Chinchilla scaling law (simplified):
L(N, D) = E + A/N^α + B/D^β
// E = irreducible entropy, A/B = fitted constants, α≈0.34, β≈0.28

Compute-optimal ratio: N_opt ≈ D_opt / 20 (≈ 20 tokens per parameter)
// GPT-3: 175B params, 300B tokens → ~1.7 tokens/param (massively under-trained by Chinchilla)
// LLaMA-3 8B: 8B params, 15T tokens → ~1875 tokens/param (inference-optimal)

🎓 Interview Questions

What are scaling laws and what do they predict about model performance?
What is the Chinchilla paper's main finding and how does it differ from Kaplan 2020?
What is the compute-optimal tokens-to-parameters ratio per Chinchilla? (~20:1)
Why is LLaMA-3 8B trained on 15T tokens — isn't that over-training? (Inference optimal)
What are emergent abilities and why are they significant/controversial?
What is the difference between compute-optimal and inference-optimal training?

✦

Complete Quick-Reference Cheat Sheet

Concept	Key Formula / Fact	Remember For Interview
Self-Attention	softmax(QKᵀ/√d_k)·V	Scale by √d_k to prevent softmax saturation
Multi-Head Attention	h parallel heads, concat + W_O	Same total compute; h different relation types
GQA	G groups of Q heads share 1 K/V head	LLaMA-3 70B: 64 Q heads, 8 KV heads
Transformer Block	MHSA + FFN, each with LayerNorm + Residual	Pre-LN in modern models; RMSNorm in LLaMA
FFN expansion	4× d_model inside	Stores factual knowledge; most parameters here
BPE (GPT-4)	cl100k_base, 100K vocab, byte-level	~4 chars/token; LLaMA-3 uses 128K vocab
Positional Encoding	RoPE (LLaMA-3), Learned (GPT-2/3), Sinusoidal (original)	RoPE → relative positions → long context
Decoder-Only	Causal self-attention + CLM objective	GPT-4, Claude, LLaMA — dominant today
Encoder-Decoder	Bidirectional enc + causal dec + cross-attention	T5, BART — better for seq2seq
KV Cache	Cache K,V for all previous tokens	O(n²) → O(n) per step at inference
CLM loss	−(1/N)·Σ log P(x_t\|x_<t)	Cross-entropy over next-token prediction
Perplexity	exp(CLM loss)	Interpretable: how many choices per token
Temperature	logits / T → softmax	T=0 ≈ greedy; T=1 = default; T>1 = creative
Top-p (nucleus)	Sample smallest set with cumsum ≥ p	Preferred over top-k; adapts to distribution shape
LoRA	ΔW = B·A, rank r; only A,B trained	~0.1% params; applied to W_q, W_v
QLoRA	LoRA on 4-bit (NF4) quantised base model	65B model on single 48GB GPU
RLHF	SFT → Reward Model → PPO with KL penalty	Used by GPT-4, Claude; DPO is simpler alternative
DPO	Classification loss on (preferred, rejected) pairs	No reward model; no RL loop; LLaMA-3-Instruct
ZeRO	Shard optim states (1), grads (2), params (3)	DeepSpeed; enables trillion-param training
Flash Attention	Tiled SRAM computation, O(n) memory	2–4× faster; mathematically identical to standard
Quantisation	INT8: 2× smaller; INT4: 4× smaller	GPTQ/AWQ for weights; minimal quality loss at INT8
Speculative Decoding	Draft k tokens → oracle verifies in 1 pass	2–4× speedup; zero quality loss
Chinchilla law	N_opt ≈ D / 20 (20 tokens per parameter)	GPT-3 was under-trained; LLaMA over-trains for inference
Emergent abilities	Appear abruptly above threshold model size	Chain-of-thought, arithmetic — debated cause

Contents

What is an LLM? — The Mental Model

Parameters

Pretraining

Fine-Tuning / RLHF

The Transformer Architecture

Self-Attention — The Core Mechanism

Multi-Head Attention

Tokenization & BPE — What GPT-4, Claude & LLaMA Use

Start: character vocab

Count pairs, merge top

Next most frequent

Repeat until vocab size

Word Embeddings & Positional Encodings

Sinusoidal PE (Original)

Learned Absolute PE (GPT-2/3)

RoPE (LLaMA, GPT-NeoX)

ALiBi (MPT, BLOOM)

Encoder-Decoder vs Decoder-Only Models

Training Objectives, Loss & Perplexity

CLM — Causal Language Modelling

MLM — Masked Language Modelling

Span Denoising (T5)

Decoding Strategies — Temperature, Top-k, Top-p (Nucleus)

Fine-Tuning Paradigms — LoRA, QLoRA, Adapters, PEFT

Alignment — RLHF, SFT, Reward Models, PPO, DPO

Supervised Fine-Tuning (SFT)

Reward Model Training

RL with PPO

Aligned Assistant

RLHF / PPO

DPO (Direct Preference Optimisation)

RLAIF (Constitutional AI)

Training at Scale — Parallelism, Mixed Precision & Gradient Checkpointing

Data Parallelism (DP)

Tensor Parallelism (TP)

Pipeline Parallelism (PP)

ZeRO (DeepSpeed)

Inference Optimisation — Quantisation, Speculative Decoding, Batching

Quantisation — INT8 / INT4

Speculative Decoding

Continuous Batching

PagedAttention (vLLM)

Scaling Laws, Chinchilla & Emergent Abilities

Complete Quick-Reference Cheat Sheet