Speculative decoding, hands on

A small draft model guesses several tokens ahead. The big target model checks all the guesses in one forward pass and keeps the ones it agrees with. Three small labs below let you see why that is faster and why the output stays correct.

Draft, verify, commit

The target model wants to produce the sentence below. Press the button to move through one round. Each round costs one pass of the big model.

Text so far (grey = prompt, green = accepted output)
Draft model guesses
Nothing drafted yet.
Target model verdict at each position
Waiting for a draft.
Press Draft. The small model will write k tokens one after another, which is cheap.
Tokens produced0
Big-model passes0
Tokens per pass-
Normal decoding needs0

Try a low accuracy like 20%. Notice you still get at least one token per pass, so you never do worse than normal decoding in passes.

Why sampling stays exact

With sampling, the draft model has its own distribution q and the target has p. Set both over four tokens. The draft proposes x from q. Accept it with probability min(1, p(x)/q(x)). If rejected, resample from the leftover distribution max(0, p - q), normalized.

Draft q (drag to change)
Target p (drag to change)
Accept chance per token, min(1, p/q)
Residual distribution used after a rejection

Measured distribution of final tokens (should match p)

When does it pay off?

Per round, the expected number of tokens is (1 - ak+1) / (1 - a), where a is the per-token acceptance rate. The round costs one target pass plus k draft steps, each costing a fraction c of a target pass.

Real systems also lose gain at large batch sizes, because the target pass stops being memory bound. This model ignores that.

Check yourself