Words in. Numbers out.
A tokenizer turns text into pieces, then looks up a number for each piece.
Build your vocabulary and watch it happen, one keystroke at a time.
01 Vocabulary
Your dictionary of tokens and their unique IDs.
Starts with 1,000 words from the Google English frequency list ↗. IDs are assigned by rank. [UNK] = 0 is reserved.
02 Input text
03 Tokenized output
● LIVEWhat’s happening here? ＋
Each token is a piece of text with a unique integer ID. In Whole words mode, words (including internal apostrophes) and punctuation are looked up individually. In Longest match mode, the longest vocabulary entry at the current position wins; unmatched characters become individual unknown tokens. Spaces are skipped unless included inside a matched token. Matching ignores case by default. Unknown pieces use the reserved ID 0.
This is a teaching tokenizer. Production tokenizers often learn subword vocabularies with methods such as BPE and may encode whitespace too. The IDs here are your own dictionary, not a specific model’s IDs.