Tokens, the model’s basic unit

A language model handles nothing but numbers. Before any computation, the text is therefore split into pieces called tokens, and each token receives a number in a fixed vocabulary. This splitting is called tokenisation.

Tokens are neither letters nor whole words. A common word often fits in a single token. A rare word, a proper name or a technical term is cut into several frequent pieces. The most widespread method, called byte-pair encoding, starts from characters and merges, step by step, the pairs that occur most often in a large corpus.

What this explains

• The size of a document, the limit of a conversation and often the cost are counted in tokens, not in words.
• A text in French or German generally needs more tokens than the same text in English, because the vocabularies were built on mostly English corpora.
• Counting the letters of a word or writing it backwards is hard for a model. It does not see the letters, it sees pieces.