Seeing, hearing and speaking: multimodal models

The assistants of 2026 do not read only text. They read a photo, a scanned page, a chart or a screenshot, and many listen and speak. They are called multimodal. The principle remains that of tokens.

An image is cut into small squares, for example 16 pixels a side. Each square becomes a vector, processed as a token by the transformer. For these image tokens to take on meaning, the model learns from a very large number of images paired with their captions. In 2021, the CLIP model brought images and the texts that describe them together in a single vector space in this way, from 400 million pairs.

Voice follows two paths. Often, it is first transcribed into text, then processed as a written request. The Whisper transcription system, published in 2022, was trained on 680,000 hours of recordings. Other models process sound directly as tokens, which lets them pick up the tone and answer with almost no delay.

• Small print and dense tables on a scanned page, where a column can slip.
• Values read from a chart, which are estimates by eye, not data.
• Counting objects or locating an element precisely in the image.
• Invented details. The model can confidently describe what is not in the image.