Words become vectors
How each token becomes a list of numbers that encodes its meaning, and why geometry replaces the dictionary.
ObjectiveExplain what a vector embedding is and what closeness between two vectors means.
A token number says nothing about meaning. The model therefore associates each token with a vector, that is, a list of several hundred to several thousand numbers. This is called an embedding. These numbers are not written by hand. They are learned during training.
The central idea is simple. Words that appear in similar contexts end up with nearby vectors. “Cat” and “dog” become neighbours, and so do “Bern” and “Geneva”. Meaning becomes a position in a space, and similarity becomes a distance.
Directions that carry meaning
The 2013 work on word2vec revealed a striking phenomenon. Certain directions in the space correspond to relationships. The vector from “man” to “king” resembles the one from “woman” to “queen”. The calculation king minus man plus woman lands near queen. This does not prove that the model understands royalty, only that the regularities of language translate into geometry.
References
- Mikolov, Chen, Corrado et al. (2013). Efficient Estimation of Word Representations in Vector Space. arXiv preprint. arxiv.org/abs/1301.3781