From Words to Vectors
Models cannot do arithmetic on words. How we turn words into numbers decides what a model can learn — and what it never can.
What you will be able to do
- Explain the weakness of one-hot representations
- Explain the distributional hypothesis and word embeddings
- Tell static from contextual embeddings
Every NLP model starts with the same problem: models work with numbers, and language is made of words. How you turn words into numbers is not a technical detail — it decides whether the model can ever know that "car" and "vehicle" mean almost the same thing.
First attempt: one-hot#
The simplest way: give each word in the vocabulary a position, and represent a word as a vector that is 1 at its position and 0 everywhere else. With 50,000 words, every word is a 50,000-dimensional vector with a single 1.
import numpy as np # vocabulary: [cat, dog, stock]cat = np.array([1, 0, 0])dog = np.array([0, 1, 0])stock = np.array([0, 0, 1]) print(cat @ dog) # dot productprint(cat @ stock)0
0A word is known by its neighbours#
The distributional hypothesis (Firth, 1957): words that appear in similar contexts tend to have similar meanings. "Cat" and "dog" both turn up near "feed", "vet" and "fur". Word2vec (Mikolov et al., 2013) put the idea to work: train a small vector — say 300 dimensions — for each word, so that it is good at predicting the words around it. Words with similar neighbours end up with similar vectors.
One word, many meanings#
Word2vec gives each word one vector. But "bank" in "we sat on the river bank" and in "I deposited cash at the bank" are two different words that happen to be spelled alike. A static embedding blends them into a single point between the two meanings.
Transformer models such as BERT (Devlin et al., 2018) fix this with contextual embeddings: a word's vector is recomputed in every sentence through attention, so the same word gets a different vector depending on the words around it.
| Representation | One word = | Knows similarity? | Knows context? |
|---|---|---|---|
| One-hot / bag-of-words | One position in the vocabulary | No | No |
| TF-IDF | A frequency-based weight | No | No |
| Word2vec, GloVe | One dense vector per word | Yes | No |
| BERT, LLMs | A new vector in every sentence | Yes | Yes |
With one-hot representations, what is the cosine similarity between "car" and "vehicle"?
What do contextual embeddings have that word2vec does not?