Word Embeddings Explained: How AI Represents Words

Quick answer: A word embedding is a learned list of numbers, called a vector, that represents how a word is used. Words used in similar contexts often get similar vectors, which lets software compare words mathematically, for example with cosine similarity. In some trained word2vec models, king − man + woman lands near queen. Modern language models turn text into tokens, map each token to an embedding, and then update those vectors through transformer layers using the surrounding context.

Computers cannot work with words directly; they work with numbers. So how does AI know that “cat” is closer to “dog” than to “car”? It gives every word a position in a space of many dimensions. These positions are called embeddings, and they are one of the most useful ideas in AI.

From Words to Vectors

An embedding is a vector, a list of numbers. Older methods used one-hot vectors (a 1 in the word’s own position and 0 everywhere else), which say nothing about meaning; learned embeddings are short, dense vectors where similar usage means similar numbers. Real embeddings have hundreds or thousands of numbers. The coordinates and labels below are invented, and chosen so the analogy works exactly, to demonstrate vector arithmetic:

WordRoyaltyMaleFemale
king0.90.90.1
queen0.90.10.9
man0.10.90.1
woman0.10.10.9

In real embeddings, the dimensions are learned and usually have no simple names; these labels are for illustration.

The Famous Example: King − Man + Woman

Subtract and add vectors entry by entry:

\[ \text{king} – \text{man} + \text{woman} = (0.9 – 0.1 + 0.1,\ 0.9 – 0.9 + 0.1,\ 0.1 – 0.1 + 0.9) = (0.9,\ 0.1,\ 0.9), \]

which is exactly “queen” in our toy table. In 2013, Tomas Mikolov and colleagues at Google reported that some trained word2vec models show relationships like this, where directions in the space capture patterns such as gender or tense. In real models the result is only near “queen,” the relationship is approximate and model-dependent, and the evaluations excluded the input words from the candidate answers.

Measuring Similarity: Cosine Similarity

To compare two embeddings, AI often uses cosine similarity, the cosine of the angle between the vectors:

\[ \cos\theta = \frac{a \cdot b}{\|a\|\,\|b\|}. \]

For nonzero vectors it ranges from −1 to 1: 1 means the same direction, 0 perpendicular, −1 opposite. Unlike the raw dot product, it ignores length: doubling a vector changes its dot products but not its cosine. These scores measure direction, not identical or unrelated meaning. Example: \((1, 2)\) and \((2, 4)\) point the same way, so the similarity is 1; \((1, 2)\) and \((2, -1)\) give \(1(2) + 2(-1) = 0\), so they are unrelated. See linear algebra for AI for dot products.

How Embeddings Are Learned

The key idea comes from linguist J. R. Firth (“A Synopsis of Linguistic Theory, 1930–1955,” published 1957): “You shall know a word by the company it keeps.” Word2vec reads billions of words and adjusts vectors so that a word predicts its neighbors (the skip-gram method) or the neighbors predict the word (CBOW). Words used in similar contexts, like “cat” and “dog,” end up with similar vectors. The adjusting is done with gradient descent (see gradient descent explained).

Embeddings in Chatbots

Modern chatbots split text into tokens, and an embedding layer maps each token ID to a vector. Unlike word2vec’s fixed vectors, transformer layers then update these representations using context, so “bank” in “river bank” and “bank loan” ends up with different representations. See the math behind ChatGPT.

Where Embeddings Are Used

  • Search: finding documents with similar meaning, not just matching words, usually with sentence or passage embeddings rather than single words.
  • Recommendations: songs, products, videos and users can be embedded too.
  • Translation: words with the same meaning in different languages land near each other.
  • AI memory and retrieval: chatbots look up relevant documents by embedding similarity.

Limits and Bias

Embeddings learn from human text, so they also learn its biases. Researchers have shown, for example, that word embeddings can associate occupations with gender stereotypes. Careful developers measure and reduce such biases.

Common Misconceptions

“Each number means something specific.” Usually not; meaning is spread across many dimensions.

“Embeddings understand words.” They capture patterns of use, which is powerful but not the same as human understanding.

“Only words get embeddings.” Images, sounds, users and products can all be embedded.

Watch: Word Embeddings Explained

If you learn better by watching, these videos are worth your time.

What are Word Embeddings? (IBM Technology). A short, clear introduction to embeddings.
Word Embedding and Word2Vec, Clearly Explained!!! (StatQuest with Josh Starmer). How word2vec learns embeddings, step by step.
Transformers, the tech behind LLMs (3Blue1Brown). Includes a beautiful visual explanation of embeddings.

Check Your Understanding

Five quick questions to see how much of this has stuck. Pick an answer and press Submit; you will see a short explanation either way.

Answer each question and press Submit to see your result right away.

1. In the toy table, what is king − man + woman?
Found a bug? Let us know!
2. What is the cosine similarity of (1, 2) and (2, 4)?
Found a bug? Let us know!
3. Are the toy embedding coordinates in this article measured from real data?
Found a bug? Let us know!
4. What does a cosine similarity of 0 mean?
Found a bug? Let us know!
5. What is different about embeddings in transformer chatbots?
Found a bug? Let us know!

Frequently Asked Questions

What are word embeddings?

Vectors of numbers that represent words so that words used in similar ways have similar vectors. AI learns them from large amounts of text.

How does king − man + woman = queen work?

In word embeddings such as word2vec, directions in the vector space can capture relationships like gender. Subtracting man and adding woman to king moves to a point near queen.

What is cosine similarity?

A measure of how similar two vectors are, equal to their dot product divided by the product of their lengths. It ranges from −1 (opposite) to 1 (same direction).

What is word2vec?

A method published by Tomas Mikolov and colleagues at Google in 2013 that learns word embeddings by predicting nearby words in large amounts of text.

Do chatbots use word embeddings?

Yes. They turn tokens into embeddings first, then transformer layers update those vectors based on context.

Keep Exploring

Sources and Further Reading

Related to This Article

What people say about "Word Embeddings Explained: How AI Represents Words - Effortless Math"?

No one replied yet.

Leave a Reply