How an LLM Reads Your Prompt
Your prompt is cut into tokens, turned into vectors, and then every token decides which other tokens matter. Those three steps explain why good prompts work.
What you will be able to do
- Explain tokens and why their count sets cost and limits
- Read similarity of meaning as distance between vectors
- Explain attention and what it means for building context
A large language model (LLM) does one thing: given some text, it computes probabilities for the next token. Answering questions, writing code, summarising a document — all of it is the next token, repeated hundreds of times. To understand why a prompt works you need to know three things that happen before that token is chosen.
1. Tokens: the unit the model actually reads#
The model does not read letters or words. Text is cut into tokens — frequently occurring chunks, roughly a piece of a word. Common English words are often one token; rare words, long numbers and many languages other than English split into more.
2. Embeddings: meaning as geometry#
Each token becomes a vector — a list of hundreds to thousands of numbers — called an embedding. Embeddings are trained so that text with similar meaning ends up close together. "Cat" sits near "dog" and far from "bond yield".
- a, b
- two embedding vectors
- a · b
- the dot product
- ‖a‖
- the length of the vector
import numpy as np def cos(a, b): return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)) a = np.array([1, 2, 0])b = np.array([2, 4, 0]) # same direction, different lengthc = np.array([0, 0, 3]) # perpendicular print(round(cos(a, b), 3))print(round(cos(a, c), 3))1.0
0.0This is the basis of retrieval and RAG: embed the documents and the question, fetch the nearest ones, and put them in the prompt. Similarity of meaning works without matching words — and it can also be wrong in ways keyword search never would be.
3. Attention: every token chooses what to look at#
A token's initial embedding is the same in every sentence. "Bank" has the same vector in "river bank" and "central bank". What tells them apart is attention: in every layer, each token computes how much it should attend to every other token, then mixes in their information by those weights.
- Q
- query — what this token is looking for
- K
- key — what each token offers
- V
- value — the information taken
- dₖ
- the key dimension; dividing keeps the softmax stable
What it means for your prompts#
| Mechanism | Practical consequence |
|---|---|
| Everything in context can be attended to | Irrelevant text is not neutral — it competes for attention. Cut what is not needed. |
| The model continues patterns | Examples (few-shot) and a consistent format are the strongest instruction you can give. |
| Long context is not read evenly | Information in the middle of a very long context is missed more often (Liu et al., 2023). Put what matters at the start or the end. |
| Tokens are cost | A tight context is cheaper, faster and often more accurate. |
You translate a system prompt from English into Indonesian. What is most likely to happen?
Embedding retrieval finds a document about "plan pricing" for the query "how much is the subscription". Why?