What Correlation Actually Measures
r measures exactly one thing: how close the points sit to a straight line. That is narrower than people assume.
What you will be able to do
- Read r as the direction and strength of a linear relationship
- Explain why r = 0 does not mean there is no relationship
- Recognise when Pearson misleads
The Pearson correlation coefficient, written r, is a single number between −1 and 1. The sign gives direction, the magnitude gives strength.
- xᵢ, yᵢ
- the i-th pair of values
- x̄, ȳ
- their respective means
- Σ
- sum over all i
The formula looks heavy but holds one idea: the numerator counts how often x and y depart from their means in the same direction. When x is above its mean at the same time y is above its, the product is positive. The denominator only normalises so the result always lands between −1 and 1.
Steepness is not strength#
This is the confusion that costs most. Two datasets can share an identical r and have wildly different slopes — and two identical slopes can have wildly different r.
import numpy as np x = np.arange(50)shallow = 0.1 * x + np.random.normal(0, 0.5, 50)steep = 10 * x + np.random.normal(0, 50, 50) print(round(np.corrcoef(x, shallow)[0, 1], 3))print(round(np.corrcoef(x, steep)[0, 1], 3))0.944
0.941One rises 0.1 per step, the other 10 — a hundred times steeper. The r values are practically the same. If you want to know how large the effect is, r is the wrong instrument; you want the regression slope.
r = 0 does not mean no relationship#
This follows directly from the word "straight" in the definition. A perfect relationship that curves can have an r of zero.
x = np.arange(-10, 11)y = x ** 2 print(round(np.corrcoef(x, y)[0, 1], 6))0.0y is entirely determined by x. There is no randomness at all. r = 0. Correlation cannot see the relationship because the relationship is not a straight line.
Pearson or Spearman#
| Pearson | Spearman | |
|---|---|---|
| Measures | A linear relationship | A monotonic one (always up, or always down) |
| Works on | The raw values | The ranks |
| Outliers | Very sensitive | Robust |
| Curves | Fails | Still catches it, if the direction is consistent |
| Use when | The scatter looks like a line | There are outliers, or the relationship curves but keeps direction |
from scipy.stats import pearsonr, spearmanr x = np.arange(1, 21)y = x ** 3 # always rising, but clearly curved print(round(pearsonr(x, y).statistic, 3))print(round(spearmanr(x, y).statistic, 3))0.929
1.0Spearman gives 1.0 — and that is right: the relationship really is perfectly monotonic. Pearson gives 0.929 because it was asked to measure straightness on something that is not straight.
And the most expensive one: correlation is not causation#
The sentence is a cliché by now, but the shape of the mistake is rarely spelled out. If A and B are correlated, there are four possibilities, and people usually assume only one of them:
| Possibility | Example |
|---|---|
| A causes B | Prices rise → demand falls |
| B causes A | A busy store → more advertising (not the other way round) |
| C causes both | Hot weather → ice cream sales rise, and drownings rise |
| Coincidence | Just search among thousands of pairs of variables |
A dataset has r = 0.02. What is the correct conclusion?
Two analysts report r = 0.94 for two different datasets. What do you still NOT know?