What a Neural Network Actually Computes
A neuron is a weighted sum plus one bending function. Stack a few and a straight line becomes whatever boundary you need.
What you will be able to do
- Write down what a single neuron computes
- Explain why depth is useless without a nonlinear activation
- Read the effect of the learning rate on gradient descent
The word "neural" makes people picture a brain. Set that aside for a moment. One artificial neuron does three things: multiplies each input by a weight, adds them up together with a bias, and passes the result through a function called the activation.
- x
- the input vector
- w
- the weights — learned
- b
- the bias — also learned
- φ
- the activation, e.g. ReLU: φ(z) = max(0, z)
If φ is the sigmoid, this single neuron is exactly logistic regression. Its decision boundary is where w · x + b = 0 — and that is always a straight line (or a flat plane, in more dimensions).
Why the nonlinearity is the whole trick#
A layer is many neurons reading the same input. A "deep" network is layers stacked: the output of one becomes the input of the next. The temptation is to think that stacking more layers automatically buys a more powerful model. Without φ, it does not.
Layers bend the space#
With two inputs, every neuron in the first hidden layer draws one line across the plane. ReLU switches off one side of that line. The next layer combines those pieces, so the final boundary is built from straight segments — and with enough segments it can bend as far as the data needs.
Two ReLU neurons can solve XOR, but training often gets stuck depending on where the weights start. Extra neurons are rarely needed for the representation — they help the optimisation enormously. You will see that pattern again and again in deep learning: networks larger than they "should" need to be are easier to train.
How the weights learn#
Training means finding the weights that make the loss — a measure of how wrong the predictions are — as small as possible. Gradient descent does it in small repeated steps: work out the direction in which the loss rises fastest (the gradient), and step the opposite way.
- L
- the loss on the current batch
- ∂L/∂w
- the gradient of the loss with respect to the weights
- η
- the learning rate — the step size
A real network has millions of weights instead of two, but the rule is the same. What makes it feasible is backpropagation: computing the gradient for every weight at once with the chain rule, for roughly the cost of one extra forward pass. That is the next lesson.
The same thing, in PyTorch#
import torchfrom torch import nn model = nn.Sequential( nn.Linear(2, 8), # 2 inputs -> 8 hidden neurons nn.ReLU(), # without this, the model stays linear nn.Linear(8, 1), # 8 -> 1 logit) loss_fn = nn.BCEWithLogitsLoss()optimizer = torch.optim.SGD(model.parameters(), lr=0.1) for X_batch, y_batch in loader: optimizer.zero_grad() # discard the previous step's gradients loss = loss_fn(model(X_batch), y_batch) loss.backward() # backpropagation optimizer.step() # w <- w - lr * gradA network has 10 Linear layers with no activation functions between them. What can it learn?
Your loss climbs and turns into NaN after a few steps. What is the first suspect?