Module 2 · Introduction to Deep Learning & Neural Networks

Training Deep Networks: lessons

Optimizers, learning-rate schedules, initialisation, normalisation and regularisation — what makes training actually converge.

4 lessons · 1h 5m

What this module teaches, what it asks of you first and how you practise it are on its syllabus page.

Optimizers: SGD, Momentum, Adam
18 min
What Adam actually does, and when plain SGD still wins.
Included with the track Simulation inside
Learning-Rate Schedules and Warmup
15 min
Cosine decay, warmup, and finding a learning rate in ten minutes.
Included with the track Simulation inside
Initialisation and Normalisation
16 min
Why Xavier and He initialisation exist, and what BatchNorm and LayerNorm fix.
Included with the track Simulation inside
Regularisation: Dropout, Weight Decay, Early Stopping
16 min
Three ways to stop a network memorising, and how to choose between them.
Included with the track Simulation inside

Optimizers: SGD, Momentum, Adam