Module 5 · LLMs, Prompt Engineering & Context Engineering

Evaluating LLM Applications: lessons

Eval sets, LLM-as-judge and regression tests for prompts — replacing "looks good to me" with a number.

3 lessons · 44 min

What this module teaches, what it asks of you first and how you practise it are on its syllabus page.

Building an Eval Set
16 min
An eval set is your own traffic, chosen deliberately, with criteria code can check — not the three questions you tried before shipping.
Included with the track Simulation inside
LLM as a Judge
15 min
A model can grade another model's answers — provided the rubric is narrow, the biases are known, and its agreement with humans has been measured with kappa, not a bare percentage.
Included with the track
Regression-Testing Prompts
13 min
A prompt is code without a compiler. Put it in the repository, run the eval set on every change, and block the release when cases that used to pass now fail.
Included with the track

Building an Eval Set