Lesson 1 · Free · 2 min
$ python run.py
Watch lesson 1 freeWhy vibes do not scale · 2 min

Evals, Observability and Guardrails for LLM Systems

AI Engineering · 28 lessons · 57 min · Advanced

You have shipped an LLM feature and now need to know whether it works, why it failed and how to keep it safe. Follow one support assistant from vibes to a measured, traced, guarded and monitored system, with runnable code at every step.

What you'll be able to do

  • Run error analysis on real traces and turn it into a failure taxonomy
  • Build an eval harness from code checks and validated LLM judges
  • Instrument an LLM and agent system with OpenTelemetry GenAI spans
  • Design guardrails and measure their false positives and latency
  • Monitor quality in production with sampled online evals
  • Gate prompt and model changes in CI and roll them out safely

How the lessons work

Lessons that move

Narrated, animated lessons with real terminals and code on screen.

Pause and ask

Stuck? Ask out loud or type. The answer plays as a short video made for your question.

Check yourself

Short quizzes inside the lessons show whether it stuck.

Course content

28 lessons · 57 min · 2 lessons are free
Unit 1 · Why vibes fail4 lessons · 8 min
01Why vibes do not scale2 minFree02Read fifty traces2 minFree
03From notes to a failure taxonomy2 min
04Three kinds of evals2 min
Unit 2 · Building evals5 lessons · 10 min
05A dataset from real usage2 min
06Code-based checks2 min
07A harness in twenty lines2 min
08The same eval in Inspect AI2 min
09Metrics that mean something2 min
Unit 3 · LLM judges4 lessons · 8 min
10One judge, one failure2 min
11Validate the judge against humans2 min
12Where judges go wrong2 min
13Correcting the pass rate2 min
Unit 4 · Tracing and observability4 lessons · 8 min
14Spans for LLM calls2 min
15OpenTelemetry GenAI conventions2 min
16What to log, and what not to2 min
17Debugging an agent trace2 min
Unit 5 · Guardrails4 lessons · 8 min
18Where guardrails sit2 min
19Prompt injection and the rule of two2 min
20Validating outputs and catching PII2 min
21The price of a guardrail2 min
Unit 6 · Production4 lessons · 8 min
22Online evals and sampling2 min
23Drift and feedback loops2 min
24Gating changes in CI2 min
25Rolling out prompt and model changes2 min
Unit 7 · Capstone: the gift card incident3 lessons · 6 min
26The alert2 min
27Diagnosis from the trace2 min
28Closing the loop2 min