Lesson 1 · Free · 3 min
$ kubectl get pods -l app=assistant-llm
Watch lesson 1 freeFollow one request to the GPU · 3 min

Run AI in production on Kubernetes

Cloud · 28 lessons · 59 min · Advanced

For engineers who already run services on Kubernetes and have not yet run a model. You follow one assistant service from a single request to a production cluster: GPUs, slow-starting pods, batching and the KV cache, scaling on queue depth, metrics, cost per token, secrets, rollouts and a real incident.

What you'll be able to do

  • Trace one streaming request through gateway, router, model server and GPU, and say where its time goes
  • Write the pod spec for a slow-starting GPU model server with the right probes, shutdown and scheduling
  • Size a model server's concurrency from its KV cache budget and tune batching limits
  • Autoscale a model server on queue depth and decide when scale to zero is worth the cold start
  • Instrument token and GPU metrics, set streaming SLOs and compute cost per token
  • Lock down secrets, egress and tenants, and roll out a new model version safely
  • Diagnose an OOM-killed or unschedulable model pod by checking the right things first

How the lessons work

Lessons that move

Narrated, animated lessons with real terminals and code on screen.

Pause and ask

Stuck? Ask out loud or type. The answer plays as a short video made for your question.

Check yourself

Short quizzes inside the lessons show whether it stuck.

Course content

28 lessons · 59 min · 2 lessons are free
Unit 1 · Follow the request4 lessons · 10 min
01Follow one request to the GPU3 minFree02The two latencies that matter3 minFree
03The stack, layer by layer2 min
04Why agents change the load2 min
Unit 2 · Weights, images and slow starts4 lessons · 8 min
05Weights are not the image2 min
06Where the weights live2 min
07Probes for slow pods2 min
08Shutting down without cutting streams2 min
Unit 3 · GPUs on Kubernetes4 lessons · 8 min
09A GPU is a counted resource2 min
10GPU nodes and taints2 min
11Sharing a GPU: time slices and MIG2 min
12Dynamic resource allocation2 min
Unit 4 · Serving and scaling5 lessons · 10 min
13The KV cache budget2 min
14Batching and its limits2 min
15Scale on the queue, not the CPU2 min
16Cold starts and scale to zero2 min
17Routing that knows the cache2 min
Unit 5 · Observability and cost4 lessons · 8 min
18The metrics that matter2 min
19Tracing an agent run2 min
20SLOs for streaming2 min
21Cost per token2 min
Unit 6 · Safety and operations4 lessons · 8 min
22Secrets and keys2 min
23Lock down the network2 min
24Limits at the gateway2 min
25Rolling out a new model version2 min
Unit 7 · Capstone: the assistant under load3 lessons · 6 min
26The design on one page2 min
27The incident: what to check first2 min
28Fix it and prove it2 min