https://datatweets.com/courses/llm-evaluation/production-monitoring-and-guardrails/Recent content in Production Monitoring & Guardrails on DATATWEETSHugoenCopyright (c) 2026 DatatweetsTue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/production-monitoring-and-guardrails/lesson-1-monitoring-dashboards-and-alerts/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/production-monitoring-and-guardrails/lesson-1-monitoring-dashboards-and-alerts/Module 7 captured what Docent did; this lesson watches it in real time. You’ll learn the six-ish metrics worth monitoring continuously for an LLM app – volume, latency percentiles, cost, error rate, refusal rate, and online quality proxies – and separate leading signals that predict trouble from lagging ones that confirm it. You’ll build a deterministic per-minute metrics stream with a deliberate incident at 09:07, compute rolling-window baselines that expose the regression a lifetime average hides, and write an alert engine that fires static and relative-to-baseline alerts. The engine flags the real spike – p95 latency 4.88s and refusal rate 40.1% at three times baseline – and everything reproduces byte-for-byte.https://datatweets.com/courses/llm-evaluation/production-monitoring-and-guardrails/lesson-2-detecting-drift-and-regressions/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/production-monitoring-and-guardrails/lesson-2-detecting-drift-and-regressions/Quality rarely fails loudly in production; it drifts. This lesson separates input/data drift (users start asking different questions than your golden set covers) from quality/behavior drift (answers get worse because a prompt or model changed underneath you) and from a sudden regression (a specific change caused a step drop). You’ll learn detection methods — comparing recent versus baseline metric windows, monitoring the input distribution for new topics, canary sampling live outputs, and watching a rolling quality proxy or refusal rate — and how to tell small-sample wobble from a real shift with a threshold and a simple significance check. Then you build a fully deterministic detector: one part flags a surge of billing questions Docent’s golden set under-covers via a total-variation distance of 0.34, the other flags a quality regression where a prompt change dropped the recent window’s mean judge score from 0.865 to 0.655, confirmed by a two-proportion z of 4.12.https://datatweets.com/courses/llm-evaluation/production-monitoring-and-guardrails/lesson-3-regression-tests-in-ci/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/production-monitoring-and-guardrails/lesson-3-regression-tests-in-ci/A prompt tweak or model upgrade can silently regress quality — and ‘we’ll eyeball it’ doesn’t scale. This lesson turns your eval suite into a CI test: a fixed golden set, a deterministic metric per case, critical cases that must never fail (the out-of-scope refusal, a known past bug), and a per-suite threshold that fails the build when quality drops. You’ll build a pytest-style regression suite for Docent, watch it PASS on a good version and BLOCK the release when a candidate regresses two critical cases, and run the deterministic suite twice for byte-identical results — plus one live-backed critical check against claude-haiku-4-5.https://datatweets.com/courses/llm-evaluation/production-monitoring-and-guardrails/lesson-4-guardrails-and-human-feedback/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/production-monitoring-and-guardrails/lesson-4-guardrails-and-human-feedback/Guardrails are checks that run around the model, not inside it: input guardrails block prompt-injection and off-topic requests and redact PII before the call; output guardrails validate format, catch ungrounded or unsafe answers, and spot false refusals after it. Each can allow, block, rewrite, or flag for review. You’ll build deterministic input and output guardrails for Docent that block bad cases and pass good ones (reproducible byte-for-byte), add a lightweight live judge-as-guardrail on claude-haiku-4-5 that allows a grounded answer and flags a hallucinated one, then aggregate synthetic thumbs feedback — computing a thumbs-down rate and promoting downvoted, corrected interactions back into Module 2’s golden dataset.https://datatweets.com/courses/llm-evaluation/production-monitoring-and-guardrails/lesson-5-guided-project-monitoring-and-guardrails-harness/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/production-monitoring-and-guardrails/lesson-5-guided-project-monitoring-and-guardrails-harness/The capstone for Module 8. You combine the four production disciplines the module taught into one harness for Docent. Stage 1 wraps Docent in an input guardrail (blocking injection and off-topic questions before any model call) and an output guardrail (a format and numeric-grounding check), returning the answer plus both guardrail decisions. Stage 2 feeds a simulated hourly metric stream with an injected incident through a rolling-window alerter that fires OPEN when grounded_rate drops and p95 latency spikes, then RESOLVED after a rollback. Stage 3 runs a pytest-style regression suite as a CI gate: a good build passes 5/5 and ships, a regressed build fails one critical case and is blocked. Stage 4 aggregates synthetic human thumbs, promotes the thumbs-down cases into the golden set, and prints the deliverable: a one-screen production-readiness report combining guardrail block rate 0.60, alert status HEALTHY, CI gate PASS, and feedback backlog 2 into a READY TO SHIP verdict. The deterministic stages reproduce exactly; one small live guarded call shows the real assistant behind the guardrails.