Title here
Summary here
Module · 5 lessons
Observability & Tracing
Evaluation tells you if the app is good; observability tells you what it's doing in production. Learn structured logging, traces and spans, and cost/latency/token accounting — then instrument Docent so every request is logged, traced, and costed.
At a glance
Level
Intermediate
Lessons
5 lessons
Time to complete
1 week
Cost
Free forever · no sign-up
Welcome to Observability & Tracing, the seventh module. Everything so far has been evaluation: judging quality against datasets before you ship. But once Docent is live and serving real users, a different question takes over — what is it actually doing right now, and when something breaks at 2 a.m., can you see why? That’s observability, and the rule is blunt: you can only debug what you captured.
You’ll build the capture layer. First, structured logging — turning every model call into a queryable record of prompt, response, model, tokens, latency, and cost. Then traces and spans — following a single request as it flows through retrieval, tool calls, and generation, so you can see exactly where time and tokens went. Finally, rigorous cost, latency, and token accounting, including the percentiles (p50, p95) that reveal the slow tail an average hides. The guided project wires all of it into Docent, so every request leaves behind a trace, a cost, and a latency you can inspect and aggregate.
Start with Lesson 1, where evaluation hands off to observability — and “is it good?” becomes “what is it doing?”
Lessons in this module
1
From Evaluation to Observability
Make the shift from pre-ship evaluation to in-production observability concrete by capturing one live Docent call as a structured observation record of model, tokens, latency, and answer.
2
Structured Logging for LLM Calls
Turn every Docent call into one queryable JSON log record — model, tokens, latency, and cost — using a logging wrapper, and keep the API key out of every record.
3
Traces & Spans: Following a Request
Learn distributed tracing for an LLM pipeline: model one Docent request as a tree of timed spans, build a minimal pure-Python tracer, and instrument a live claude-haiku-4-5 call to see where the time and tokens went.
4
Cost, Latency & Token Accounting
Measure the operational metrics that decide whether Docent is viable in production: input vs output tokens from usage, real per-request and total cost from an illustrative price, and latency percentiles (p50/p95/p99) that expose the slow tail a mean hides.
5
Guided Project: Instrumenting Docent
Add a full observability layer to Docent — structured logging, traces and spans, and cost/latency/token accounting — then run it live over a batch and read the mini dashboard.
Achievement
Complete all 5 lessons to finish the Observability & Tracing module.
