https://datatweets.com/courses/llm-evaluation/observability-and-tracing/Recent content in Observability & Tracing on DATATWEETSHugoenCopyright (c) 2026 DatatweetsTue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/observability-and-tracing/lesson-1-from-evaluation-to-observability/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/observability-and-tracing/lesson-1-from-evaluation-to-observability/Evaluation asks &lsquo;is it good?&rsquo; against a dataset before you ship; observability asks &lsquo;what is it doing right now?&rsquo; on live traffic. This lesson frames that handoff and introduces the three pillars of observability for an LLM app: logs (discrete per-call events), metrics (aggregate numbers over time like request rate and p95 latency and cost per day), and traces (the causal path of one request through retrieval and generation). You&rsquo;ll see exactly what an LLM app must capture beyond ordinary web telemetry, learn why LLM observability is distinct because of non-determinism and per-call cost and silent quality drift, and make one real live Docent call while building a structured observation record of its model, tokens, and latency to preview the module.https://datatweets.com/courses/llm-evaluation/observability-and-tracing/lesson-2-structured-logging/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/observability-and-tracing/lesson-2-structured-logging/Print statements can&rsquo;t be filtered, summed, or alerted on; structured JSON logs can. This lesson defines the fields every LLM call should emit — timestamp, trace id, model, params, input/output, token counts, latency, estimated cost, success, and app metadata — then builds a logged_docent() wrapper that emits one JSON record per real call to claude-haiku-4-5. You&rsquo;ll run it live, read the actual tokens/latency/cost it captured, aggregate a fixed batch of records deterministically (count by model, total tokens, mean latency), and enforce the one non-negotiable rule: the API key never appears in a log.https://datatweets.com/courses/llm-evaluation/observability-and-tracing/lesson-3-traces-and-spans/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/observability-and-tracing/lesson-3-traces-and-spans/A single Docent request touches retrieval and one or more model calls, and a flat log line can&rsquo;t tell you which step was slow or which one failed. Distributed tracing can: one request becomes a trace, a tree of timed spans, each recording a name, a parent, start and end times, and attributes like tokens and doc ids. This lesson builds a minimal Span/Tracer in pure Python &ndash; a context manager you write as <code>with tracer.span(&quot;generate&quot;) as s:</code> &ndash; proves its tree-and-duration math is reproducible on a fixed clock, then instruments a real Docent request so a root docent.request span contains a fast retrieve child and a live claude-haiku-4-5 generate child, revealing that the model call was over 95 percent of the latency. It closes by tracing the agentic Docent so multiple generate and tool spans appear in one multi-span trace.https://datatweets.com/courses/llm-evaluation/observability-and-tracing/lesson-4-cost-latency-and-token-accounting/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/observability-and-tracing/lesson-4-cost-latency-and-token-accounting/Evaluation says whether Docent is good; operational accounting says whether you can afford to run it. This lesson captures real per-request input and output tokens from msg.usage and real wall-clock latency over a live batch of Docent calls, computes per-request and total cost from a clearly-illustrative claude-haiku-4-5 price, and implements p50/p95/p99 latency percentiles to show why the slow tail — not the mean — is what users feel and what SLAs target. It ends with a single aggregate operational report: total and mean tokens, total cost, cost per 1000 requests, and mean/p50/p95 latency, with the aggregation math proven reproducible.https://datatweets.com/courses/llm-evaluation/observability-and-tracing/lesson-5-guided-project-instrumenting-docent/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/observability-and-tracing/lesson-5-guided-project-instrumenting-docent/The capstone for Module 7. You wrap Docent in a real observability layer that combines everything the module taught: a small Tracer and Span from the tracing lesson, a structured log record from the logging lesson, and an illustrative per-token cost helper from the accounting lesson. You assemble instrumented_docent(question), which traces the retrieve and generate stages, logs one structured record per request, and returns the answer, its trace, and its log record. Then you run it live on claude-haiku-4-5 over seven golden questions, inject one failed call, and aggregate the collected records into a mini dashboard — requests, tokens, cost, mean/p50/p95 latency, and error rate — that reproduces exactly on a re-run. The payoff is Stage 4: using the dashboard and traces to find the slowest request, see where cost concentrates, and decide what to optimize.