Module: 10 · Duration: ~45 min
[SLIDE 1] Module Ten: Observability and Debugging. Forty-five minutes. A harness you can't observe is a harness you can't improve. Module One-point-four gave you the per-turn payload; this module is the full observability stack it enables.
[SLIDE 2] Six observability layers. Structured logs — tool calls with inputs, outputs, timestamps; Module One-point-four's eight-field payload is this layer, the foundation. Traces and spans — the full execution tree with latency per step, for performance profiling. Token accounting — usage per message, per tool, per session, for cost attribution. Replay — the ability to re-run a logged session step by step, for bug reproduction. Metrics export — OpenTelemetry or Prometheus counters, for production monitoring. And session diffing — comparing two sessions to find behavioral change, for regression testing.
[SLIDE 3] OpenTelemetry is the standard. CNCF-backed schema for LLM calls, tool calls, and agent workflows. Adopted by Google Cloud, AWS, Azure, and Datadog. Using OTel avoids observability vendor lock-in — switch backends without changing instrumentation. The agent-workflow layer — tasks, teams, handoffs, memory — is still maturing; the model-call and tool-call layers are stable now.
[SLIDE 4] The five post-mortem questions, in order. One: what did the model SEE — the full context at the moment of failure. Two: what did it DECIDE — its reasoning or tool call. Three: what did the tool RETURN — the result inserted into context. Four: what did the harness DO with that — append, retry, halt. Five: what was in CONTEXT at the time — the accumulated state, Module Three. Most debugging stops at question two, "the model made a bad decision." The real cause is usually in one, three, or five: the model saw the wrong thing, the tool returned something misleading, or the context had rotted.
[SLIDE 5] Replay-driven debugging. Reproduce the failure deterministically by replaying the logged session step by step. The structured logs with input and output hashes let you reconstruct the exact context at each turn. Replay against a different model to test whether a model upgrade fixes or causes the bug.
[SLIDE 6] The invisible-state problem. Bugs that only appear in sessions longer than thirty turns. The harness works fine in testing — short sessions — and fails in production — long sessions. The cause is usually context rot, Module Three. The accumulated history degrades the model's grip on the task. Without long-session observability, these bugs are invisible until production. The eight-field payload every turn is what makes them visible.
[SLIDE 7] Per the visual stack, observability is a fan-out from the per-turn payload. One payload feeds three consumers: the structured log, the metrics export, and the stuck-loop check. The same eight fields that Module One-point-four defined are the single substrate for all six observability layers AND for Module Seven's stuck-loop detection. One instrumentation, many consumers.
[SLIDE 8] Five takeaways. Six layers — logs are the foundation, traces, tokens, replay, metrics, diffing. Five post-mortem questions — real causes in Q1, Q3, Q5, not Q2. Replay reproduces failures deterministically. Invisible state — long-session bugs need per-turn observability. One payload, many consumers — the eight-field payload feeds everything. Next: Module Eleven — Security Engineering. Ninety minutes. The OWASP Agentic AI Top Ten.
# Teaching Script — Module 10: Observability & Debugging **Module**: 10 · **Duration**: ~45 min [SLIDE 1] Module Ten: Observability and Debugging. Forty-five minutes. A harness you can't observe is a harness you can't improve. Module One-point-four gave you the per-turn payload; this module is the full observability stack it enables. [SLIDE 2] Six observability layers. Structured logs — tool calls with inputs, outputs, timestamps; Module One-point-four's eight-field payload is this layer, the foundation. Traces and spans — the full execution tree with latency per step, for performance profiling. Token accounting — usage per message, per tool, per session, for cost attribution. Replay — the ability to re-run a logged session step by step, for bug reproduction. Metrics export — OpenTelemetry or Prometheus counters, for production monitoring. And session diffing — comparing two sessions to find behavioral change, for regression testing. [SLIDE 3] OpenTelemetry is the standard. CNCF-backed schema for LLM calls, tool calls, and agent workflows. Adopted by Google Cloud, AWS, Azure, and Datadog. Using OTel avoids observability vendor lock-in — switch backends without changing instrumentation. The agent-workflow layer — tasks, teams, handoffs, memory — is still maturing; the model-call and tool-call layers are stable now. [SLIDE 4] The five post-mortem questions, in order. One: what did the model SEE — the full context at the moment of failure. Two: what did it DECIDE — its reasoning or tool call. Three: what did the tool RETURN — the result inserted into context. Four: what did the harness DO with that — append, retry, halt. Five: what was in CONTEXT at the time — the accumulated state, Module Three. Most debugging stops at question two, "the model made a bad decision." The real cause is usually in one, three, or five: the model saw the wrong thing, the tool returned something misleading, or the context had rotted. [SLIDE 5] Replay-driven debugging. Reproduce the failure deterministically by replaying the logged session step by step. The structured logs with input and output hashes let you reconstruct the exact context at each turn. Replay against a different model to test whether a model upgrade fixes or causes the bug. [SLIDE 6] The invisible-state problem. Bugs that only appear in sessions longer than thirty turns. The harness works fine in testing — short sessions — and fails in production — long sessions. The cause is usually context rot, Module Three. The accumulated history degrades the model's grip on the task. Without long-session observability, these bugs are invisible until production. The eight-field payload every turn is what makes them visible. [SLIDE 7] Per the visual stack, observability is a fan-out from the per-turn payload. One payload feeds three consumers: the structured log, the metrics export, and the stuck-loop check. The same eight fields that Module One-point-four defined are the single substrate for all six observability layers AND for Module Seven's stuck-loop detection. One instrumentation, many consumers. [SLIDE 8] Five takeaways. Six layers — logs are the foundation, traces, tokens, replay, metrics, diffing. Five post-mortem questions — real causes in Q1, Q3, Q5, not Q2. Replay reproduces failures deterministically. Invisible state — long-session bugs need per-turn observability. One payload, many consumers — the eight-field payload feeds everything. Next: Module Eleven — Security Engineering. Ninety minutes. The OWASP Agentic AI Top Ten.