Observability & Debugging

Module 10 · Master Course

45 min · A harness you can't observe is a harness you can't improve

Six observability layers

Layer	Captures
Structured logs	tool calls + I/O + timestamps (Module 1.4 payload)
Traces/spans	execution tree + latency per step
Token accounting	per message/tool/session
Replay	re-run session step-by-step (bug reproduction)
Metrics export	OpenTelemetry/Prometheus (production monitoring)
Session diffing	compare 2 sessions (regression testing)

Module 1.4's 8-field payload is the foundation. Everything builds on it.

OpenTelemetry — the standard

CNCF-backed standard schema for LLM/tool/agent-workflow spans. Adopted by Google, AWS, Azure, Datadog. Use it to avoid vendor lock-in.

Agent-workflow layer (tasks, teams, handoffs, memory) still maturing (ATSC draft). Model-call + tool-call layers stable now.

The 5 post-mortem questions

#	Question
1	What did the model SEE? (full context at failure)
2	What did it DECIDE? (reasoning/tool call)
3	What did the tool RETURN? (result into context)
4	What did the harness DO? (append/retry/halt)
5	What was in CONTEXT? (accumulated state — Module 3)

Most debugging stops at Q2 ('bad model decision'). The real cause is usually Q1, Q3, or Q5.

Replay-driven debugging

Reproduce the failure deterministically by replaying the logged session step-by-step. Reconstruct the exact context at each turn.

Replay against a different model to test whether an upgrade fixes (or causes) the bug.

The invisible-state problem

Bugs that only appear in sessions > 30 turns. Works in testing (short); fails in production (long). Cause: context rot (Module 3).

Without long-session observability, these bugs are invisible until production. The 8-field payload every turn is what makes them visible.

Observability as an n8n fan-out

Per-turn payload (8 fields, Module 1.4)
   │
   ├──► Structured Log (append JSONL) ── debugging substrate
   ├──► Metrics Export (Prometheus) ── production dashboards
   └──► Stuck-loop Check (hash count) ── Module 7 detection

One payload, many consumers. The same 8 fields feed logs, metrics, AND cross-module detectors. Single instrumentation.

Takeaways

6 layers — logs (foundation), traces, tokens, replay, metrics, diffing.
5 post-mortem questions — real causes in Q1/Q3/Q5, not Q2.
Replay reproduces failures deterministically.
Invisible state — long-session bugs need per-turn observability.
One payload, many consumers — the 8-field payload feeds everything.

Next: Module 11 — Security Engineering (90 min). The OWASP Agentic AI Top 10.