Lab Specification — Module 10: Observability & Debugging

function emitTurn(payload) {
  const record = { ...payload, ts: new Date().toISOString() };
  fs.appendFileSync("/tmp/trace.jsonl", JSON.stringify(record) + "\n");
}
// In the loop, after each model call and tool call:
emitTurn({ trace_id, turn: turnCount++, tool_name, input_hash: hash(input), output_hash: hash(result), latency_ms, token_delta, stop_reason });

function emitMetrics() {
  const lines = fs.readFileSync("/tmp/trace.jsonl","utf8").trim().split("\n");
  const records = lines.map(l => JSON.parse(l));
  const totalTokens = records.reduce((s,r) => s + r.token_delta, 0);
  const totalLatency = records.reduce((s,r) => s + r.latency_ms, 0);
  fs.writeFileSync("/tmp/metrics.prom", `# TYPE agent_tokens_total counter\nagent_tokens_total ${totalTokens}\n# TYPE agent_latency_ms_total counter\nagent_latency_ms_total ${totalLatency}\n`);
}

# Lab Specification — Module 10: Observability & Debugging

**Module**: 10 · **Duration**: 40 min · **Environment**: Codespace, Node 18+.

## Learning objectives
1. **Instrument** a harness with the 8-field per-turn payload (structured logs).
2. **Export** metrics in Prometheus format.
3. **Debug via replay**: intentionally introduce a bug; find it using ONLY observability data (not model output).

## Phase 1 — Instrument (15 min)
```typescript
function emitTurn(payload) {
  const record = { ...payload, ts: new Date().toISOString() };
  fs.appendFileSync("/tmp/trace.jsonl", JSON.stringify(record) + "\n");
}
// In the loop, after each model call and tool call:
emitTurn({ trace_id, turn: turnCount++, tool_name, input_hash: hash(input), output_hash: hash(result), latency_ms, token_delta, stop_reason });
```

## Phase 2 — Metrics export (10 min)
```typescript
function emitMetrics() {
  const lines = fs.readFileSync("/tmp/trace.jsonl","utf8").trim().split("\n");
  const records = lines.map(l => JSON.parse(l));
  const totalTokens = records.reduce((s,r) => s + r.token_delta, 0);
  const totalLatency = records.reduce((s,r) => s + r.latency_ms, 0);
  fs.writeFileSync("/tmp/metrics.prom", `# TYPE agent_tokens_total counter\nagent_tokens_total ${totalTokens}\n# TYPE agent_latency_ms_total counter\nagent_latency_ms_total ${totalLatency}\n`);
}
```

## Phase 3 — Debug via replay (15 min)
Intentionally break the agent (e.g., make read_file return truncated content). Run a task that fails. Now find the bug using ONLY /tmp/trace.jsonl — not the model's output. Which fields reveal it? (latency_ms anomaly? token_delta drop? input_hash+output_hash showing the truncation?)

## Deliverables
- [ ] Phase 1: trace.jsonl with 8 fields per turn
- [ ] Phase 2: metrics.prom with token + latency counters
- [ ] Phase 3: the bug found via observability only; which field(s) revealed it

## Stretch goals
1. Add OpenTelemetry export (OTLP) instead of Prometheus format.
2. Implement session diffing: run the same task twice (one with a prompt change), diff the two trace.jsonl files to find divergence.