LLM Observability & Tracing
Why LLM systems need their own observability stack
Traditional application observability (logs, metrics, traces) was built for deterministic code paths — the same input reliably produces the same execution. LLM applications break that assumption: the same prompt can yield different outputs across runs, multi-step agentic pipelines branch based on model decisions at runtime, and the 'bug' you're chasing might be a subtly wrong retrieval, a prompt that drifted after an edit, or a model update that silently changed behavior. LLM observability tools (LangSmith, Langfuse, Arize Phoenix, Helicone) exist to make these non-deterministic, multi-hop pipelines debuggable.
The core unit: a trace
A trace captures one full request through an LLM pipeline, broken into spans — individual steps like a retrieval call, an LLM generation, or a tool invocation — nested to reflect the actual call structure (an agent's trace might contain a top-level span, several nested LLM call spans, and tool-call spans). Each span typically records its inputs, outputs, latency, token usage, and cost, so you can walk through exactly what happened at each step of a failed or surprising response, rather than guessing from a flat application log.
import time, uuid
class Span:
def __init__(self, name, parent=None):
self.id = str(uuid.uuid4())[:8]
self.name = name
self.parent = parent
self.start = time.time()
self.metadata = {}
def end(self, output=None, **meta):
self.duration = time.time() - self.start
self.output = output
self.metadata.update(meta)
print(f"[span {self.id}] {self.name} ({self.duration:.2f}s) -> {output}")
# Example: tracing a retrieval + generation pipeline
root = Span("answer_question")
retrieval_span = Span("retrieve_docs", parent=root.id)
docs = ["doc1", "doc2"] # pretend retrieval happened
retrieval_span.end(output=f"{len(docs)} docs", tokens=0, cost_usd=0.0001)
gen_span = Span("generate_answer", parent=root.id)
answer = "This is the generated answer." # pretend LLM call happened
gen_span.end(output=answer, tokens=320, cost_usd=0.004)
root.end(output=answer)Evaluation and observability are two sides of the same data
What to capture per span
| Field | Why it matters |
|---|---|
| Input prompt (exact, post-templating) | Reproduce the exact call, catch template bugs |
| Output | Diagnose quality issues, feed evaluation datasets |
| Latency | Identify slow steps, especially in multi-hop agents |
| Token counts (prompt/completion) | Cost attribution, context-length debugging |
| Model/version identifier | Correlate quality shifts with model or prompt changes |
| User/session ID (anonymized if needed) | Reconstruct full conversation context, segment by cohort |
Online monitoring: catching drift in production
Beyond per-request debugging, observability platforms aggregate traces into monitoring dashboards: average latency and cost over time, token usage trends, rates of guardrail triggers or user-flagged responses, and — increasingly — automated scoring of sampled production outputs using LLM-as-judge evaluators. This is what catches silent regressions: a vendor model update that subtly changes tone, a prompt template edit that unintentionally breaks a formatting requirement for 5% of requests, or a retrieval index that's gone stale.
Logging full prompts and outputs raises privacy obligations
What's Next?
Observability tells you what happened and roughly why — the natural next question for any team running these systems at scale is how to control what all this logging, retrying, and multi-step reasoning actually costs to run.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.