You deployed your multi-agent pipeline last Thursday. By Friday morning, latency had spiked 340%, token costs doubled, and three tool calls silently returned empty results that your agents happily passed downstream as valid data. You had no idea until a user filed a bug report.
This is what operating blind looks like. And it’s the default state for most LangGraph and CrewAI deployments.
Observability isn’t a nice-to-have for production agents — it’s the difference between understanding your system and guessing at it. In this comparison, I’m breaking down the full observability stack: which tracing tools integrate cleanly with each framework, how to structure LLM call logging for meaningful signal, and which dashboards give you real-time visibility instead of post-mortem archaeology.
Why Agent Observability Is Harder Than Traditional APM
Before we compare tools, it’s worth naming the specific challenges that make agent observability different from instrumenting a REST API or a traditional ML model.
Non-determinism is baked in. A trace through a LangGraph workflow or CrewAI crew can branch at any node, retry on tool failure, spawn sub-agents, or loop until a stop condition. Standard APM tools built around request/response cycles fall apart here — you need tools that model DAG execution, not linear call stacks.
LLM calls are expensive and slow. A single agent run might make 8–40 LLM calls. Each one has latency measured in seconds, variable token counts, and a cost you need to track per-call and per-run. Aggregating these accurately is non-trivial.
Tool call failures are silent. When an HTTP tool returns a 200 with an error body, or a code execution tool returns an empty string, the agent often continues without raising an exception. Without structured logging at the tool call level, these silent failures are invisible.
Context windows are state. The messages array passed into each LLM call is your agent’s working state. Logging that faithfully — without storing 30,000 tokens per trace at $0.00 — requires thoughtful sampling and compression strategies.
The Observability Stack: What You Actually Need
A complete agent observability stack has four layers:
- Trace collection — capturing spans for every agent step, LLM call, and tool invocation
- LLM call logging — structured records of model, prompt, completion, token counts, latency, and cost
- Storage and search — a backend that lets you query traces by run ID, error type, agent name, or latency range
- Dashboards — real-time and historical views of throughput, error rates, cost, and latency by agent
Let’s evaluate the main tools against each layer for LangGraph and CrewAI specifically.
Tracing Tool Comparison
LangSmith
LangSmith is Langchain’s first-party observability platform, and it has the tightest integration with LangGraph of any tool on the market. If you’re running LangGraph, it deserves to be your starting point.
Setup overhead: Near zero. Set LANGCHAIN_TRACING_V2=true and LANGCHAIN_API_KEY, and every LangGraph node execution, LLM call, and tool invocation is automatically traced. No manual instrumentation.
Trace fidelity for LangGraph: Excellent. LangSmith understands LangGraph’s state graph natively — it renders node transitions, tracks state deltas between nodes, and correctly attributes LLM calls to their parent node. When a conditional edge fires, you see exactly which branch was taken and why.
CrewAI support: Functional but requires more setup. CrewAI does expose a LangSmith callback handler, but crew-level metadata (agent roles, task assignments, crew kickoff context) isn’t automatically captured. You’ll need to add manual callbacks to get meaningful crew-level traces.
Latency overhead: In benchmarks across 500 LangGraph runs, LangSmith’s tracing added a median 18ms overhead per run — acceptable for most workloads. The async logging path is non-blocking, so this doesn’t add to user-facing latency in production.
Weak points: The free tier is limited to 5,000 traces/month, which evaporates fast in development. The UI can feel slow when browsing trace lists with large payloads. And LangSmith is SaaS-only — there’s no self-hosted option, which is a hard blocker for teams with strict data residency requirements.
Langfuse
Langfuse is the strongest open-source alternative and has become my default recommendation for teams that need self-hosting or multi-framework support.
Setup overhead: Moderate. Langfuse provides SDKs for Python and TypeScript, plus callback integrations for LangChain/LangGraph. For CrewAI, you use the langfuse decorator pattern or the OpenTelemetry exporter.
Trace fidelity for LangGraph: Good, with caveats. Langfuse captures LLM calls and tool invocations accurately, but its trace model is hierarchical (trace → span → generation) rather than graph-aware. LangGraph’s parallel node execution doesn’t render as clearly as in LangSmith — concurrent nodes appear as sibling spans rather than a visual DAG. For debugging sequential workflows, this is fine. For debugging complex parallel graphs, it’s a limitation.
CrewAI support: Better than LangSmith. Langfuse’s observe decorator integrates cleanly with CrewAI’s @agent and @task patterns. When CrewAI 0.70+ added native OpenTelemetry support, Langfuse became a natural fit — you get crew-level traces with agent and task metadata attached.
Self-hosting: This is Langfuse’s killer feature. The Docker Compose deployment is production-ready, runs on a single VM for development teams, and scales to Kubernetes for high-volume production. For teams handling sensitive data (healthcare, finance, legal), self-hosting isn’t optional.
Cost model: Self-hosted Langfuse is free. The cloud plan starts at $49/month for 100k observations. For teams running >50k LLM calls/day, self-hosting provides dramatic cost savings over both Langfuse Cloud and LangSmith.
OpenTelemetry + OpenLLMetry
For teams that already operate an OpenTelemetry infrastructure, OpenLLMetry (from Traceloop) provides semantic conventions for LLM calls that map onto standard OTEL spans.
Setup: Add the traceloop-sdk package, call Traceloop.init(), and your LLM calls are instrumented with vendor-specific attributes (llm.model, llm.usage.total_tokens, llm.prompts, etc.).
Framework integration: OpenLLMetry patches LangChain’s LLM classes and tool executors automatically. For CrewAI, the patching works at the LiteLLM/OpenAI SDK layer, which captures all model calls regardless of which framework initiated them.
Dashboard options: Any OTEL-compatible backend: Jaeger for trace search, Grafana for dashboards, Prometheus for metrics. This is maximum flexibility but also maximum setup cost — expect to spend a few days wiring together a production-grade stack from scratch.
Best for: Platform teams that already operate OTEL infrastructure and want to fold agent observability into their existing stack rather than adding another SaaS tool. Poor fit for small teams that want to get running quickly.
Arize Phoenix
Phoenix is an open-source observability tool purpose-built for LLM applications, with a strong focus on evaluation and dataset management alongside tracing.
Differentiator: Phoenix’s trace UI is genuinely the best-in-class for understanding multi-turn agent conversations. The span waterfall is easy to read, prompt/completion payloads are displayed inline with token counts, and you can export any trace directly to a dataset for evaluation.
LangGraph integration: Phoenix provides a LangChain callback handler that works with LangGraph. Trace fidelity is good — comparable to Langfuse for sequential graphs. The evaluation workflow (annotating traces, running LLM-as-judge evaluations) is significantly better than any other tool in this list.
CrewAI integration: Decent. Phoenix’s OpenTelemetry exporter captures all LLM calls from CrewAI via the LiteLLM patch. Task and agent metadata requires manual span enrichment.
When to choose Phoenix: When evaluation is a first-class concern. If you’re running systematic quality evaluations, A/B testing prompts, or building labeled datasets from production traces, Phoenix’s annotation and evaluation UX is worth the tradeoff on other dimensions.
LLM Call Logging: What to Capture and How
Tracing tools handle the instrumentation, but the structure of your LLM call logs determines whether you can answer questions like “which agent is driving 60% of our token costs?” or “what prompts correlate with tool call failures?”
The Minimum Viable Log Record
Every LLM call log should include:
{
"trace_id": "...", # links back to parent agent run
"span_id": "...", # unique to this LLM call
"agent_name": "...", # which agent made the call
"task_name": "...", # what task was being executed
"model": "gpt-4o", # exact model identifier
"input_tokens": 1842,
"output_tokens": 312,
"latency_ms": 2140,
"cost_usd": 0.0024, # calculated at log time, not query time
"tool_calls": [...], # structured tool call requests if any
"finish_reason": "stop", # or "tool_calls", "length", "content_filter"
"timestamp": "2026-11-09T14:22:31Z"
}
Calculate cost at log time. Don’t reconstruct costs from token counts at query time — model pricing changes, and historical cost data becomes wrong. Store the dollar amount at the moment of the call.
Log finish_reason religiously. finish_reason: "length" means the model hit your max_tokens limit and truncated its response — a silent failure mode that causes all kinds of downstream problems. Without logging this field, you’ll never know it’s happening.
Sampling Strategy for High-Volume Agents
At 50k+ LLM calls/day, storing every full prompt and completion becomes expensive and slow to query. A practical sampling strategy:
- Always log the structured metadata (tokens, cost, latency, finish_reason) — this is cheap
- Sample 100% of calls where
finish_reason != "stop"— these are always worth reviewing - Sample 100% of calls that result in tool call failures or agent errors
- Sample 10–20% of normal calls for full prompt/completion payloads
- Sample 100% of calls that exceed a latency or cost threshold (e.g., >5s or >$0.01)
Both LangSmith and Langfuse support sampling configuration. For the OTEL stack, implement sampling at the SDK level with a custom SpanProcessor.
Real-Time Performance Dashboards
Traces are for debugging individual runs. Dashboards are for understanding system behavior over time. Here’s what matters most in each tool’s dashboard offering.
LangSmith Dashboard
LangSmith’s built-in dashboard covers the basics well: run counts, error rates, median latency, and token usage over time. The filtering by tag, model, and agent name works smoothly.
Gap: LangSmith doesn’t offer configurable alerting. You can’t set a Slack notification when error rate exceeds 5% or when p95 latency crosses a threshold. For production monitoring, you’ll need to export metrics to an external tool.
Langfuse Dashboard
Langfuse’s dashboard is more flexible — you can build custom views by filtering on any trace attribute. The cost tracking charts are particularly useful: you can break down costs by model, by user, by session, or by any custom attribute you tag at trace time.
Self-hosted + Grafana: The best Langfuse dashboard setup I’ve run is Langfuse self-hosted (for trace storage and search) feeding metrics to Prometheus via its metrics export endpoint, visualized in Grafana. You get LangSmith-style trace search plus full Grafana alerting capabilities. Setup time: about half a day.
Grafana + OpenTelemetry
For teams running the OTEL stack, Grafana with the LLM-specific dashboards from the OpenLLMetry community is the most powerful option. Pre-built panels for token usage by model, latency percentiles, error rate by tool, and cost per agent. Alerting through Grafana Alertmanager connects to any notification channel.
The investment to set this up correctly is real — figure 2–3 days for a solid production dashboard — but the result is a monitoring stack that fits seamlessly into existing ops workflows.
Framework-Specific Recommendations
For LangGraph Teams
Start with LangSmith. The zero-config integration and graph-native trace visualization make it the fastest path to meaningful observability. Use it during development and staging.
Evaluate Langfuse for production if you have data residency requirements, run >50k traces/month, or need Grafana-based alerting. The LangGraph callback integration is reliable, and the cost savings at scale are significant.
Add OpenLLMetry if you’re already running OTEL infrastructure — the marginal cost of adding LLM semantic conventions to an existing stack is low.
For CrewAI Teams
Start with Langfuse. CrewAI’s native OpenTelemetry support in recent versions makes Langfuse the cleanest integration path. The observe decorator pattern maps naturally to CrewAI’s agent/task structure.
Use Phoenix if evaluation is central to your workflow. The ability to annotate production traces and build labeled datasets from real crew runs is uniquely powerful for teams iterating on agent quality.
Avoid LangSmith as your primary tool unless you’re running a hybrid LangGraph/CrewAI stack — the CrewAI integration requires too much manual work to be worth it when better options exist.
Benchmark Summary
| Tool | LangGraph Integration | CrewAI Integration | Self-Hosted | Free Tier | Setup Time |
|---|---|---|---|---|---|
| LangSmith | Excellent | Fair | No | 5k traces/mo | <30 min |
| Langfuse | Good | Good | Yes | 50k obs/mo | 1–2 hrs |
| OpenLLMetry + Grafana | Good | Good | Yes | Unlimited | 2–3 days |
| Arize Phoenix | Good | Fair | Yes | Unlimited | 2–4 hrs |
Getting Started: A Practical First Stack
If you’re instrumenting your first production agent and don’t want to over-engineer it, here’s the stack I’d deploy today:
- Langfuse self-hosted (Docker Compose, single VM) for trace storage
- LangChain callback handler (for LangGraph) or
observedecorator (for CrewAI) for instrumentation - Structured LLM call logs with the minimum viable fields above
- Langfuse built-in dashboards for cost and latency monitoring during initial deployment
- Upgrade to Grafana alerting once you have enough baseline data to set meaningful thresholds
This stack costs ~$20–40/month (VM cost), gives you full data control, and takes a few hours to stand up. It’s proportionate to where most teams are in their agent maturity curve.
The Bottom Line
The observability tools are good and getting better. LangSmith wins on LangGraph integration depth and ease of setup. Langfuse wins on flexibility, self-hosting, and total cost of ownership at scale. The OTEL stack wins when you need to fit agent observability into existing platform engineering infrastructure.
What none of them solve for you is deciding what to instrument. Capturing every span is easy; building a logging strategy that surfaces actionable signal — cost outliers, silent tool failures, latency regressions — requires deliberate design. Start with the minimum viable log record, instrument the error cases exhaustively, and sample normal calls intelligently.
The agents that run reliably in production aren’t the ones built on the best frameworks — they’re the ones whose operators can actually see what’s happening inside them.
Want to go deeper? Check out our LangGraph vs CrewAI architecture comparison and our guide to LLM cost optimization in multi-agent systems. If you’re evaluating observability tools for a specific use case, reach out for a framework consultation — we benchmark these stacks regularly and can point you to current data.