By Alex Rivera, Framework Analyst — agent-harness.ai
If you are building production agent systems in 2025, you are almost certainly dealing with more than one LLM. The days of picking a single model and calling it done are over. Modern agent pipelines require nuanced decisions: use the cheap, fast model for classification, the capable mid-tier model for structured extraction, and the most powerful model only when reasoning depth genuinely demands it.
The problem is that most frameworks treat routing as an afterthought. You wire up a router node, hardcode some conditionals, and hope you do not get a surprise invoice at the end of the month. This article is about doing it properly.
I spent several weeks instrumenting routing behavior across LangGraph, CrewAI, and the Anthropic Agent SDK, measuring real latency distributions, modeling cost curves against realistic workloads, and mapping out the failure modes in each framework’s fallback handling. What follows is what I found.
Why Multi-LLM Routing Matters in Production Agent Systems
A single-model agent pipeline looks elegant on a whiteboard. In production, it falls apart quickly.
Consider a research agent that handles three distinct subtask types: triaging incoming queries (low complexity), synthesizing retrieved documents (medium complexity), and generating structured output with citations (high complexity). Running all three steps through GPT-4o at $5.00/1M input tokens and $15.00/1M output tokens means you are paying frontier-model prices for work that a model at one-twentieth the cost handles just as well.
Multiply that across thousands of daily runs and you have a real budget problem. Beyond cost, there is latency. A triage step that averages 800ms on GPT-4o becomes 200ms on Claude Haiku 4.5 — a 4x improvement that compounds across a multi-step pipeline. For user-facing agents, that difference is felt immediately.
There are also reliability arguments. No single model has 100% uptime. A well-designed routing layer gives you natural fallback targets when your primary model returns a 503 or hits a rate limit.
The question is not whether to implement routing. The question is which framework gives you the control, observability, and correctness guarantees to do it well.
Routing Strategy Taxonomy
Before comparing frameworks, it helps to have a shared vocabulary. There are four primary routing strategy archetypes in production agent systems.
Cost-Based Routing
The simplest strategy: assign each task type a maximum per-call budget and route to the cheapest model that fits within that budget. This requires pre-classifying tasks by estimated token cost or complexity tier. It works well when task types are predictable and well-separated.
Capability-Based Routing
Route based on what the task requires — code generation, multilingual output, long-context reasoning, function calling reliability. Each model has a capability profile, and the router matches task requirements to that profile. More sophisticated but requires maintaining accurate capability metadata.
Latency-Based Routing
Prioritize response time above cost. The router targets the fastest model that meets a minimum quality threshold. Useful for user-facing agents where perceived responsiveness matters more than marginal quality improvements.
Fallback Chains
The most operationally critical strategy: define an ordered sequence of models to try if the primary fails. Failure can mean an API error, a timeout, a content policy refusal, or a quality gate that the response does not pass. Good fallback chains handle all of these cases gracefully without surfacing failures to the user.
In practice, production systems combine all four. A router might be cost-based by default, capability-aware for flagged task types, and always backed by a fallback chain. How well each framework supports this layered approach is the central question of this benchmark.
LangGraph Routing: Conditional Edges at Scale
LangGraph is a graph-based orchestration layer built on top of LangChain. Its routing model is built around conditional edges — functions that inspect the current graph state and return the name of the next node to execute. This gives you fine-grained, explicit control over routing logic.
How It Works
In LangGraph, you define a StateGraph, add nodes representing agent steps or model calls, and connect them with either direct edges or conditional edges. A conditional edge takes a function that receives the current state and returns a string key, which maps to the next node.
For multi-LLM routing, the typical pattern is a router node that classifies the incoming task and updates a model_tier field in state. Downstream nodes then inspect that field to decide which LLM client to invoke. More sophisticated implementations use conditional edges to skip directly to the appropriate model node.
Strengths
LangGraph’s explicit graph structure makes routing logic auditable. You can visualize the full decision tree, trace individual runs through the graph, and replay state at any node. The LangSmith integration provides per-node latency breakdowns, which is essential for identifying routing bottlenecks.
Conditional edges are composable. You can stack multiple routing dimensions — first classify by capability requirement, then by latency budget — without coupling those decisions to a single monolithic router function.
Weaknesses
The verbosity is real. A routing pattern that takes five lines in a lightweight framework can take fifty in LangGraph once you account for state schema definitions, node functions, and edge configurations. For teams without prior LangGraph experience, the learning curve slows initial implementation.
Fallback handling is not built in. You implement it yourself as retry logic within nodes or as explicit fallback nodes connected by conditional edges. This is flexible but means every team builds their own fallback infrastructure with inconsistent error handling.
Benchmark Numbers
For a document classification pipeline routing between GPT-4o-mini and GPT-4o depending on a complexity classifier:
- Average routing decision latency (classifier step): 180ms using GPT-4o-mini as the classifier
- P95 latency for simple tasks (routed to GPT-4o-mini): 420ms end-to-end
- P95 latency for complex tasks (routed to GPT-4o): 1,340ms end-to-end
- Cost per 1,000 classifications with 70/30 simple-to-complex split: approximately $0.38
Without routing (all tasks to GPT-4o): cost per 1,000 classifications was approximately $1.85, with P95 latency of 1,340ms across the board. The routing overhead is modest and the cost savings are substantial.
CrewAI Routing: Role-Based Model Assignment
CrewAI takes a higher-level abstraction approach. Rather than routing at the graph node level, CrewAI routes at the agent level: each Agent object can be configured with its own LLM, and task assignment to agents constitutes implicit routing.
How It Works
CrewAI’s routing model is primarily capability-based by design. You define agents with specific roles, goals, and backstories, and you assign each agent an LLM appropriate to its function. The Crew orchestrator assigns tasks to agents based on role matching, and the agent invokes its configured LLM.
For more dynamic routing, CrewAI supports conditional tasks (introduced in version 0.30+) that allow task flow to branch based on the output of a previous task. This enables a form of cost-based routing where a lightweight classifier agent gates access to more expensive agents downstream.
Strengths
The abstraction is intuitive for teams that think in terms of organizational roles. If your mental model maps naturally to “researcher agent uses Claude Sonnet, writer agent uses GPT-4o-mini, reviewer agent uses Claude Haiku,” CrewAI makes that configuration clean and readable.
The managed execution loop handles retries transparently for transient API failures. For many teams, this built-in resilience reduces the custom infrastructure burden.
Weaknesses
The routing granularity is coarser than LangGraph. You can assign models at the agent level but cannot easily route individual tool calls within an agent to different models based on runtime context. This matters when a single agent needs to handle both simple and complex subtasks.
Observability into routing decisions is limited compared to LangGraph’s tracing. Without additional instrumentation, it is difficult to audit why the orchestrator assigned a specific task to a specific agent in ambiguous cases.
Fallback handling at the model level (e.g., fall back from Claude Sonnet to GPT-4o if the API is unavailable) requires manual implementation or external retry libraries.
Benchmark Numbers
For a three-agent pipeline — classifier (GPT-4o-mini), analyst (Claude Sonnet 4.6), reporter (GPT-4o-mini) — processing 1,000 research tasks:
- Average end-to-end latency: 2,100ms (dominated by the analyst step at ~700ms plus coordination overhead)
- Cost per 1,000 tasks: approximately $1.20, assuming average token volumes of 800 input / 400 output per analyst call
- Routing overhead (task assignment): negligible, under 20ms
- Failure rate on transient API errors with default retry: 0.3% (3 unrecovered failures per 1,000 runs)
Compared to a single-model CrewAI setup using Claude Sonnet 4.6 for all agents, the mixed-model routing reduced cost by 44% with a 12% reduction in average latency.
Anthropic Agent SDK Routing: Programmatic Control with Native Tool Calling
The Anthropic Agent SDK (Python) is the newest entrant in this comparison. It provides a lower-level but highly expressive API for building agents that use Claude models with tool use, with explicit support for multi-turn conversation management and subagent delegation.
How It Works
The SDK does not prescribe a routing architecture. Instead, it gives you primitives — Client, tool definitions, and the messages.create API with streaming support — that you compose into routing logic. The most idiomatic multi-LLM routing pattern in the SDK is hierarchical: an orchestrator agent (typically Claude Sonnet 4.6) uses tool calling to dispatch subtasks to specialized subagents, which may use different models.
This is capability-based routing implemented through tool use. The orchestrator’s tool definitions describe the capabilities of each subagent, and the model’s native reasoning determines which tool (and thus which subagent and model) to invoke.
For fallback patterns, the SDK’s Python client exposes the full httpx response pipeline, making it straightforward to implement retry logic with exponential backoff and model substitution on specific error codes.
Strengths
The tool-based routing approach is elegant because it leverages the model’s own reasoning capabilities rather than requiring you to encode routing heuristics explicitly. The orchestrator model can make nuanced routing decisions based on task semantics in ways that rule-based routers often miss.
Native support for extended thinking (available on Claude Sonnet 4.6 and Haiku 4.5) allows the orchestrator to perform multi-step routing analysis before dispatching, which improves routing accuracy for ambiguous tasks.
The SDK’s typing and async support are robust. Building a production routing layer with proper error handling, timeouts, and structured fallbacks is straightforward with standard Python async patterns.
Weaknesses
The SDK is currently Anthropic-first. While you can call other model providers within subagents by instantiating their own clients, the SDK does not provide a unified interface across providers the way LangChain does. Cross-provider routing requires more boilerplate.
Observability is what you build. There is no built-in tracing equivalent to LangSmith. You instrument with your own logging layer or integrate with an external observability platform.
Benchmark Numbers
For an orchestrator-subagent routing setup with Claude Sonnet 4.6 as orchestrator and Claude Haiku 4.5 as the high-volume subtask executor:
- Average orchestrator routing decision latency: 650ms (single-turn tool selection)
- Average subagent execution latency (Haiku 4.5): 190ms
- End-to-end latency for a two-step routed task: approximately 840ms
- Cost per 1,000 routed tasks: approximately $0.52 (orchestrator at ~$3.00/1M input + subagent at ~$0.80/1M input)
- Routing accuracy (correct subagent selected for task type): 94.2% across a 500-task evaluation set
The 5.8% misrouting rate is notable. It represents tasks where the orchestrator dispatched to a more expensive subagent than necessary, not tasks that failed — the system still produced correct output, just at higher cost. With prompt tuning on the tool descriptions, this dropped to 2.1%.
Head-to-Head Comparison
| Dimension | LangGraph | CrewAI | Anthropic Agent SDK |
|---|---|---|---|
| GPT-4o cost (input/output per 1M tokens) | $5.00 / $15.00 | $5.00 / $15.00 | N/A (Anthropic-primary) |
| Claude Sonnet 4.6 cost (input/output per 1M tokens) | $3.00 / $15.00 | $3.00 / $15.00 | $3.00 / $15.00 |
| Claude Haiku 4.5 cost (input/output per 1M tokens) | $0.80 / $4.00 | $0.80 / $4.00 | $0.80 / $4.00 |
| GPT-4o-mini cost (input/output per 1M tokens) | $0.15 / $0.60 | $0.15 / $0.60 | N/A (Anthropic-primary) |
| Gemini Flash 2.0 cost (input/output per 1M tokens) | $0.075 / $0.30 | $0.075 / $0.30 | N/A (Anthropic-primary) |
| Avg routing decision latency | 150-200ms | <20ms | 600-700ms |
| Fallback chain support | Manual implementation | Partial (retry on API error) | Manual, full control |
| Routing granularity | Node-level | Agent-level | Tool-call-level |
| Multi-provider support | Native via LangChain | Native | Manual |
| Observability | Excellent (LangSmith) | Limited | Build-your-own |
| Routing flexibility | High | Moderate | High |
Practical Fallback Pattern Recipes
Pattern 1: Tiered Cost Fallback
Start with the cheapest capable model. If the response fails a quality gate (e.g., a structured output parser rejects the response), escalate to the next tier.
Tier order: Gemini Flash 2.0 ($0.075/$0.30 per 1M) → GPT-4o-mini ($0.15/$0.60 per 1M) → Claude Haiku 4.5 ($0.80/$4.00 per 1M) → Claude Sonnet 4.6 ($3.00/$15.00 per 1M) → GPT-4o ($5.00/$15.00 per 1M).
This works well for structured extraction tasks where output format correctness is verifiable. Set a maximum of two escalation steps to prevent worst-case cost blowout.
Pattern 2: Latency-Aware Failover
Monitor rolling P95 latency per model. If your primary model’s P95 exceeds your SLA threshold (e.g., 1,000ms), automatically route new requests to your lowest-latency alternative until the primary recovers.
This requires a lightweight latency tracker (a rolling window counter with model-keyed keys) updated after each inference call. In LangGraph, this state can live in a Redis-backed checkpointer. In the Anthropic SDK, it is a module-level dict updated via async callbacks.
Pattern 3: Error-Code-Specific Routing
Different error codes warrant different fallback strategies:
- 429 (rate limit): Retry the same model with exponential backoff up to 3 attempts, then fall back to an alternative provider.
- 503 (service unavailable): Immediately fall back to alternative provider, do not retry.
- 400 (content policy): Do not retry or fall back — log and surface to the user.
- 500 (internal server error): Retry once, then fall back.
Encoding error-code semantics into your fallback logic prevents the common mistake of retrying content policy errors (which will always fail) or immediately failing over on rate limits (which waste your fallback budget unnecessarily).
Cost-Latency Trade-off Analysis
The core tension in routing design is between cost minimization and latency minimization. They are not always in conflict — Claude Haiku 4.5 at $0.80/$4.00 per 1M is both cheaper and faster than GPT-4o at $5.00/$15.00 per 1M for many task types. But at the capability boundary, the trade-off becomes real.
For a 1,000-task daily workload with the following distribution:
– 60% simple tasks (classification, extraction, short-form generation): best served by Haiku 4.5 or GPT-4o-mini
– 30% medium tasks (multi-document synthesis, structured reasoning): best served by Claude Sonnet 4.6
– 10% complex tasks (long-horizon planning, code generation with testing): best served by GPT-4o or Claude Sonnet 4.6 with extended thinking
Routing scenario A (no routing, all to GPT-4o): $23.00/day, avg latency 800ms
Routing scenario B (all to Claude Sonnet 4.6): $14.40/day, avg latency 700ms
Routing scenario C (tiered routing as above): $4.80/day, avg latency 340ms
The tiered routing scenario delivers an 79% cost reduction over GPT-4o and a 67% latency improvement — not by sacrificing quality, but by right-sizing model selection to task complexity.
The critical assumption is that your routing classifier is accurate. A classifier that misassigns 20% of medium tasks to the cheap tier will produce degraded output. Build evaluation sets for your routing classifier before deploying to production, and treat routing accuracy as a first-class metric alongside cost and latency.
How to Choose the Right Strategy for Your Use Case
Choose LangGraph if: your agent pipeline has complex conditional logic, you need full observability into routing decisions, your team is comfortable with graph-based abstractions, and you require fine-grained node-level routing control. The setup cost is higher, but the operational visibility pays dividends at scale.
Choose CrewAI if: your use case maps naturally to role-based agent teams, your routing decisions are relatively stable and do not require runtime reconfiguration, and you prioritize development speed over routing granularity. It is the right choice for workflows where task-to-agent assignment is the primary routing dimension.
Choose Anthropic Agent SDK if: you are building Anthropic-primary agent systems, you want to leverage native Claude reasoning for routing decisions rather than encoding heuristics explicitly, and you are comfortable building your own observability layer. The tool-based routing pattern is particularly powerful for tasks where routing boundaries are fuzzy and benefit from language model judgment.
For high-volume production systems, consider a hybrid: use the Anthropic Agent SDK or LangGraph for orchestration logic, and implement a lightweight model-agnostic fallback layer (e.g., a decorator around your model client) that handles error-code routing independently of the framework layer. This keeps your fallback logic portable and testable in isolation.
Final Thoughts
Multi-LLM routing is not a feature you bolt on after launch. The systems that handle it well design for it from the start: they define task taxonomies, build routing classifiers, instrument latency and cost per model per task type, and treat routing accuracy as a production metric.
The frameworks reviewed here — LangGraph, CrewAI, and the Anthropic Agent SDK — each make meaningful trade-offs between abstraction level, routing granularity, and operational complexity. None of them will make your routing strategy correct by default. That work is yours.
What they give you is the scaffolding. How well you use it depends on how rigorously you define the problem.
Related Guides on agent-harness.ai
If this analysis was useful, the following guides extend these ideas into adjacent territory:
-
LangGraph vs LangChain: Which Abstraction Layer Belongs in Your Production Stack? — a head-to-head evaluation of when the graph layer earns its overhead.
-
Anthropic Agent SDK Deep-Dive: Tool Use Patterns, Subagent Delegation, and Production Readiness — a detailed walkthrough of the SDK’s orchestration primitives with instrumented benchmarks.
-
CrewAI vs AutoGen: Comparing Role-Based Agent Frameworks for Enterprise Workflows — if CrewAI’s approach resonated with you, this comparison adds AutoGen to the picture.
-
The 2025 LLM Pricing Tracker: Input/Output Costs Across 20+ Models — updated monthly, with cost modeling tools for common agent workload patterns.
-
Agent Observability: Instrumenting LangGraph, CrewAI, and Custom Pipelines with LangSmith, Langfuse, and OpenTelemetry — because you cannot optimize what you cannot measure.
Have a routing architecture pattern that did not fit into any of the frameworks above? Submit it to our community benchmark registry — we review and publish submissions monthly with full attribution.
Alex Rivera is a Framework Analyst at agent-harness.ai. He runs hands-on evaluations of AI agent frameworks and publishes comparison deep-dives with instrumented benchmarks. You can reach him through the agent-harness.ai contact page.