Why I'm Building a Deterministic LLM Router Instead of Using Claude's Smart Routing

I was three weeks into debugging what should have been a straightforward user session consistency issue when I realized the problem wasn't in my code—it was in Claude's smart routing. Same user, same conversation context, same input parameters, but wildly different response patterns depending on which model Anthropic's black box decided to route me to. That's when I knew I had to build my own router.

The Problem I Didn't See Coming

When Claude introduced smart routing earlier this year, I was honestly excited. Automatic optimization, cost savings, and I wouldn't need to think about model selection—what's not to love? I was building a research assistant that needed to maintain consistent reasoning patterns across multi-turn conversations, and the promise of "optimal" model selection seemed perfect.

The first red flag came during user testing. A researcher would ask the same type of analytical question in different sessions and get responses with completely different structures and depths. One session would produce detailed step-by-step reasoning, another would give a brief summary-style answer. Both were arguably "correct," but the inconsistency was breaking the user experience I was trying to create.

I spent days debugging my prompt templates, convinced I had some subtle context-bleeding issue. I added more explicit instructions, tightened my system prompts, even rebuilt parts of my conversation state management. Nothing worked because I was debugging the wrong layer entirely.

The real nightmare was reproduction. I couldn't consistently replicate issues because I never knew which model was actually processing my requests. Claude's smart routing was making decisions based on criteria I couldn't see, much less control. I was essentially debugging two variables simultaneously: my application logic and Anthropic's routing algorithm.

When Smart Routing Becomes Unpredictable Routing

The breaking point came during a demo to potential users. I was showing how the assistant could help with systematic literature reviews—a task requiring consistent categorization and analysis patterns. Halfway through the demo, the responses suddenly shifted to a more conversational, less structured format. Same prompts, same context, but clearly a different model handling the requests.

I realized I'd lost control over a critical infrastructure layer. Smart routing optimizes for aggregate performance across all of Anthropic's users, but my product needed predictable behavior for individual user workflows. The tension became clear: vendor optimization versus product predictability.

This wasn't just about consistency either. I was trying to tune different aspects of my application—response formatting, reasoning depth, creative versus analytical tone—but couldn't isolate variables. Was a response improvement due to my prompt changes or because smart routing decided to use a different model? I was essentially flying blind.

Wayfinder Router's HN Success and My Realization

A few weeks later, I saw Wayfinder Router gaining traction on Hacker News. Reading through the comments, I realized I wasn't alone in this frustration. Developers were building their own routing layers specifically to regain control over model selection.

I dug into what Wayfinder built—a proxy layer that intelligently routes requests based on task classification and performance requirements. It was elegant, but analyzing their approach highlighted what I actually needed: not just smart routing with different criteria, but deterministic routing where I could map specific task types to specific models with explicit fallback rules.

The existing solutions still felt like black boxes, just with different optimization targets. I wanted something closer to infrastructure—predictable, debuggable, and under my direct control. That's when I decided to build my own router from scratch.

What I'm Building Instead

My deterministic router starts with explicit task classification. Instead of letting an algorithm decide, I map request types to models based on what I've learned about their strengths:

• Analytical tasks (data analysis, systematic reasoning): Claude 3.5 Sonnet
• Creative work (writing, ideation): GPT-4 or Claude 3 Opus depending on the specific creative domain
• Factual retrieval (straightforward Q&A): GPT-3.5 or Claude 3 Haiku for speed and cost
• Code generation: GPT-4 for complex logic, Claude 3.5 Sonnet for documentation

Cost and latency constraints are first-class parameters in my routing decisions. I can explicitly choose a "worse" model if the task doesn't justify the expense or if response time matters more than quality. Smart routing systems might make these tradeoffs, but I can't see or control them.

The router includes explicit fallback chains. If my primary choice is unavailable or rate-limited, I know exactly which model will handle the request instead. No surprises, no debugging sessions where I can't reproduce behavior because the fallback logic is opaque.

Early Results and Unexpected Discoveries

Six weeks in, the performance comparison is revealing. For my specific use case—research assistance with conversation continuity—my deterministic routing consistently outperforms Claude's smart routing on user satisfaction metrics, even when it's technically using "suboptimal" models for individual requests.

The most interesting discovery: there are cases where deliberately choosing a less capable model improves the user experience. For quick clarification questions, GPT-3.5's brevity and speed creates better conversation flow than waiting for a more sophisticated model to generate a detailed response the user doesn't actually want.

I'm also learning things about model strengths that smart routing might miss. Claude 3.5 Sonnet has a particular way of structuring analytical responses that works well for my users, but that preference might not show up in aggregate performance metrics across all use cases.

The development velocity improvement has been dramatic. I can now isolate variables properly when testing prompt changes or conversation flow modifications. When something breaks, I know exactly which model processed the request and can reproduce the issue reliably.

The Control vs. Optimization Spectrum

Smart routing makes sense for high-volume applications with diverse, unpredictable workloads. If you're building a general-purpose chatbot or processing thousands of varied requests daily, algorithmic optimization probably beats manual model selection.

But for product development, especially when you're building specialized workflows or need consistent user experiences, deterministic routing wins. The ability to debug, reproduce issues, and control exactly how your product behaves is worth the overhead of manual optimization.

There's a deeper infrastructure philosophy question here: who should control the intelligence layer of our applications? Smart routing assumes the vendor knows best how to optimize across all use cases. Deterministic routing assumes the application developer understands their specific requirements better than a general-purpose algorithm.

I keep thinking about this as AI becomes more central to product functionality. The more critical these models become to our applications, the more we might need infrastructure that prioritizes predictability and control over automated optimization.