The Model Routing Problem: Why I Built Switching Logic Into My Code Editor

When my Claude Code bill hit $340 last month, I finally started paying attention to what these AI coding tools were actually doing. I watched in real-time as my editor burned through expensive API calls to complete variable names, close brackets, and suggest imports. GPT-4 was being summoned to tell me that const user should probably be followed by = getUserData(). The waste was obvious, and I realized I had no idea what kinds of requests I was actually making.

So I built a logging system to find out.

Measuring the Waste

Over 30 days, I tracked every single request my coding tools made across three different projects — a React dashboard, a Python data pipeline, and some Rust CLI utilities. I categorized each request by complexity: basic autocomplete, syntax fixes, refactoring suggestions, debugging help, and architectural advice.

The results surprised me. Most of my requests were simple enough that a smaller model could handle them perfectly well. I was using Claude Sonnet to close parentheses and GPT-4 to suggest variable names. The pattern was everywhere:

• Autocomplete for standard library functions
• Basic syntax error corrections
• Simple refactoring like extracting variables
• Boilerplate generation for common patterns

The expensive models were only truly necessary for about a quarter of requests — complex debugging sessions, architectural decisions, or working with unfamiliar libraries where I needed genuine reasoning ability.

Building the Decision Tree

My first attempt at routing was embarrassingly naive. I tried to classify by file type: Python gets model A, JavaScript gets model B. It failed immediately. File type told me nothing about whether I was asking for a simple import statement or help debugging a complex async race condition.

After several failed approaches, I settled on routing based on three factors:

• Token count: Requests under 50 tokens almost always need basic completion
• AST complexity: Simple expressions route to smaller models, nested callbacks don't
• Dependency context: Requests involving external APIs or complex libraries get the big models

The logic isn't perfect. Complex regex patterns fooled my classifier for weeks — they look simple syntactically but often need sophisticated reasoning. Nested callbacks in JavaScript consistently broke my early versions. I ended up building a manual override system for the edge cases I couldn't predict.

The Models I Route Between

My current setup splits requests across four tiers:

• GPT-3.5: Handles most requests — autocomplete, simple fixes, basic refactoring
• Claude Sonnet: Takes mid-complexity refactoring, code explanations, moderate debugging
• GPT-4: Gets architecture decisions, complex debugging, working with new frameworks
• Local CodeLlama: Covers offline work and anything involving sensitive business logic

The distribution shifts depending on what I'm working on. During exploratory phases with new libraries, GPT-4 usage jumps significantly. When I'm in heads-down implementation mode, GPT-3.5 handles the vast majority of requests.

What I Learned About Task Classification

The biggest surprise was how much request phrasing affected optimal routing. "Fix this bug" and "Why isn't this working?" are functionally identical requests, but my classifier treated them differently for months. I was overly confident that automated classification would work reliably.

I also discovered patterns I didn't expect. Late-night coding sessions consistently skewed toward complex models, even for simple tasks. When I'm tired, I apparently phrase requests in ways that trigger the heavyweight routing, even when I just need basic completion. The manual override system I built gets used several times per week, mostly for cases where I know the request is simpler than it appears.

The Real Insight: Context Switching Costs

Six months in, I've realized that seamless model switching matters more than raw model performance. When I know which model is responding, it changes how I interpret the suggestions. GPT-3.5 completions feel less trustworthy even when they're correct. Claude responses carry more weight even for simple tasks.

The mental overhead of manual model selection was crushing my flow state. I'd spend 10 seconds deciding whether a debugging session warranted GPT-4, then lose my train of thought entirely. Automated routing eliminated that friction, but introduced a new problem: I started second-guessing the router's decisions.

The sweet spot seems to be invisible routing with visible confidence indicators. I show which model responded, but only after the fact. It preserves flow while building appropriate trust in the suggestions.

Current State and Open Questions

After six months of iteration, my routing system correctly selects the appropriate model most of the time. The remaining cases are edge cases I still can't predict — usually involving domain-specific knowledge or requests that seem simple but require sophisticated reasoning.

My API costs dropped significantly while coding productivity stayed roughly the same. The bigger win was eliminating the decision fatigue around model selection. I'm back to thinking about code instead of thinking about which AI to ask.

But I'm left wondering if this whole exercise is optimizing for the wrong thing. Maybe the problem isn't model selection at all, but our assumption that one-size-fits-all tools should exist in the first place. The more I use this system, the more I think the future of AI assistance might be more fragmented and specialized than I initially expected.