What Building My Own Agent Harness Taught Me After Everyone Declared Orchestration Dead
When Langchain announced their agent framework was "production ready" and AutoGen claimed to have "solved multi-agent coordination," I did what felt like the contrarian thing: I started building my own agent harness from scratch. Not because I'm a glutton for punishment, but because something felt off about the victory lap everyone was taking.
The Dead Consensus
The narrative feels too clean. Agent orchestration is "figured out." Pick your framework, wire up some agents, and ship to production. The adoption charts look impressive—developers building with these tools, success stories flowing through Twitter, conference talks declaring the coordination problem solved.
But I kept noticing something in my own experiments with the popular frameworks. The demos worked beautifully. The tutorials ran smooth as silk. Then I'd try to build something slightly off the happy path, and I'd hit walls that felt… fundamental. Not bugs or missing features, but architectural assumptions that seemed to crumble under real complexity.
The more I dug into framework internals, the more I suspected the abstractions were doing exactly what abstractions do: hiding complexity rather than solving it. So I made what might have been a stupid decision—I decided to build coordination from the ground up, to see what these frameworks were actually abstracting away.
What My Harness Actually Does
I'm not building anything revolutionary. It's a simple coordination layer that manages communication between three specialized agents: one for research gathering, one for analysis, and one for synthesis. Custom message routing, basic state management, direct control over agent lifecycle and memory persistence.
I chose this architecture because I wanted to understand the handoff points. When Agent A finishes gathering information and needs to pass structured data to Agent B, what actually happens in that moment? When Agent C needs to reference something from Agent A's earlier work, how does that dependency get managed?
The harness lets me instrument every interaction, log every state transition, and intervene at every coordination point. It's probably overengineered for what it does, but that's the point—I wanted to see everything that the frameworks handle invisibly.
The First Surprise: Message Ordering Chaos
Within my first week of real testing, I discovered that agent responses don't arrive in any predictable order. This sounds obvious—of course async operations complete at different times—but the implications cascade in ways I hadn't anticipated.
Agent A sends context to Agent B, then immediately sends a follow-up clarification. Agent B receives the clarification first and starts working with incomplete information. By the time the original context arrives, Agent B has already committed to a response thread that makes no sense.
The timestamp drift between different LLM providers made this worse. My research agent uses GPT-4, my analysis agent uses Claude, and my synthesis agent uses a local model. Their internal clocks drift enough that "timestamp ordering" becomes meaningless across a conversation that spans twenty minutes.
I tried queuing systems, waiting periods, explicit acknowledgment protocols. Each solution created new failure modes. The queuing introduced latency that made agents "forget" earlier context. The acknowledgment protocols turned simple handoffs into multi-round negotiations that failed when agents misunderstood the confirmation prompts.
The Memory Problem Nobody Talks About
Shared context across agents is where my harness really started breaking down in interesting ways. I built what I thought was a clean global state system—agents could read and write to shared memory, with conflict resolution and versioning.
Within a few complex conversations, I started seeing phantom references. Agent C would cite specific quotes or data points that never appeared in any agent's actual output. When I traced the references back, I found they were mutations of real information that had been transformed through the shared memory handoffs.
Here's what I think was happening: Agent A would summarize information into shared memory. Agent B would read that summary, integrate it with new information, and update the memory with a synthesis. Agent C would read Agent B's synthesis as if it were Agent A's original research.
The problem isn't that this is wrong, exactly. It's that the agents developed confidence in information that had been through multiple interpretation layers without any audit trail of the transformations. They were citing "sources" that were actually conclusions from other agents, but treating them as primary data.
Conversation memory and task memory turned out to be completely different beasts. Agents could remember what they'd said to each other, but they couldn't reliably track what they'd collectively accomplished toward a larger goal.
Failure Patterns That Don't Show Up in Demos
The most frustrating failures were the ones that looked like successes until you examined them closely. I started seeing what I called "confident hallucination handoffs"—Agent A would make a subtle error or unfounded assumption, pass it to Agent B with high confidence, and Agent B would build elaborate analysis on top of the faulty foundation.
The error compounding was vicious because each agent added its own layer of reasoning, making the final output sound more sophisticated and authoritative than anything a single agent would produce. But tracing backwards, the whole chain rested on Agent A's initial mistake.
Agent personality drift during long conversations created another class of problems. My synthesis agent would gradually become more verbose and academic over the course of a session, even though nothing in its prompt changed. By hour three of a complex research task, it was producing completely different styles of output than it had at the beginning.
Retry logic nearly broke everything. When an agent failed to respond properly, my harness would retry the request with the same context. But in a multi-agent scenario, the context had often shifted by the time the retry executed. Agent B would retry a request to Agent A based on a conversation state that was now five interactions behind. The retry would succeed, but with completely irrelevant information.
What Control Actually Reveals
Building my own coordination layer revealed patterns I'd never seen in framework-based systems. Agents develop implicit timing dependencies that aren't visible when you're working through abstractions. My analysis agent learned to expect research data within a certain window and would start making assumptions if the research agent ran long.
Most of what I'd been thinking of as "agent failures" turned out to be orchestration architecture failures. An agent would produce a perfectly reasonable response to the context it received, but that context was corrupted by handoff timing, memory mutations, or dependency ordering that happened at the coordination layer.
When I could see the entire stack, I realized how much of the "intelligence" in multi-agent systems actually happens in the spaces between agents. The routing decisions, the context transformations, the memory management—these coordination functions were doing as much cognitive work as the agents themselves.
The Irony of Building Your Own
The frameworks everyone's using aren't solving the wrong problems—they're solving the problems that make demos work reliably. Clean handoffs, predictable state transitions, error modes that fail gracefully in controlled environments.
But production reliability in agent coordination seems to require grappling with exactly the complexity that frameworks abstract away. The timing dependencies, the memory corruption patterns, the compound error propagation—these aren't bugs in the frameworks. They're fundamental characteristics of agent coordination that get hidden when you optimize for ease of use.
Building my own harness didn't make me an orchestration expert—it made me realize how much we're still guessing about agent coordination. I'm starting to wonder if the interesting work in agent systems isn't in the agents themselves, but in these spaces between them that we've been too eager to abstract away.