Everyone Has the Same Model

A few months ago I ran the same task twice and got two completely different outcomes from the same model.

The task was boring on purpose: take a 400-page medical demand packet, pull out the treatment timeline, flag anything that looks pre-existing rather than accident-related. The first time, I opened a fresh Claude window and described what I wanted. It did a reasonable job. The second time, I handed the exact same request to the system I'd built on top of Claude Code. It pulled in my constraints, picked the right tools, ran the extraction in passes, checked its own work against criteria I'd written weeks earlier, and handed me something I could actually use.

Same model. Same underlying weights. The only thing that changed was everything around it.

That gap is the thing I've been obsessed with this year. Not which model is smartest. I don't mean the models are interchangeable in some flat way, because they obviously aren't, and the spread between the best and the rest is enormous. I mean that for any given task I'm doing, the specific model I reach for is increasingly swappable for the next one, and the difference I can actually act on lives in the layer above it. That layer is the harness.

The engine and the car

There's a line from something I read while I was going down this rabbit hole that stuck with me: a model is the engine, the harness is the car, and most benchmarks test the car.

The harness is the software wrapped around the model that turns next-token prediction into something that reliably does a job. It builds the prompt the model actually sees. It decides which tools exist and how they're described. It manages the context window so the model isn't drowning in irrelevant tokens. It handles what happens when the model fails or loops. It parses the output, stores and retrieves memory across turns, and decides when the task is actually done.

None of that is the model. All of it determines whether the model is useful.

And the effect sizes here are not subtle. Holding the model fixed and changing only the harness, the numbers people are publishing are genuinely strange. Vercel stripped eighty percent of the tools out of their agent and watched success rates climb from 80% to 100%, with tokens dropping by more than half and latency falling from 724 seconds to 141. Princeton's CORE-Bench measured the same model scoring 42% under one scaffold and 78% under another. That's a thirty-six-point swing from nothing but the wrapper.

You have to be a little skeptical here, because nobody publishes the post where they added scaffolding and it got worse. The case studies are selected. But the direction is consistent across enough teams that the conclusion holds: when the harness can move your results by thirty points without touching the model, the harness is not a detail. It's the product. The model is the part underneath that everyone already has.

This isn't only something I notice in my own work, the benchmarks point the same way. As of March 2026, Stanford's AI Index reports the top US model leads the top Chinese model by just 2.7% on the Chatbot Arena leaderboard, down from a gap of 17.5 to 31.6 points in May 2023, which Stanford frames as the US-China performance gap having "effectively closed." It's one benchmark, it's a single Elo number that bounces around, and the two countries keep trading the lead, so I wouldn't hang the whole argument on it. But the direction is the point: the national frontiers are clustering at the top while the harness still swings the same model thirty-six points underneath them. The thing converging is the engine everyone already has; the thing still moving the result is the car.

What's actually in the harness

Claude Code, the thing I build on, is itself a harness. Underneath the editor and the terminal it's a loop: send the model a prompt, let it call some tools, feed the results back, repeat until the work is done. That loop is the engine bay. What makes it extensible is that it gives you four places to bolt things on, and once you understand those four, you understand most of what modern AI engineering actually is.

Skills are self-activating capabilities. A skill is a folder of instructions and tools that the system loads into context only when it's relevant, through something called progressive disclosure. The model sees a one-line description of every skill at all times, and only pulls in the full instructions when a task actually calls for it. I'm running 54 of them: writing, research, security analysis, code review, adversarial critique, project scaffolding. They don't sit in the prompt eating tokens. They show up when they're needed and disappear when they're not.

Hooks are the opposite of self-activating. They're deterministic code that fires at fixed points in the loop, whether the model wants it to or not. Before a tool runs, after a tool runs, when a session starts, when the context is about to compact. This is the part people miss about reliability: the model is probabilistic, but the rules around it don't have to be. Hooks are how you bolt non-negotiable behavior onto a non-deterministic core.

MCP, the Model Context Protocol, is an open standard for the edge where the model meets the outside world. Before it, every model-to-data-source pairing was its own custom build. MCP collapses that into one protocol, so any assistant can plug into any tool or data source without a bespoke adapter for each one. The detail I find telling: MCP started as Anthropic's own standard and then got handed to a neutral foundation, with the other big labs signing on. The company selling the model is helping commoditize the layer right above it. That's not charity. That's everyone agreeing on where the value isn't.

The fourth point is the agent loop itself, which is where it gets interesting.

Reliability is a different problem than capability

Getting an agent to do something once is easy. Getting it to do the right thing every single time is a completely different problem, and almost all of the real engineering lives in that gap.

There's a metric that makes this concrete. Most people measure agents with pass@k, which asks: did it work at least once in k tries? That's the right question if you have a verifier to pick the winning answer out of the pile. But the metric that matters for anything you'd deploy is pass^k: did it work every time? An agent that's 90% reliable on a task drops to about 43% if you need it to clear that same task on all eight of eight attempts, because the failures multiply instead of averaging out. Sierra found that even state-of-the-art agents stay consistent on fewer than a quarter of retail tasks across eight runs. The demo works. The product is the part that survives being run a thousand times.

So how do you buy reliability? Mostly by spending compute at the moment of inference instead of hoping the model nails it on the first pass. The textbook technique is best-of-N: generate several candidate answers and use a verifier to keep the best one. It works because the model usually already places some probability on the right answer, so the bottleneck is selection, not generation. The catch is that it only helps when the model can already get there sometimes. If the answer is outside its reach entirely, sampling more just buys you more confident wrong ones.

And here's the part that's easy to miss, the one that actually matters. Best-of-N and majority voting raise pass@k. They do not automatically raise pass^k. Picking the best of five answers makes you more likely to be right once; it only makes you reliable if the picker is good enough to run on every single deployed turn. Spend compute to win the lottery and you still don't have a dependable system. The whole thing collapses back onto how good your verification is. That, not the sampling, is the actual skill.

My own system doesn't run best-of-N in the textbook sense. What it leans on is the principle underneath it: let the model generate, then verify hard against criteria that return a plain yes or no, and fan the work out to parallel specialist agents when it splits. I built it around a forced verification structure I run for anything non-trivial. Turn a vague request into a written spec with binary criteria, then don't let the work be called done until those criteria pass. It runs across seven phases and spawns those agents when the work needs them. Anthropic reported that one of their multi-agent setups beat a single agent by about 90% on an internal research eval. Their number, their benchmark, not reproducible from outside, and they were candid that the gain came mostly from parallel coverage and a lot more tokens rather than a smarter model. Different lever than verification, same direction: the leverage is structure, not IQ.

The layer on top

The framework I built all of this inside is called PAI, Personal AI Infrastructure, and it comes from Daniel Miessler. I found his work about seven months ago and have been building my own version since.

What I took from him wasn't a tool, it was a thesis: AI doesn't have a capabilities problem, it has an integration problem. My own gloss on it is that a model with no durable context is a genius with amnesia, and the value only shows up once you give it continuity. My system gets better most weeks, and almost never because the model changed. The model I'm using is not dramatically smarter than it was seven months ago. The system is, because every run leaves something behind.

How I actually use it

The most useful thing it does is collapse the distance between a vague intent and a finished thing.

One problem I keep it pointed at is insurance document review, which is a firehose of medical and legal paperwork that someone has to read carefully and most people would rather not. I built a pipeline on top of this system that ingests a demand packet, reconstructs the medical timeline, flags pre-existing versus accident-related injuries, and writes a discrepancy summary, with every claim cited back to a page. A human still makes every real decision. The thing never decides; it reads, organizes, and shows its work, and a person signs off. That distinction is the entire design, because everything serious I've read about AI in legal and medical work points the same way: these systems still hallucinate enough that a human in the loop is not optional.

Outside of that it's the same shape pointed at different problems. I built a customer-service agent against an open airline benchmark, mostly to feel where an agent's reliability actually breaks, and the most impressive thing it did was refuse to do something the policy forbade and then not quietly do it anyway. I run a fitness system that pulls my watch data and prescribes training against my actual load and recovery, and it already knows my history and what I'm training for, so I never re-explain any of it. The harness travels between all of these. The context is what carries across.

What I actually believe now

There's a real argument against everything I just wrote, and you have to take it seriously to land anywhere honest.

The counter-position, roughly Rich Sutton's bitter lesson, is that elaborate human-built structure tends to lose to general methods that scale with compute, and that a lot of scaffolding is just coping with a model that isn't good enough yet. The sharper version, which I've seen from people building frontier coding agents, is that scaffolding is coping rather than scaling, and that as the models get better the clever harness dissolves.

I think that's half right, and the half it gets right is important. A lot of scaffolding is coping. Chains of prompts that paper over a reasoning gap, retry cascades, brittle output parsers, anything that exists only because the model has a weakness this quarter. All of it is temporary, and the next model erases it. If you're building that, you're building a sandcastle.

But there's a different kind of scaffolding that doesn't erase, because it isn't compensating for the model at all. Verification doesn't get less valuable as models improve; a smarter model still needs its work checked against a spec when the cost of being wrong is real. Memory and context curation matter more as the work gets longer, not less. And you'll always want some rules to run every time, regardless of what the model feels like in the moment. That kind of scaffolding compounds with model quality instead of competing with it, and the skill is knowing which kind you're building.

Then there's the hardest version of the objection, which is the one I find most convincing. If the model commoditizes, and the tool layer commoditizes through something like MCP, the harness commoditizes too. Every lab ships one now, all converging on the same agent loop with the same handful of extension points. So if the engine is a commodity and the car is becoming one, where is anything durable?

The answer I keep landing on is the one layer nobody can sell you, because it was never theirs to sell. It's the context. Not memory-as-a-feature, which every lab is racing to ship, but the specific accumulated record of how I work, what I care about, what I've already tried, the standard I hold something to before I'll call it done. The files can move; that's not the point. The point is that the thing itself is mine, built run by run, and there is nothing to buy off a shelf that replaces it. A lab can hand everyone a better engine and a better car. It can't hand them my mileage. That's the part that compounds, and the reason it resists commoditizing is that it was never a product in the first place.

So here's where I've landed after a year inside this. The model is the most powerful, most substitutable component in the stack, and it's the one I think about least. The harness is where reliability lives, and it's converging for everyone. The only thing underneath that's truly mine is the context, and that turns out to be the whole point.

The model just shows up and does the thinking.

What's next

The system is never done, which is sort of the point. Every task sharpens the criteria, adds to the memory, leaves the harness a little better than it found it. The next thing I want to understand is what holds when it scales: not a few dozen skills but hundreds, not one task but thousands running at once. Reliability is cheap to fake when the world is small. I want to know what holds it when the world isn't.

The architecture is public, cleaned of anything personal but structurally complete, at github.com/MaxHarar/PAIArchitecture. If you want to build something like this, that's where I'd start. Not with a better prompt. With a better car, and a record of your own mileage that nobody else can hand you.