Back to Insights
·5 min read·Adam Roozen

Hidden Meta: The Routing Layer

Route most of your traffic to small models and the bill falls 40 to 85%. The edge moved out of the model and into whatever decides which model to call.

A Forbes headline from June 25 would have sounded absurd a year and a half ago. Small language models beat frontier AI on cost, on speed, and on accuracy.

Not on some narrow benchmark. On the things buyers actually pay for.

I've run my own stack this way for a while, so the finding didn't surprise me. What surprises me is how few companies have restructured around it.

Route 70% of traffic to a small model and 30% to a frontier model. The input-token bill drops about two thirds. Push it to 80/20 and you approach 79%. Teams that tune the routing layer report 40 to 85% savings with no visible quality loss.

Getting a Baseline

A frontier model is trained to be good at everything. So what are you buying on a call that tags a support ticket? Capability you never use.

Classification runs fine without it. So does extraction, summarization, reformatting, tagging and validation.

That's most of what a production system does all day.

Llama 3.1 8B, Phi-3 Mini, Mistral 7B and Haiku handle that work for a fraction of the price. They also answer faster, which matters more than people expect once a workflow chains ten calls together.

The Shape That Works

One frontier model sits at the coordination layer. It reads the task, breaks it into pieces, and hands each piece to a worker running something cheap.

It only comes back when a worker hits something it can't resolve.

Expensive tokens go where judgment happens. Everything else runs at commodity prices. If that sounds like how you'd staff a team of people, it should.

So Where's the Edge?

Everybody has the same models. Nobody gets an edge from picking Opus over Gemini.

The edge is in the routing logic, the verification loops, the small models tuned on your own data, and the evals that tell you when a route quietly started degrading.

That's the part a competitor can't buy. It's also the part most AI budgets underfund, because it doesn't demo well to a board.

The Trap

A badly tuned router is worse than no router.

Send a hard problem to a 7B model and you don't get a cheaper answer. You get a confidently wrong one, and wrong answers cost more downstream than tokens ever will.

The real work is in escalation – a worker has to know when it's out of its depth, and it has to say so instead of guessing. Most teams skip that part, then blame the small model.

A Question for Your Vendor

What percentage of your calls hit a frontier model?

If the architecture is one big model answering everything, you're paying frontier prices to classify support tickets. If they can't tell you the number, that's your answer.

Wrap-up

Here's why I care beyond the invoice.

If most production tokens flow through small models on cheap hardware, then demand for the biggest training runs and demand for inference capacity stop moving together.

Right now the market prices them as one thing.

I don't think that holds.

Written by

Adam Roozen

Strategic Advisor. AI Strategy, Digital Commerce, Technology Transformation

Nearly 30 years of operating experience · Walmart · Sam's Club · Echidna

Work with Adam