Back to Insights
·4 min read·Adam Roozen

Expensive attention

A Forbes headline, a routing layer, and the price of knowing when to hand it back

A Forbes headline from June 25 reported that small language models beat frontier AI on cost, speed, and accuracy. I keep getting stuck on the word beat. The interesting thing under it, I think, is not a winner at all. It is a split - two kinds of work hiding inside one budget.

Let's look at how the routing works first, and then carry the idea somewhere else.

"Expensive attention" means sending scarce, costly judgment to the places where being wrong changes what happens next, and letting repeatable work run where being right is easy to check.

The routing layer

Most of what flows through an AI system turns out to be small errands. Classification, extraction, summarization, reformatting, tagging, and validation can run without a frontier model. Llama 3.1 8B, Phi-3 Mini, Mistral 7B, and Haiku will do those jobs for a fraction of what a frontier model charges, and they answer faster too.

Once the work splits that way, the bill stops being one rate. Route 70% of the traffic to the small side and the input-token bill drops by about two thirds. Teams that spend real time tuning this layer report savings of 40 to 85% with no visible quality loss.

The high end of that range deserves a gentle eyebrow. Savings depend on the mix - tickets, docs, chats, edge cases - and no two mixes seem to misbehave in the same way.

The coordinator

Now here's someone running it. We'll call him Bob. Bob is the coordination layer - one frontier model, and the only expensive seat in the building. Let's say a customer ticket arrives. Bob reads the ticket, breaks it into pieces, and hands each piece to a worker running a cheaper model. Pulling the order number is extraction. Tagging the topic is classification. Those jobs come back fine. Then one worker gets the piece where the customer is asking for something no form has a box for.

Let's walk the mechanic the rest of the way:

(a) Bob reads the task and decides what kind of thing it is
(b) the clear pieces go down to workers running cheaper models
(c) Bob comes back only when a worker hits a piece it cannot resolve

The whole arrangement holds or breaks at step (c). The worker that cannot resolve its piece has to know it, and it has to say so instead of guessing.

Because the alternative is not a cheaper answer. Sending a hard problem to a 7B model produces a confidently wrong one, and the 7B model will not notice. The coordinator only comes back when somebody raises a hand.

The escalation

That raised hand is the part I keep circling. It sounds almost moral, which is too grand a word for a router. The mechanic itself is plain: when a worker is out of its depth, it has to say so and stop, before a wrong answer starts moving.

I think this is also where the headline's frame goes thin. Cost, speed, and accuracy are columns you can fill in. But the architecture leans on a fourth thing that is not quite a metric: the worker's willingness to admit the miss. Nothing on the price sheet measures that. And if the worker guesses instead of admitting, who catches it?

Let's isolate expensive attention and see where else it lives.

Other rooms

Let's take the idea out of the token bill and walk it through rooms that have nothing to do with AI.

In a clinic, most of the day is forms, repeats, reminders, the same questions in different coats. Then one detail in someone's story sits wrong. Not louder. Just wrong. That detail is where expensive attention goes. The rest of the chart can run on the cheap side.

In a kitchen, the repeatable work is prep, portion, label, clean, count. (My own kitchen routing is bad. I escalate the toast.) None of it needs the most experienced eyes every minute. The eyes belong on the moment the room changes: a supplier box that looks off, a table waiting too long, a new hand moving faster than the quiet rules got taught.

In an inbox, a friend sends ordinary notes all year and then one where the verb tense slips. The ordinary ones take almost nothing. The other one needs a different kind of reading, and it is probably not small to the person who sent it.

Across those rooms, the pattern never asks for a fancier tool first. It asks where a miss travels. If a miss stays local and visible, cheap attention is enough. If it compounds, or hides, or teaches the next step the wrong lesson, the expensive kind has to be there.

The admission is the part I can't get past. Not the savings, and not the model names. The whole arrangement leans on the cheapest workers in the building knowing when they are beaten and saying so out loud. Honestly, I don't know where that gets priced, or who is supposed to test for it.

Written by

Adam Roozen

Strategic Advisor. AI Strategy, Digital Commerce, Technology Transformation

Nearly 30 years of operating experience · Walmart · Sam's Club · Echidna

Work with Adam