The Everything Else
On why a working pilot can still go nowhere.
A company decides it's time to do something with AI. Budget gets set aside, a pilot gets built, the demo goes fine, the model performs more or less as advertised. And then, some months later, the whole effort has quietly evaporated. Nobody cancelled it. Nobody can quite say what broke, either.
Let's look at how that happens, and then lift it out, because it isn't really about AI.
The everything else
First I want to name the thing we're hunting, because it doesn't seem to have a name of its own, at least not one I've heard.
"The everything else – all of the stuff surrounding a thing that ends up deciding whether the thing worked."
The idea is simple enough. Whether something counts as a success seems to get decided by everything around the thing (the inputs, the ownership, the plan for what happens next), not by the thing itself. So the thing can work and the outcome can still go badly. And when the everything else is missing, a working thing and a broken thing look pretty much identical from the hallway, at least for a while.
The online store era
None of this is exactly new, I think. During the e-commerce inflection in the early 2000s, companies rushed to get online stores live and barely thought about fulfillment, about inventory, or about what was supposed to happen after the click. In plenty of cases the website itself – the thing, to be plain about it – probably worked the way it was meant to. People could browse and buy. What seems to have fallen apart was the part of the business that only starts after the click, the warehouses and the stock and the rest of it, because nobody had built that part.
One pilot, up close
Same pattern, newer costume. Let's put someone inside it, because these things read truer with somebody standing in the mechanism. Call her Dana.
Dana builds things at a company big enough to have committees. The job she gives the AI is ticket triage: read an incoming support ticket, decide which queue it belongs in, send it there. Small enough to show in a meeting. Let's walk it with her.
Her training data is whatever ticket history she can export, and what she can export is partial. Fields half filled in, notes that stop, categories that appear to have changed meaning at some point. She knows a model only ever performs as well as the data underneath it, and that clean, accessible data has to come first. She also knows hers doesn't, not fully. She builds anyway, because partial data is the data there is.
Then she demos it. And here's the part that changes how the whole story reads: the model seems to work. It reads tickets, routes them, routes them reasonably well. The AI did what it was designed to do. What failed, as far as I can tell, sat around the model rather than inside it. Concretely:
Look at that list again. As far as I can tell, none of those are AI problems. The model could maybe have been twice as accurate and the ending would have been the same.
Some time passes. Nobody kills the pilot. It just never becomes anything. And the story the company ends up telling itself is that AI isn't ready, or maybe that they weren't ready for AI. Both of those read wrong to me, because the thing worked, more or less. What was missing was the everything else.
Whose fault is that, then? Honestly, probably nobody's in particular. Dana shipped a working model; her piece was done. Nobody owned the pilot, so no one was ever required to carry it forward. Success was never defined, so there was no moment where anyone had to look up and notice the thing working. Each gap looks minor in isolation. Together they mean the tool works but never quite gets used. (I've sat in that meeting. Both sides of it, tbh.)
The pattern somewhere else
Alright. Let's isolate the everything else so we can use it elsewhere. That's the whole point of giving it a name.
Take cooking. A recipe can be tested, measured, verified step by step. If nobody went shopping, the right pan doesn't exist in the kitchen, and guests show up in an hour, dinner fails and the cook takes the blame. The recipe was sound, maybe even excellent. What failed was the shopping, the equipment, the prep hour nobody scheduled.
Or hiring. A company lands a genuinely strong hire, then gives her no real onboarding, no manager who owns her first stretch, no shared picture of what a good start even looks like. A few months later, it 'just didn't work out.' I'd guess what didn't work was the arrangement around her: no owner, no agreed measure, no path from a signed offer to actual responsibility.
A training program seems to behave the same way. The program itself might be well designed. The outcome gets decided by the sleep, the food, the schedule, all the hours in the day that the program never touches.
The ownership piece is the part I can't resolve. The model has an owner. The data has an owner, usually, or at least a custodian. The everything else mostly lives in the gaps between the job descriptions – the work that no role was ever written to cover, because it didn't fit neatly under any of them. And work that belongs to no role seems to drift until it belongs to no one, which is maybe how the everything else so often ends up with nobody looking after it. I keep coming back to that. I don't have a tidy fix for it.
Written by
Adam Roozen
Strategic Advisor. AI Strategy, Digital Commerce, Technology Transformation
Nearly 30 years of operating experience · Walmart · Sam's Club · Echidna
Work with Adam