Why AI Pilots Fail:
Nate B. Jones on What to Do Instead

Nate B. Jones, who advises Fortune 500 companies and global banks, opened CXOTalk episode 927 with a blunt diagnosis. "AI pilots fail because they're pilots. That is the critical error that most leaders I talk to run across after the fact."

In conversation with host Michael Krigsman, Jones argued that the pilot frame does the damage all by itself. "We envision the pilot as a way to de-risk AI, but what we find in practice is that by naming and defining it as a pilot, you end up putting less resources behind it than you should. You pick more fragile and less important goals than you should, and you don't get the learnings you want ultimately, because the question of AI is a question of whole organization transformation."

Start with leaders, then pick work that matters

Jones gives executives two starting points. The first is personal. He tells CEOs that "transformation starts with you," which means using the tools they expect their teams to adopt. "You cannot just be talking about AI. You have to be living and practicing it." In his view, "for most people who are not named CTO, you're going to have to become more technical."

The second is project selection. He finds that "almost everyone has a charge from their board at this point to say, you know what? We got to do AI," and a project list to match, picked up from LinkedIn, from the CTO, or from the leader's own reading. Jones tells them to pick the project where success would be transformational, then name the inputs that success depends on and trace where each one lives in the business. A process whose value is still ambiguous can wait, however important it looks.

Where adoption breaks down

Asked whether the obstacles are technical or organizational, Jones put the split at "about 20% a technology problem and 80% a people problem." Better models have not changed that. "I have seen model release after model release after model release," he said, and none of it "has changed the people dynamics in these businesses."

The people work starts with senior middle managers, who carry the change on the ground. He said that "if they're not champions in practice, nothing works." Once a project reaches working teams, Jones expects "a very predictable bell curve of adoption." The top ten or twenty percent need no persuading; they are already experimenting with AI at home and are relieved the company caught up. The middle of the curve moves only when people understand the change matters for their careers and feels doable, which for most teams means tools that are less technical and a definition of success they can actually hit.

The technical issue he sees most often is data. Without a clear picture of how data flows through the business, Jones warned, "you're going to have effectively a Ferrari engine with your LLM, but you're not going to have anything to feed it."

The harness is the company's intellectual property

For AI in production, Jones kept returning to what he calls the harness, the surrounding system of context, memory, reusable procedures, and review gates. He described it as "a way of encoding the organization's understanding of how to do business, its competitive advantage around the LLM." The frontier labs have reached the same conclusion, he noted, which is why they now send engineers into customer organizations: "you can't just stick a raw LLM into an enterprise and expect magic to result."

A strong harness also changes what a new model release means. Jones described companies where the harness is "relatively thick" and the model inside it is, in his phrase, "just the utility intelligence." There, the harness determines the business outcome, and a new release is absorbed by adjusting the harness, not by starting the transformation over.

Talk in dollars

On economics, Jones pushed leaders to change the unit of measurement. Cost per token comes up constantly, he said, but cost per task is "a much more actionable measure of the value of AI," and each business has to define its own "cost per completed action." The budget conversation follows the same logic. He reminded leaders that "capital gets allocated where you can get a return. And so if you can talk in dollars, you get so much farther with the CFO, you get so much farther with the people who are actually figuring out the budget."

Jones was just as blunt about jobs. Among the knowledge workers he talks to, fear that AI will take their jobs is the number one fear, particularly in the United States, and he tells C-suites to treat it as the elephant in the room. Some leaders have told him outright that they want AI as a staff cut. His warning back: "Teams know how to smell that kind of fear, and they will sniff it out." Workers can withhold what is in their heads, keep running the old process quietly, and "sabotage the AI effort in dozens of small ways that are very hard to notice."

Walking away is not failure

Abandoning a pilot, Jones argued, is only a failure when nothing is learned. "The question of whether it's a failure or not is really a function of the organization's ability to learn." The right moment to stop is when the goal itself changes. "That often happens either when business circumstances change or when you understand more about AI during the course of the pilot and you're like, actually, I picked the wrong goal."

Behind the specifics sits a larger claim about the firm itself. In his words, "for hundreds of years, the firm has been about people figuring out how to get things done in teams together, given capital. And now the firm is about people figuring out how to partner with artificial intelligence to get things done given capital."

Watch the full conversation and read the complete transcript on the episode page.

CXOTalk prepared this article with AI assistance from the verbatim transcript of episode 927. Quotations are unedited from the broadcast.