AI Prompting for Business Leaders: What Works and Why

Understanding Prompting and Its Importance

Michael Krigsman: Prompting is the secret skill that taps into AI's real capabilities, transforming large language models from flashy demos into engines of real-world productivity. Today on CXOTalk number 883, we unpack how prompting works, what it is, why it matters, and how to get it right. I'm Michael Krigsman, and with me is Nate B. Jones, a widely respected AI expert whose sharp insights and no-nonsense advice have earned him nearly 300,000 TikTok followers.

Nate Jones: We need to talk a lot about prompting for two reasons. One, human language is fairly vague. That's why we invented computer language back when we were programming computers in the first place because it's much more precise. And now we're using effectively natural language to program computers again, and that's challenging.

The second reason is that even though these models are very intelligent in certain respects, they are not incredibly reliable yet at inferring your intent if you are not precise about what you mean or want. They don't do that reliably. They guess, and they might guess right, and they might guess wrong. And both because we have to get clear with our language and because models don't yet infer with tremendous precision, prompting is what bridges that gap.

Michael Krigsman: When we are prompting, we're programming the AI.

Nate Jones: This is really going to take you back, but like in the old days, in the '60s with punch card computing, you would literally bring your little punch card and put it into the computer, and you would run it. And you would see in 20 or 30 minutes whether you got that right or not, or maybe longer than that if it was a big program.

We're doing exactly the same thing with natural language now. We're handing the prompt to an inference model, maybe O3 Pro. It does take that long, 20 or 30 minutes, and we're going to come back, and we're going to see if our little natural language program did anything. It's fascinating how time is a circle in that regard. We're back to where we started.

Michael Krigsman: The logic of prompting is effectively the logic of software development. Is that a correct way to say it?

Nate Jones: You could say it as like the marriage of software development and business intent. In a sense, software development has been predicated primarily on building interfaces that allow business operations to be conducted, business logic to be encoded, et cetera.

But now because these models have the ability to bring intelligence to bear, you're not just asking it to do one specific thing. You're not spending your time engineering a specific interface. Instead, you're asking the model to think with you. And it's this weird mix of the principles of engineering with the business clarity of intent that has always characterized a very strong executive brief, for example.

The Evolution and Challenges of Prompting

Michael Krigsman: As the models get better, what does that do to prompting? Does it make prompting easier or more difficult?

Nate Jones: On the one hand, you don't have to do some of the stage management that you had to do in 2022 and 2023 anymore. You'll recall when ChatGPT first came out, the prompting guides were like, "Okay, tell it to pretend it's the best editor in the world. Tell it to pretend this or that," and then it began to turn into chain-of-thought prompting. "Tell it to think step-by-step. Do this, and do this, and do this, and do this." That stage management is thankfully no longer really necessary.

You can instantiate the model by saying, you are in a particular space, like you're in a consultative strategist space, you're in a CFO space, whatever. You can say that, but you don't have to put in the adjectives and hope and pray that the model understands what you mean. You can just say, "This is where we are." You don't have to specify chain of thought anymore. The frontline models know to use chain of thought when they need to.

And in that regard, prompting has gotten simpler. On the other hand, the importance of specifying what you're looking for, what success criteria looks like, what the constraints are, that's only gotten more important because these models are much more powerful.

And before, like if it was a very simple ask that you had for a smaller model, you could go back and forth a few times and figure out what you wanted, and it was fine. But if you give something to a frontline model and it's running for 6 minutes, 8 minutes, 10 minutes, 20 minutes, it comes back, and you just did not clearly specify the scope, you're going to be frustrated because you wasted all of that compute.

And in that sense, some of the stage management and scripting that you're used to, you don't have to do anymore, but the importance of specifying the work very clearly has grown. Like, you have to really take that seriously now.

Michael Krigsman: You really need to match the prompt to the model.

Nate Jones: A lot of the art of it is in figuring out what is this subject, what is my intent, what is the right model for that? And once I have all of that figured out, now how do I craft a prompt and then bring in the context the model needs it can do a good job for me?

Michael Krigsman: For example, with OpenAI, a number of the models allow you to either include deep reasoning, or research, or not, and actually other companies as well, other LLMs, same thing. Give us some examples of this.

Nate Jones: This is one of the things where model makers have not done a great job at the product surface of explaining what their models do. For example, deep research is really a very narrow web agent that is trained as a research assistant to go out to look across the entire browsable web, it doesn't yet look behind paywalls, and to come back with a consolidated view, and they train it specifically on citations. It's good at citations, it lists what it knows and why.

OpenAI pioneered this with Deep Research, but Deep Research is now available on Perplexity, it's available on Claude, it's available with Google. Lots of others have picked this up because it turns out that reasoning across the web is a lot of what we do, and there's just inherent value in report generation.

But people don't realize that all you're getting with Deep Research is the O3 model, if it's ChatGPT, specifically tuned to web search, and that is different from whatever else you've been talking about with whatever model you've been talking about in ChatGPT previously.

If you've been having a conversation with 4o for a bit and then you turn on Deep Research, it's not that 4o suddenly picks up a cape and becomes a superhero and turns into Deep Research. It's that you are invoking a separate agentic tool, getting a separate prompt in, starting a new flow, and then that report is going to come back and you're going to be able to continue the chat.

And I think that a lot of people don't think about it that way, and it's become even more confusing in the last week because O3 Pro on the surface looks very, very similar. It's got a long thinking time, you give it a prompt, it goes away, it comes back. And people have asked me, "Did they just release a clone of Deep Research and rename it?" And the answer is no.

The answer is that O3 Pro is a generalizable model with a lot of different tool calls under the surface, but precisely because it's under the surface, it's difficult to know that staring at the chat window when it takes a similar amount of time and comes back. And I think that some of what I do is just try and convey the nuances of these models and how understanding them with a little bit of a fingertip feel can shape the way we prompt.

The Future of Prompting and Its Impact

Michael Krigsman: Again, there's a level of confusion here. I mean, I use many different models every single day, and I am, on an ongoing basis, having to kind of experiment. You know, it's like this whole domain is very immature because the models are changing and the models give indeterminate results in any case, and that means you keep having to adjust your prompts on an ongoing basis. It's really a waste of time.

Nate Jones: I think if it was a waste of time, we wouldn't be seeing the kind of tremendous uptake we see on groundswell usage with these models. One of the biggest challenges with IT and security this year is shadow IT, where people are finding these models useful for the work that they do that they are using them even outside traditional IT security practices.

And, in that sense, I share your frustration. I find that when I am not getting what I want, there's nothing more frustrating than pounding my head on the wall and trying to figure out what the model needs to hear from me it can give me what I want.

But net-net, if I look across my overall productivity for the day, for the week, I am much more productive now, even with all of that factored in, than I was two years ago. And it's because I'm learning enough about how to work with these models that I'm able to get a tremendous amount of value back, and I think a lot of people are having that experience.

Michael Krigsman: And maybe I shouldn't say it's a waste of time, although I do think it's a waste of time, but let's just say that there's a lot of overhead that seems like it shouldn't be there.

Nate Jones: That's a really fair call.

Michael Krigsman: That's basically a complaint, and not that it makes any difference at all because that's the nature of the maturity of these models as products at this point in time.

Nate Jones: What I am curious to see answered by the model makers in the next probably 18 months is the extent to which prompting remains a durable skill set that provides tremendous alpha to people who know how to use it well versus the extent to which it commoditizes, not necessarily because everyone learns the same amount, but because models get very, very good at inferring intent across a range of prompts for the same subject. And people are widely divergent on what they think will happen.

The Evolving Role of Prompting in AI

Nate Jones: My own view is I'm trying to take seriously the fact that I expected initially prompting to be a very one-off 2022, 2023 edge, and that's not been the case. It's been stronger and stronger over time instead. I tend to lean toward the idea that, at least for the intermediate term, prompting is going to continue to have a tremendous amount of value because that's what we've seen as a trend far.

There are people who think that if we can get to a level of generalizability with these models, we will suddenly unlock a tipping point and we will find a way to infer very reliably where we haven't before. And that might be. And if that's the case, then suddenly prompting will become less painful and less needed somewhere in the next 18 months.

Michael Krigsman: Subscribe to the CXOTalk newsletter you can be part of our community. We have amazing shows coming up.

Best Practices for Effective Prompting

Michael Krigsman: What makes a good prompt? Do you have any practical advice?

Nate Jones: Number one, be really clear about the outcome that you are looking for and about how the model can know that it's done. I think a lot of people will be fairly loose about specifying the outcome or they'll be loose about the goal. They'll be very, very loose or nonexistent about how the model can know that it's finished adequately. And the more you can specify and be clear about what you're looking for and what good looks like, the better off you're going to be for the rest of the prompt.

Number two, you want the model to have all the context that it needs to do that job, and you would prefer it to not have any extra context that it doesn't need. A lot of what we call hallucinations are effectively models reasoning outside your desired context window.

And if you can be more clean and clear about, "This is what I want you to focus on in a web search," or, "Here's some documents I want you to review. I want you to keep your thinking focused around this particular set of meeting transcripts," or whatever it is, it will really help the model to be confident that it's doing the right job and able to deliver a reasoned result that closely matches the kind of work you were looking for. And the context piece is another one.

And then the third is really making sure that you understand the constraints and guardrails that you want to put around. If you have an outcome or goal, if you have context you feed it, you then need to make sure that the model knows, "Don't do this. Where do I not go?" And I find that that is often one that people either barely put in or tend to avoid because we tend to be thinking in a positive stance of like, "Hey, this is what I want done. Let me just give the task and go."

And maybe this is because we're anthropomorphizing models. We don't tend to regard a senior colleague as someone who needs a tremendous number of warnings and constraints for a task. We just say, "Hey, go tackle this. I'm sure you'll do a great job. Come back and let me think about what you get."

These models need those constraints still. Even if they, in many ways, are very senior in their thinking, they still need helpful constraints they know where the guardrails are in the space and they don't start to reason off the rails into a direction that isn't helpful.

Because at the end of the day, what they're really trying to do is just infer from your utterance what they think you mean, figure out where in latent space they can go and get a reasonable pattern match, do some searching across the web, in the case of an inference model, do a lot of that iteratively they can figure out what's best, and then put together something. And they do need those guardrails to constrain.

Michael Krigsman: Models are people too. And you can't... Just as you can't expect your friend or your spouse to read your mind, how can we expect models to anticipate every possibility that's out there and map it to what happens to be in your mind at this given time, what you want when you write this prompt? And thus the need to be explicit. And that's why I say models are people too.

Nate Jones: We as humans are very, very good at retaining long context from multiple conversations with our colleagues and extracting what's really important out of that and getting to clear points of discussion. Like, I can talk with a software development manager about a project that's been going for six months. We can have a really meaningful discussion about the sticking point, decisions we made in the past, what we need to change. That is how humans have done work for a long time. We iterate over time. Effectively, the prompt evolves through conversation over time. It's shared work together.

With the model, we can't have the same iterative conversation. We actually have to front load all of that thinking and give it to it in a really clean prompt we can get a really clean answer. And I think part of what's hard for us about prompting is we're conversational people. We like to chat just like you and I are chatting. We make meaning that way. But the model needs us to compress that semantic meaning into a really clean initial prompt that will help it to work effectively.

Michael Krigsman: I think this is a very important point that you're making is there is the need to, as you said, to compress the context into a digestible set of words and chunks that the model can then use to execute the explicit task. Going back to an earlier comment you made that, in effect, you are programming into the model, into the... and driving the conversation through that programming essentially.

Nate Jones: Right. Because we humans effectively collectively derive intent and collectively reach decisions through conversation, but the model needs you to be the one that provides the intent, that provides the driving force. There's a higher expectation of human agency in prompting.

Expertise, Iteration, and the Future of Prompting

Michael Krigsman: Let's jump to some questions. If you're watching on Twitter, pop your questions into Twitter using the hashtag CXOTalk. If you're watching on LinkedIn, pop your questions into the LinkedIn chat.

And this first question on Twitter, X, goes to Arsalan Khan, who says, "It seems like asking specific questions in your industry to the AI would deter fake experts, but how would an end client know the difference between real experts using AI versus just good or fake experts or salespeople using AI?" And if I can restate that question in terms of prompting, if somebody is really, really good at prompting, can't they present the appearance of being an expert where it's almost impossible to tell them apart from somebody that has the PhD in whatever the subject might be?

Nate Jones: That is more true than many of us would like to admit. I think it's part of why there are many consultants springing up, many tools springing up. The industry has a need for authenticity. But AI by its nature is enabling many people to claim expertise that they don't genuinely have.

And there's, just like there's not a silver bullet solution for detecting text in student essays and saying who wrote which bit of text, there's also not a silver bullet solution for detecting expertise.

I find, in practice, what tends to be most helpful at distinguishing a true expert from the AI-generated straw man expert is acknowledge the source material. It's probably going to be very good because AI helped prepare it, it's very thorough, et cetera. Make sure you understand it. And then ask a question that's designed to push off balance, push out of the comfort zone, and a true expert will be able to adjust and have an interesting and thoughtful perspective and not get too frustrated or flustered. And someone who's depending heavily on the prompting is often going to struggle because they won't be able to actually have that flexible intelligence across the domain that characterizes true expertise.

Michael Krigsman: You're saying that there's a level of depth that the model doesn't... or that a person who's simply relying on the model doesn't have. Is that another way of saying it?

Nate Jones: I'll give you an example. I was doing an article on prompting in O3 Pro versus O3, right? 'Cause O3 Pro came out this week. And I asked it to prepare a roadmap because I'm very familiar with roadmaps. I came up through product management. I've seen more of them than I would care to admit.

And I asked for that because I knew I could judge it. I knew I had the expertise to assess it. And I was talking with someone afterward, and I was saying, "O3 Pro did a much better job on the roadmap." And they were like, "Well, how did you know? Aren't roadmaps subjective?"

And I immediately pulled up three or four reasons why roadmaps are not subjective, why it's actually a craft you can understand, and nine out of ten experts will agree with you that a particular roadmap is better than another because it's a proactive stance, it takes into account all of the strategic advantages the company has, it thoroughly understands the marketplace. I could just go on and on. It's all at the top of my head.

And having that expertise helps you to assess the true quality of model response. And in a sense, what we're seeing here is that these models are getting to a level of intelligence where their very best work takes an expert to truly understand and appreciate.

Michael Krigsman: We have some really interesting questions that are coming in on LinkedIn right now. And Greg Walters is responding from the, to the point you made earlier, Nate, where you were describing the need for a compressed, highly efficient prompt, and Greg says this: "Isn't the magic in prompt iteration? Instead of having one compressed, highly efficient, and explicit task or prompts, shouldn't we be collectively prompting?"

Nate Jones: It depends on the kind of task that you're looking for. This gets back to the relationship between prompting and model selection. For certain kinds of models, they're more suitable to iterative thinking and iterative brainstorming.

We haven't really talked about the relationship between model and interface, but I find if I'm using advanced voice mode, it's just a very different experience for my brain because I'm talking instead of typing, and I am much looser and it's much more conversational. It is, in a sense, much more iterative, and I keep it that way on purpose.

But if I'm working with a long inference time model, and it's not just that ChatGPT has a monopoly on those. Claude 3 Opus is a great example from Claude. I want to be clear in what I'm looking for because frankly, it is expensive to iterate when the cycle times take that long.

The Role of AI in Leadership and Decision-Making

Nate Jones: And I pick the problem and I pick the model, and that guides me to a prompting style.

Michael Krigsman: We have a really interesting question from Wayne Anderson on LinkedIn and he says this: "How would you address the fear that leaders using large language models could inhibit and erode decision-making and critical thinking? When does effective prompting help and when do you think leaders should avoid using AI?"

Nate Jones: It's kind of like asking, do you want your doctor to avoid using AI if we have studies that show that medical reasoning is something that these models are very good at? I would love my doctor to use AI as long as my doctor understands how to use it well.

And in that sense, my response is, I want AI leaders to be using AI all the time. I just want them to understand the limitations of these models and where they need to think beyond the edges.

And really, I think it's more precise to say these are extraordinary models. In some places they are advancing the far edges of human thought and research. We have AI-developed drugs in pipeline. But they're narrow. They have like particular ways in which we can prompt them that generate extraordinarily effective results, and the strength of a good leader is not being only narrow. It's that T-shaped leader where you have that breadth of experience as well.

And what I would look for a great leader to do with AI is to know when he or she needs to go to AI for a deep, precise, thoughtful perspective on something, and then to bring that generalized experience of the business to bear to say, "This is how I would contextualize that and understand it for my broader problem set."

Prompting, Creativity, and Environmental Impact of AI

Michael Krigsman: But let me just go back to the comment that Wayne Anderson made. When I write certain things, I'll write something and I'll ask ChatGPT or whatever the model is, "What do you think?" and it will make suggestions. And this new canvas feature, I guess it's not new anymore, of ChatGPT makes it really easy to, like, drill down to very small segments. It produces good results. But in the back of my mind, I'm thinking to myself, "It's giving me kind of the least common denominator, mass market, generalized solution."

Nate Jones: Not necessarily. That might be your prompting. And I think that's what's interesting about these models is that you are correct that if you're not intentional about how you frame the model's position in latent space, it will default towards something that's more highly probable, which we often translate as the least common denominator.

If you are intentional, though, and you want to lean in and say, "I don't want a mid-answer, I don't want a common answer, I want a really creative answer, I want a really thoughtful answer, I want an answer that you haven't heard or seen elsewhere," models are perfectly capable of going that far and thinking more creatively, thinking more substantively.

But they don't do it by default because the way they're trained is to be helpful for as much of the population as possible. And in a sense, our own population distribution shapes the way the model makers are tuning these models for general helpfulness. And if we want something more on the far side of the distribution to push for it.

Michael Krigsman: We're drifting from prompting here, but is prompting going to create the next set of Bach inventions?

Nate Jones: No, I don't think. And I think especially in the creative arts, like, I would say that humans have tried... I'm actually a huge fan of Bach and the Cello Suites. I love them. I listen to them almost every weekend. And people have tried to expand on, to invent after Bach even through the 20th century. And in my view, no one has done for the cello what Bach has done for the cello. And no, I don't believe that we are in any danger of a machine coming along and doing a better job than Bach at Cello Suites.

Michael Krigsman: Let's jump over to Twitter from Chris Peterson, who says, "Tokens and time measures are all very well, but doesn't every round trip of prompting eat up more electricity and water for cooling, thus making some of the numbers from OpenAI and others highly misleading?"

Nate Jones: Or yes, it does, and yes, it matters in aggregate, and yes, we should talk about power use in aggregate. I think it's an appropriate conversation to have. But individual prompt usage by people doesn't compare to some of the other things we do day to day that use energy and water.

Taking a hot bath is much, much more expensive in water than any kind of ChatGPT prompt you're going to run. Watching an hour of football on the big screen is much, much more expensive in electricity than, I think it runs up to a couple of hundred ChatGPT prompts.

And does it matter in aggregate? Yes, because suddenly a billion of us are using this. It's important. We should talk about it. Not saying that we don't have relevant conversations to have. But I think the idea that an individual prompt is fantastically expensive is incorrect when we actually factor in the energy usage of a day-to-day life.

Michael Krigsman: Let's jump over to another question.

AI Tools for Data Analysis and Advances in Prompting

Michael Krigsman: This is from Chris Jablonski on LinkedIn, who says, "Do you have any tips for using gen AI for a data analyst to process a large dataset and generate visualizations?"

Nate Jones: I'm not sure if it's the right tool for the job. I've talked about this a little bit with folks who are managing large datasets, and what I find AI extraordinarily good at is handling datasets that don't have clean numeric data, right?

If you have clean numeric data, we have fantastic tools for that, and they may include machine learning or they may be just traditional SQL, but we're very, very good at handling that efficiently with compute. I don't know why we would switch that out and ask a large language model to do that when the language model wasn't even designed primarily to be numbers-driven.

They use Python and other tools to handle numbers now, and that's great, but if you're talking about a truly large dataset, we have tools that handle those datasets and visualizations really effectively.

And what I find people using in practice when they're looking at large datasets and AI is they're using AI to help them craft SQL statements, they're using AI to help them think through the data schema that they want to set up sometimes, they're using AI to help them prototype visualizations that they will want to get to quickly. Claude is great for that.

And all of those are by-the-way uses of AI that help you to use that data more effectively, but that's different from the traditional assumption that you can just type the query in and you will magically get a better answer than you would get with really efficient SQL.

Michael Krigsman: It's a really, really good point that you've got to have an understanding of the particular tool that you're using and what will be the most effective use of that tool. And as you said, prompts are great if you have a body of data and you're trying to figure out, "What have I got here, and how can I present it and is there something that I'm missing?" Prototyping, as you say. But there are tools out there that are designed for, you know, millions of records and that do it really well. I don't think you'd want to put millions of records into ChatGPT.

Nate Jones: It's not really designed... If you think about what we mean when we talk about context and prompts, it's designed to look across the overall picture. And oftentimes with data, we don't just want an overall picture, we want precision, and that's something that we have existing tools that do very, very well at.

Michael Krigsman: Let's get into the structure, the nature of large language models, how they think, "think" in quotes, and operate, and what that means for prompts. Maybe take us down that path a little bit.

Nate Jones: It's probably worth calling out that a lot of the difference in how prompting has evolved is being driven by this movement from large language models that are what would call vanilla, that it's just coming back with a response based on weights and vector space developed through pre-training data, which is what we had into 2024.

And then the newer version, which is inference time compute models, where they have that same underlying architecture, but at the time you press enter and send in your query, they are running threads in the background trying to figure out what the correct response is.

And there's different ways of doing that. Sometimes it's an combination of expert models in the background that are coming up with answers and deciding amongst themselves. Sometimes it's running the same query multiple times in parallel in the background, trying to find the most common answer.

Regardless of the underlying architecture, the effect of having more time to run cycles on your query is tremendous. It's a night and day difference in terms of the intelligence that the model is able to respond with. And that is a lot of what has shaped different prompting.

A lot of the reason we don't have to give chain of thought instructions anymore is because the models already have a way of deeply processing the queries we give them when they are inference models, and they don't need our help to do anymore.

And when I say you don't need chain of thought, but you want to be clear on your goals, I'm basically saying try and write a prompt that understands that you are going to be running multiple parallel streams of thought in the background or multiple parallel streams of tokens in the background. Constrain it.

Setting Goals and Chain-of-Thought Prompts

Nate Jones: Like if you have 10 that are going to run, you don't know it's 10, but let's pretend it's 10 for simplicity's sake. Make sure all 10 are focused on what you care about because you want to constrain the scope of the query it's actually focused on where you want to go with the conversation. And that's why I emphasize much set a goal, make sure the model knows what good looks like, make sure you set guardrails, et cetera, et cetera.

Michael Krigsman: Describe to us what you mean by a chain-of-thought prompt.

Nate Jones: It's where you said, "I want you to answer my query to a traditional model using pre-training data, weights," and it would come back and answer. But you wanted the token stream to go through a particular sequence.

And it's going to go through, and from a transformer perspective, like the transformers there, it's basically using your query, it's matching it in vector space once it vectorizes it with what it has for weights and pre-training data, and it's coming back.

And you're basically saying, "Let me give you a deeper query with a lot of things I want you to think about and do." Start with, this is who you are, you're an expert on marketing. Second, I want you to think very deliberately about this campaign that I want to launch. Third, these are the steps of thinking I want you to go through. First, develop a plan. Second, critique your plan. Third, understand the consequences of the plan in the market.

And you can kind of go through, and that's like chain of thought, right? When you do that, you're basically being very particular about the places in vector space that you want the model to go and hit when it's generating the response. And because models read like humans do, they read top down, when the model hits that point, it's going to be effectively sequentially reasoning back to you because of the way you programmed it.

Programming Models for Outcomes and Critical Thinking

Nate Jones: And this gets back earlier in our conversation, Michael, when we talked about this idea of natural language programming, that we are effectively programming the model. That was what we were doing. And all we're saying now is we still have to program the model. We don't have to program it quite that way anymore.

Michael Krigsman: How do we program it today?

Nate Jones: Today, when we program the model, we want to be focused more on outcomes and goals. And in the past, it was focused more on process.

And if I'm looking for a report, like I digested a 130, 140 page economic report this morning from the World... I think it was the World Economic Forum, something like that. I wanted the model to understand what I wanted out of the report and the goal of the summary. I didn't just want a vanilla summary. And my focus was on making sure it knew the angle I wanted on the report, and I trusted it to know how to read, digest, summarize, think through, all the things I would have had to specify earlier.

Michael Krigsman: In that case, again, the context, giving it the background and the goals becomes the key focus of the prompt as opposed to telling the LLM how to do its job.

Nate Jones: That's right.

Michael Krigsman: We have another question from Arsalan Khan on Twitter. Arsalan says, "To prompt or not to prompt? When is it appropriate and when is it just a rabbit hole for your confirmation bias?"

Nate Jones: I think one of the biggest differences in the way people use models right now is people who are focused with their models can use the model as a mirror that focuses on a particular subject really effectively, and there are people who are less focused, and the mirror becomes a scatterer for them. It scatters their thinking. They become more confused as they use it. And I've seen both.

What I find is interesting about the critical thinking piece, imagine the mirror, and typically it faces you, right? Then it becomes a reflection of yourself. You're absolutely right. There's no critical thinking there. It's just coming back with confirmation.

But if you're smart, you can turn the mirror away from yourself, and you can focus it on something else. And you can come back with a disconfirming or divergent opinion. And I will frequently ask the model to fight with me. I will ask it to disagree. I will ask it to come up with a steel man argument because I think it's much more interesting, and my thinking gets sharpened when I do that.

Michael Krigsman: Actually, I do the same thing. I very often will say to the model, "Be very critical. Don't worry about hurting my feelings. Be sharp."

Nate Jones: That's right. Like an iron sharpens iron vibe is what I like to go for.

Michael Krigsman: Makes sense. What's the best way to, again, craft that prompt?

Crafting Effective Prompts for Complex Tasks

Michael Krigsman: We were trying to accomplish something...

Nate Jones: Shall I show my screen? Would it be helpful to just kind of take a peek at a prompt I wrote?

Michael Krigsman: Sure. Let's do that.

Nate Jones: All right. This is a real prompt that I wrote, and this is an example of me picking something where I feel pretty good about my overall ability to assess quality of response, but I don't have a direct answer to this question. And this was part of a Substack article that I was writing to test O3 Pro.

This is for O3 Pro. You can see it up there. And I'm asking it to step through this analysis with me. Think, you're a senior product leader brought in to design a 12-month AI adoption roadmap for a real firm. First, I could have given the model the choice of firm, and I tried a separate prompt where I gave it that option. That was very interesting. In the end, I wanted something with a company that I was familiar with since I was working on it for testing purposes, used Datadog.

I ask it to do some very specific information gathering. Build the source corpus. I want publicly available information. I want 10-Ks. I want job postings. I want SEC FINRA guidance. And then I want three responses, and I actually specify the word count output, and I specify what I want there, right? There's a strategy memo first, there's a tech stack overview, and there's a regulatory constraints piece.

And what's interesting is, by using the word internal, I am suggesting to the model that the model can craft these inside the chain of thought that it's running behind the scenes without me having to see it.

And then step two, produce. Now, I'm starting to ask for output. I'm starting to ask the model to come back with one document with an executive summary, month-by-month roadmap, a KPI per quarter, anticipated failure modes and mitigations, and an advisor briefing.

And then I'm giving it styling that I want. I want it to be really brutally honest. This is an example of not looking for confirmatory thinking. I do not want tables. I want just bullets if need be, and I would like to get a sense of what shaped your recommendations, right? I want to know where you got some of this thinking from. And then I give it a limit at the top. It can't be more than 7,500 words.

It ran. It thought about it. It was a six, seven-minute, basically a seven-minute run. It chooses Datadog, which I specified. It does a little bit of basic research on Datadog. It builds the source corpus. It gives you a sense of what's in the box there.

Market context, Datadog's edge. It's starting to adopt the persona, it's saying where we lag and talking about other updates in the competitive space, getting into growth goals. And what I love here is that it actually called out, like, the statement by the CEO, by Olivier, around what they're looking for and why, and it's taking that into effect and taking that into account the way, frankly, a good roadmap builder should.

It's looking at client mix, and this is a situation where it's done its own research to come up with that assessment and given a very rough assessment of that. It's looking at the AI aspirations it can find from each of the different C-suite members. It's looking at strategic gaps to close. This is all just in preparation. It hasn't even really started the assignment yet. It's just kind of thinking it through.

It's now going into the current stack in great detail, looking at vendor contracts, security posture. You can see where we're going here. Eventually, it's actually going to get to what it wants to say, and it's actually a very cogent thesis. It talks about how you dominate the data exhaust space and what that means, and then it starts to get into the roadmap piece. But my point here, like, we could go through this, but we don't have time. The point is basically because I structured the prompt carefully, I got exactly what I was looking for back.

Michael Krigsman: What if you're not trying to get a research report, but you're trying to do, say, small research? Find out the answer to some set of questions, for example.

Nate Jones: I did one. I don't know if I have it handy or not. I will see if it's... There may be emerging trends in investing. Yeah, I think it's this one. This is a much shorter report. See, that's the whole thing right there. Like, that, it feels short, comparatively. And it was a very short ask.

Please analyze this economic report, and I'm really interested in... Again, I'm trying to push it. I want to understand emerging trends, and I want to understand areas not commonly discussed, right? I'm looking for it to push beyond, but it's not a very long prompt per se.... and then it jumps right in. It reads all 138 pages.

Exploring AI Models and Their Capabilities

Nate Jones: It gives me a snapshot. It talks about commodities and where they're at globally, mispricing issues driven by LNG, and it basically goes through themes it's seen. And then at the end, it nets it out. Like, this is the big picture assessment for the next six months, this is the macro assessment, and this is how you start to lean in.

And what's interesting is this is much more specific. I ran the same prompt with O3 and O3 Pro, and it was interesting to see the relationship between the two because O3 focused on this bifurcation piece, and O3 Pro had a slightly different perspective. And I know we have, like, eight minutes, we probably don't have time to get into it, but I thought it was fascinating to run a short prompt on both and see the differences.

Memory Features and Prompting Strategies

Michael Krigsman: We have a question from LinkedIn, and this... and I was going to ask something very similar, which is... and is from Laura Finlayson, and she says, "With a prompt like this, which model will do the best at retaining the prompt information for future use? She built and refined her job application prompt in Gemini, but it seems to forget some of the deliverables each time she goes back with a new job description. Claude allows her to save a project, but she doesn't love its writing." And we need to talk about which model is the better model to use and how do you choose, and we only have a few minutes left, but this... You know, we could go on forever here, what do we do?

Nate Jones: There's two things going on there. The first is memory, and ChatGPT really has a killer feature edge with memory right now because they do have... it's not perfect, but they have a memory feature that enables the models to start to actually have a living context of information about other chats you've had inside the same model surface, right?

If it's in ChatGPT, it doesn't matter if you're talking with O3 or 4o or whatever, there's going to be a loose understanding of recent conversations you've had along with some specific facts that the model has remembered about you that you can actually audit and check in the settings section.

That turns out to be very useful for problems like this where you want it to do a repetitive task, and you want it to have a sense that it's done the task before. Even, I still find I want to be precise about each of the assets I need it to process if I need that, and that is one of the reasons why I do tend to favor long prompts that I will keep in a Notion page or keep elsewhere that I can just copy and paste in as needed, because I don't want it to forget anything. I don't want to go to the trouble of writing out that prompt again. I just want it to remember every single thing and do it again.

And I wish that I had an answer for you that these were going to be flexible, deep memory models that just would remember that you did it exactly like this and never forget that step. We're just not there yet, and prompts are part of how we bridge that gap.

Michael Krigsman: One of the problems that I have is I like to try prompts on different models to see the results and compare the results. I think it's, it leads to a lot of creative thinking, and it becomes a real burden and an obstacle because I've interacted for 20 minutes with model A, and now I want to go to model B, and I've got to start all over again.

Nate Jones: There is a way to make that slightly less painful. What I like to do is if I want to transition, I like to ask the model I've been chatting with, "Could you please give me a very detailed summary of our conversation far? And make sure it's as clean and clear as you possibly can make it," and then it will do that, and it will give me a great summary of the conversation.

I then pull that summary into a new model conversation I'm starting and say, "Here's where we're at right now. I would love to continue this conversation with you, and this is what I'm looking for." And it's still a little bit painful, but it's less painful than it would be otherwise.

Choosing the Right AI Model and Subscription Tier

Michael Krigsman: Which model or which company do you gravitate towards the most? You have access to everything. What do you use the most?

Nate Jones: The memory feature is one of the most powerful product features I can remember on ChatGPT because I find that the fact that it remembers something about me drives a recursive behavioral loop for me. I'm very aware of it, right? Like, I've worked in product for a long time. I know what they're doing, but it still works because I find that having a model that remembers me a bit is super, super helpful, and ChatGPT drives a lot of interest for me.

O3 is a daily driver for me. I pick it up by default, but that doesn't stop me from going other places. Like, when I am working on a complex piece of writing, I will use Perplexity. I will use both of the new Claude models, Claude 3 Opus and Claude 3.5 Sonnet. I will sometimes go to Gemini 2.5 Pro, and I almost look at those as, like, additional pieces that I want to go to for specific things.

Claude 3.5 Sonnet is great for writing. I love it. Claude 3 Opus, I love the way it does, like, very thoughtfully considered reading and research. There's something qualitative about it that's very strong there. And even though I end up in ChatGPT a lot, that doesn't stop me from reaching my fingers into the rest of the ecosystem and grabbing what's useful.

Michael Krigsman: I also really like ChatGPT in general. They make it pretty easy, and that memory feature... You know, you start, you start with your prompt, and you feel like it has... It's bizarre to say this. It has, like, this intuition of what makes sense for you.

Nate Jones: Right. And that sense of being recognized, I think is very powerful from a product experience perspective and people respond to it.

Michael Krigsman: Marketers talk about personalization, and usually personalization is, "Well, we've watched their shopping cart and in the past they've bought XYZ product, and we'll recommend the next product that they'll really, really like or the next movie or whatever." But we're talking here about a level of subtlety with personalization that's like light years beyond the typical marketing personalization that we know of.

Nate Jones: That's it really is, and it's going to be super interesting in the next six months or 18 months to see how the product platform evolves for ChatGPT as they build on this memory feature and they add more new models, et cetera. My sense is, especially as they lean into the partnership with Shopify, they're going to lean more into commerce, there are going to be opportunities for personalization with commerce that we've never seen before, but we'll just have to see how that evolves.

Michael Krigsman: Should I pay the $200 a month for ChatGPT Pro? I know you do, but should I? Is it worth it? Or should I just pay the 20 bucks a month that I pay right now?

Nate Jones: That depends on the kind of user that you are. And I have seen the article that came out, I think it was this week, that basically said ChatGPT has done such a phenomenal job pushing value down the chain to the free tier. Why would we pay at all? Because it's impressive, and I think for a lot of average daily use, that is a correct assessment. That is true.

For me, I want... It's not even just that I want it I can test it and show it to people. It's that I want to have no token limits and no usage limits on the smartest models out there because I find myself doing better with my own brain if I have the smartest thinking partner possible.

And we've talked a lot around the edges of what thinking and intelligence means, that's probably a conversation for another day, but from a economics perspective, if I have a thinking partner like that and I push the edges like that and it helps me make one or two better decisions in a given month, the ROI is off the charts at $200 a month. It's very, very easy to do that math. And I think it depends on what you're looking for it to achieve.

Michael Krigsman: I want the best possible results from the thinking, because if I'm spending time...

Nate Jones: Then pay the 200. You'll get O3 Pro and it's qualitatively better in a way that you will notice. It has a resonance to it where like the insights it has stick in my head and I'm like chewing them over in a way that I haven't had with other models, which is super interesting.

Michael Krigsman: Well, I'll have to try it. Well, with that, we are out of time. A huge thank you to Nate Jones. Thank you much for taking your time to be with us today. It's valuable for us when you're here.

Nate Jones: It was such a delight. I enjoyed it a tremendous... Thank you for having me, Michael. I'm glad we got to talk about prompting. I felt like we could have gone on for hours 'cause it's such an interesting topic, but I think we really got to a lot of cool stuff over the course of this 60 minutes together.

Michael Krigsman: And thank you to everybody who watched. Now before you go, subscribe to the CXOTalk newsletter you can be part of our community. We have amazing shows coming up. We have the Chief Technology Officer of AMD coming up and we have all kinds of amazing people. go to cxotalk.com, subscribe to the newsletter, and we'll see you again next time. Thanks much everybody and hope you have a great day.