Salesforce’s Chief Ethical and Humane Use Officer:
How to Build Great AI Use Cases

Paula Goldman: When someone has just been diagnosed with a serious disease, they do not ask AI to schedule that first appointment. Why is that?

Michael Krigsman: Companies must decide which work goes to AI and what stays with people. Paula Goldman is Salesforce's first Chief Ethical and Humane Use Officer and EVP of Product. Her new book is Manage the Machine.

Not just can AI, but should it

Paula Goldman: The book is full of really powerful examples of how AI can make a difference at work, but I want to start maybe with a story. I have a friend who works in a healthcare company. Here's a simple use case: scheduling an appointment. AI is capable of that. It's not actually that complicated for AI to help customers schedule appointments besides just the mechanics of scheduling the appointment. There's something relational. We could go on. So I have another customer, 1-800Accountant, financial services for small businesses across the United States.

Every year at tax time, they're inundated with questions. Has my tax return been filed? AI is terrific at that. It helps me not be frustrated waiting in line for an hour to get an answer to a question. And yet, if someone starts to ask, what about my tax strategy, my financial strategy, they will get escalated to a human. Again, why is that? This becomes a regulated use case. It has certain boundaries that need to be followed, but also it's a great place to build a relationship and deepen customer value.

I mean, I could go on, but those are the types of questions that we need to be asking. Not just can AI do something, but should it?

Michael Krigsman: You've raised two very important and interesting issues: what we should use AI for, and then previously you were just talking about AI being relational. Can you dive into that aspect, AI being relational, and then we can talk about what you should use AI for, where that makes sense?

Paula Goldman: Part of that effort of reaching out to someone that may be in this vulnerable state and getting them set up for their care program is about showing up with empathy, showing up with trust, building that relationship with the customer in this instance. Can AI be empathetic? Yes, actually, it turns out, you know, there are studies of this that under pressure, like even in a medical setting, AI can consistently sort of respond to heightened emotion with reassuring words and so on.

Is that the same thing as a human showing up, you know, with deep empathy, building trust, building a relationship? No, these are not the same things. And I think it's important to know the difference. It's important that, you know, one moment I had when I was researching this book was I was looking at coaching tools, AI coaching tools for managers. And I went in really skeptical because I was like, well, you know, this is just a pure relationship thing.

It's really important that managers and the people that they manage have a deep bond of trust. I found these tools actually to be very valuable in rehearsing hard conversations, thinking before we have these conversations. But the tools don't replace the conversation. That's the point. So we want to be pointing these tools in service of strengthening, not replacing human connection.

Michael Krigsman: How do you decide where to include AI? What makes sense, and how do you think about that?

Paula Goldman: I think we've just gone through this period of great experimentation where you find in most companies they've got a budget, they're sort of throwing AI at a lot of people. They're asking them to play with the tools, to come up with use cases. All of that is deeply important, kind of that bottoms-up innovation. But what we're also starting to find are these hero use cases where people are actually thinking through how actually does work get done? Where are these use cases where AI can be used at scale?

And how do we make sure that we get responsible results from the AI? And it's about in advance understanding, you know, what is it capable of and where does it have limitations? How do we, and, you know, one of the things Salesforce's Agentforce platform has built in is a way to test things in advance and see lots of different variations of answers to questions, even adversarially, to get sort of lots of different places where you might get, you know, 20 different variations of the same question.

Does it answer consistently and according to the same rules? And also, you know, is it going to go say something off topic or embarrassing? All of those controls have to be built in from the start. Same thing with accuracy. Same thing with making sure it works across, you know, different populations, different accents, and so on. So there's a lot of stuff that factors into it. But the core thing actually is fairly simple, which is, is this actually within the capability of AI to answer these questions?

And where might it go wrong? And where do I want it to escalate to a person for their judgment?

Michael Krigsman: So, a core aspect, then, is figuring out the kind of brittle points, would you say? Is that a way of describing it? As you said, where things can go wrong and where you need that human in the loop, or the term you use, the phrase that you use, is human at the helm.

Paula Goldman: And I might say, so to the first part of that question, also, where is it that you don't want AI making judgment calls? So it's one thing if you've seen a question 1,000 times, again, going back to the customer service domain, where's my order? Where is my return? These are things that, you know, companies see over and over and over again, and they're great questions to turn AI on. But here's a novel scenario that a customer experienced. Here's a tricky question that came up in a legal context.

It's not that AI tools can't be helpful in that latter example, help with research, but a human ultimately is accountable for the answer. And that's very, very important. You also asked about this question of human in the loop. So should we dive into that?

Michael Krigsman: Yeah, it's such an important one. And again, you use the phrase human at the helm.

Human at the helm, not the loop

Paula Goldman: The phrase human in the loop. So for people that are working in this AI space, that's a familiar phrase. It actually emerged from the Cold War. Super interesting. So all of a sudden in the Cold War, technology gets very powerful. And in the defense setting, it can start to detect inbound missiles. The question is, What happens? What type, what decisions should technology be able to make and what decisions should humans be accountable for?

Hence this concept, human in the loop, which sort of became colloquially understood once ChatGPT and generative AI came out, kind of got understood, I would argue, maybe in the wrong way. It was like, okay, AI is going to draft this email, but I'm going to edit it and send it, which is fine, but doesn't really work for the world of AI agents. So the whole point of an agent is that it's going to reason through a complex series of steps en route to a goal and then execute them.

And it is not practical for a person to look through 50,000 data points and verify each one. It sort of defeats the point of having an agent. But it is deeply important that when you're working with an agent, that you're able to understand if something's going wrong, that you're able to set the parameters in advance, that anomalies are going to get escalated to you. So that's what we call human at the helm.

It's a different way of thinking about it where ultimately people are still accountable and in control and can fine-tune the right, you know, the tool to get the right results. But you're also harnessing that power of an agent to get stuff done, on your behalf.

Michael Krigsman: There is a very strong pull that happens when people start getting involved with agents, which is they see the power, and they want the agent to do everything. How does an organization, or a product designer, or anybody working with agents, how can we make those distinctions, distinguishing between where we should let the agent run and where we need to be in control, at the helm, as you say?

Paula Goldman: One of the things that's really interesting to me is, you know, again, when generative AI first came out, we started seeing these news stories about hallucinated legal briefs coming up in actual court filings and lawyers getting cited for it. And I thought, oh, that's a novelty. Probably people will figure that out over time. But no, it keeps happening. So I keep looking at, there's a tracker on the internet. You can look this up. And it's now well past 1,000 cases of actual citations by courts. It just keeps happening.

So what does that tell us? It tells us that there are, there's kind of an overtrust and overreliance on these tools. Does that mean that the tools are not fit for purpose? No, it means that there's a principle around human accountability and oversight that we need to be leaning into a little bit more. So it's interesting. I was talking to a friend of mine who's a partner at a very big law firm.

And she said to me, I think about this question that you just asked, what do we trust the AI for? And what do we not? Similar to how I think about what do I trust when I'm bringing on a junior employee? Oh no, she's not conflating people and AI, but she's saying essentially, you know, I give them some rope. I see how they do. I give them enough context. And then I see what the result is. And then based on that, I can expand the scope.

And I think that's a very important thing, not only because it shows, you know, we're starting with stuff that we're confident it's going to get right and growing from there. But it also shows that skill of delegation. The book is called Manage the Machine for a reason.

It's because I think we are entering this era where, in fact, we are all managers of AI agents, and it behooves ourselves to think of ourselves like that because ultimately we are accountable for the work that they are doing on our behalf, and we need to know how to manage these tools to get the right results.

AI has data, humans have context

Michael Krigsman: You know, I have to say, sometimes I think of AI agents as being the most brilliant 10-year-old in the world. They know everything, but they have no judgment.

Paula Goldman: This is so hard because, you know, these words, I think what's really challenging in this macro conversation about AI is that, you know, we want to search for these words that are like, well, this thing is uniquely human. Only people have judgment. Only people have empathy. Empathy or whatnot. But it is actually true. So as these lines are getting very blurry, it is actually true that artificial intelligence is not the same thing as human intelligence.

And the way that an AI, as you just said, an AI model has access to way more data than I could ever possibly go over, possibly in my entire lifetime. That is a huge advantage, if I can harness it for a goal that is, you know, behooves me or behooves my organization. What it doesn't have is the contextual understanding of, I'm sitting here with you exchanging visual cues. I know, how am I feeling in my gut? What is all my experience leading up to this moment?

What is my conviction, my belief about what I stand for, and so on? And I think that, the whole idea is how do we harness AI and human intelligence together to get the best results? And that's what I find over and over again. You know, I think it's really easy in the moment to think that AI is mostly a technical challenge.

But what I was trying to do in Manage the Machine, what I see over and over again talking to our customers, why I spend my days going into the technical details of oversight of AI and the guardrails we put in our products, is because of that, because to get the right results, you want the best of AI and the best of human intelligence, and you want to bring it together.

Michael Krigsman: And here's a question on LinkedIn from Swami Vaidyanathan, and he says, if the human at the helm is accountable for AI strategy but also shaped by AI-generated inputs,

Paula Goldman: Mhm

Michael Krigsman: Who is actually setting the strategy, the human or the AI? And, as AI brings together more information faster than any leader can independently process, how do humans stay at the helm in any meaningful sense?

Paula Goldman: Yes, these are questions that we really need to be paying attention to. What I see smart companies doing and smart individuals doing is that they're first of all, curating the data that they're pointing AI to. So they're not just relying on a generic model. They're saying, look, my company has decades of data on this topic. My company has decades of data about what messages resonate with customers or don't, or what questions people are asking us about our particular products, not just generic products that are out on the market, etc.

And then they're also doing their own form of supervision of the technology so that it's not just a complete deference to it. They're making sure that the brand tone is right, that the level of empathy or the level of spiciness in the brand tone, all of that stuff is right. And they are ultimately putting people accountable for those outcomes. So I guess the answer is it needs to be managed.

But I want to say something else too, which is I do think that there is this problem of overreliance or overtrust or what some people are calling skill decay. I think that's real as well, and that I also see people building in intentional pauses into the way that they are working with AI. Places where they may want a person to go first and think about what it is that they're asking and the contours of the problem before they're delegating it, or after when they're getting a recommendation from AI.

All of these things about preserving human judgment become actually even more important as the technology gets more powerful.

Michael Krigsman: But, as Swami says, if we're mixing our business goals, design goals, what have you, with inputs that are coming from the AI, and we're relying on the AI to then help shape our strategy more

Paula Goldman: Yeah,

Michael Krigsman: and more and more because that AI, you know, it's as you said, it has all the knowledge, all of that data, then who's really at the helm?

AI finds insights, people catch mistakes

Paula Goldman: we are. We have to be. So, I mean, let me turn to a different domain. So one of my favorite chapters in the book was about AI and product innovation. And what I found was so fascinating was there are insights, you know, if you're working on innovation, there are insights that arguably either only AI can find or it's going to make it much, much easier and much cheaper to find these insights.

So a consumer packaging goods company that's creating lip balm, and they're wondering what is it that our customers really care about, and they're able to do listening at scale with AI and find out that it maybe it's not the texture of the lip balm, but it's actually the scent. That's what they care about.

Or a company that was creating golf clubs that was really trying to make it easier for beginners that are getting into golf to find the right angle on their golf club and being able to use vision technology and lots of different forms of data to be able to figure out the exact right design that would make it a bit more forgiving and less prone to error with beginners. These are things that become much faster and much easier with AI. But AI also makes mistakes.

So there was a, Nike was using AI, and it told a story of using AI and famous athletes and designing custom products for that. A sneaker for a marathon runner that had this unique aerodynamics to let him go faster and more buoyant. And he sat down and looked at the sneaker and he noticed that there were kind of holes in the back. And he was like, you know, I know from my lived experience that the sneaker is going to collect dirt and that is not good.

And so let me give you a fix for it. Or that same CPG company seeing that AI was, you know, riffing on ideas and saying, well, if you love your cat, maybe you love its cat litter as well. And so you want to do something like, obviously not a great idea. So that judgment, getting it right requires building in the space for human judgment and accountability and lived experience. You hear people talking about taste-making all the time. That's part of it too.

So yes, I think it's actually deeply important what Swami put, what Jess commented on, which is AI is influencing our thinking. It is. It's influencing our work product. You want to harness the best of it and also anticipate where it may go wrong or what questions and judgments you do not want to delegate to it. And preserve the space for people to think critically about those questions.

Michael Krigsman: We have a bunch of questions that are starting to come in, both on LinkedIn and on Twitter, so I'm glad you guys are asking questions. But, I have to ask you, Paula. You're so deeply involved in all of this. Does the idea of the AIs taking over ever scare you? That's been in the news lately.

Demand AI you can monitor

Paula Goldman: It has been in the news. And I will say, you know, I approach all topics around AI with a great degree of humility, something that I wish was more omnipresent. Because, you know, the truth is no one knows exactly how fast the technology is going to progress and exactly what capabilities are coming next and what the impact of it is. But I will say, you know, when you look at what's going on, a lot of what's happening is we need to be putting the right controls in the technology.

We need to be making sure that the right security parameters are put in. That's something our customers care a ton about. We need to make sure that AI is monitorable. This is a great concern to me. This is why I spend all day working with our product teams and our engineering teams on exactly that, on adversarially testing our products to make sure they don't go off the rails.

And I think it's deeply important that companies do this, but I think it's also important that anyone that's buying or using AI demands that from the companies. And so I believe that if we get those things right, we are in a much better position to handle these emerging risks.

Michael Krigsman: Very briefly, when you work with the product teams, can you describe that for us a little bit?

Paula Goldman: Let me take a step back. So I'm the Chief Ethical and Humane Use Officer at Salesforce. What does that mean? I get that question all the time. Much to the credit, I think, of both our leadership and our employees who were asking questions about advanced technology, especially AI, eight years ago, well before this wave. And they were saying, you know, there's not necessarily a blueprint that we can just cut and paste for how to get this right. There's going to be big, meaty questions we have to ask.

And so I was really fortunate that the company created this office and then said, go figure this out. And while this has evolved over time as the technology has evolved, we work with our product and engineering teams to try to anticipate, you know, how could this technology, for example, go wrong? What might be some of the unintended consequences? How could it go extremely right also? And how can we help with that? And what are the guardrails that are needed?

And so sometimes we will, you know, actually try to break the product before it ships. We'll try to say, Does this work for everyone? If we come to it with an off-the-wall perspective, with a different accent, with whatever it is, are we going to get the right results? If we try to get it to do something it shouldn't, will it comply?

And by that, we are not only making sure the individual products work well, but we are learning and building our collective knowledge about what those techniques are that scale across different types of products?

Michael Krigsman: So, you're trying to, in a sense, break the product, but through an AI lens.

Paula Goldman: Yeah, well, we're trying to see, does this, you know, I think one of the great challenges of this generation of AI is that it is what's called non-deterministic. So we're used to thinking of technology as like it's built for this one thing. You're going to ask question A, it's going to give you response B, over and over and over again. But these AI models are not, they're probabilistic. So you can ask the same question 10 times, you might get 10 different answers. That's great if you're doing a creative writing project.

It's less great if you're a multinational corporation and you need a consistent answer when a question is asked, particularly if that question, for example, has legal implications or whatnot. And so yes, we are trying to, as we are understanding the ever-evolving nature of this technology, of its power and its limitations, we're trying to make sure that those limitations are controlled and that customers are still getting what they need from the products themselves and that there aren't these unanticipated failures.

Michael Krigsman: Let's finally jump to some questions. So, here on X/Twitter from Gus Bekdash, he says, when I sail, I do monitor and adjust as the wind changes, but I know

Paula Goldman: Yeah.

Michael Krigsman: the general plan before I even hit the water.

Paula Goldman: Yes, yes.

Michael Krigsman: AI and automation is the same. How to best do create that overall plan?

Set the agent's boundaries in advance

Paula Goldman: When you're setting up an agent, the first most important thing you do is you say, what topics, what is the goal of this agent? What do you want it to be working on? And then you have this on-off topic control, which then puts certain topics or tasks out of bounds. You're also able to say, you know, here are the certain situations where if they come up, I want you to not work on this or escalate to a person or tell me when something happens.

I would argue that step, it sounds very commonsensical, but it is the most important thing. And then you're able to sort of also give other parameters. Now, what I'm thinking, of course, about these places where there's great product market fit around AI agents, it might be AI and coding, it might be AI and customer service, as we've talked about. And so some of the answers to the questions about how do you plan the journey, are specific to the domains that you're working on.

Not the same thing when you're doing legal research as when you're doing a marketing campaign. But you're thinking in advance of what is the box in which I want this to work? What are the tripwires? So in marketing, it might be, you know, I want to avoid getting into sensitive, possibly controversial topics. I want to make sure this stays on brand voice. I do not want this to go beyond a particular audience or a particular timeframe. You think about that in advance.

It's the same sort of roadmap that you use when you're planning a campaign without AI. You're just making sure that the AI is kind of going through that same checklist.

Michael Krigsman: But then, you have the non-deterministic, that probabilistic aspect because if you're designing a campaign based on a script, for example,

Paula Goldman: Yeah.

Michael Krigsman: it's going to go the same every time, and you know you can predict that outcome very carefully. But,

Paula Goldman: Right.

Michael Krigsman: with AI, there's so much uncertainty in some cases, and it can bring in, it can hallucinate incorrect facts

Paula Goldman: Uh-huh.

Michael Krigsman: and present those facts with the full confidence of someone who absolutely believes their own lies.

Paula Goldman: Yes, that's why generally just throwing an AI model at a problem is the wrong way to approach it, or a consumer AI agent at a business problem may lead you down the wrong path. And why you want to be able to say, for example, about the problem of hallucination and marketing, let's look at marketing for a second. Here is the domain of knowledge I want you to stick to when coming up with marketing messages. I do not want you to go outside this. Here's the domain of knowledge about my product.

Please don't make up stuff about my product that's not true. Here's the domain of knowledge about my brand tone and our visuals. Like, please don't go outside my colors or the aspect ratios or so on. This is why you need those controls that are not probabilistic, that are in fact deterministic, and you get the right results by mixing these things together.

Conversely, you also want to take advantage of that probabilistic system because, you know, when you're interacting with, let's say, in a customer service setting, you and I might ask the question with the same intent with very different wording. And we've all had the experience of being on, you know, a phone with an old-school bot, where it doesn't understand the way we've said something and we're stuck in a loop.

But the beauty of these models is they do understand that, that they are able to respond to the beautiful variety that actually happens in the real world. And the magic is when you marry these probabilistic and deterministic systems together to get the right results.

Michael Krigsman: Let's go to another question. This is on LinkedIn from LogiRoot Runtime AI Governance. I like the fact he's got his product name as his username. And, he's asking about audit trails and any thoughts on being able to prove that policies were actually followed?

Audit trails give you the receipts

Paula Goldman: Audit trails are deeply, deeply important. And again, that's why you don't want to just throw an agent at a, especially if it's a big business problem. You want those controls built in. Audit trails are one of the most important of them because it gives you receipts, tells you exactly what the agent did. It tells you, we're now moving into these multiplayer worlds where lots of people are going to be interacting with an AI agent, both from a teammate perspective but also from a customer perspective.

And you want to know all of those different interactions and what happened. And, you know, what did a person say versus what did AI do? All of that. And then the interesting opportunity is being able to set the controls that tell you exactly what anomalies you're looking for. Because that's a lot of data, you can't go through it manually. And you want to be looking out for certain risk categories in advance.

Michael Krigsman: You guys, keep asking your questions! When else can you ask Paula Goldman from Salesforce pretty much whatever you want? So, keep your questions coming! And you guys always impress me. You're so smart! So, keep your questions coming! So, Arsalan Khan is questioning, defining use cases. He's saying there are times when we don't want a use case. He said, for example, what if we want some agents to have no specific goal but the ability to find inefficiencies across departments?

Paula Goldman: So let's say you give an AI agent, you say, I want you to find efficiencies between departments. Actually, it's a great use case because in big organizations, there tends to be a lot of duplication. There can be a lot of confusion about people, you know, who's in charge of which pieces of the organization and who I spent a lot of time, or in the past I remember spending time, you know, trying to figure out who was it that I needed to talk to in this different part of the organization.

So great AI use case to figure out where there might be opportunity for more efficiency between departments. You probably don't want to let the AI do whatever it wants once it finds that information. You probably also want to make sure that when you set an AI up to do that, that it's respecting privacy protocols, that it's not going into people's employee files and finding personally, you know, sensitive personal medical records. You probably don't want it acting on financial transactions.

Of course there are boundaries that you're going to want to be putting on it. That was really my point is you want to think through before you allow an AI to take action, to write and not just read information, you want to make sure you've thought through what the possible failure modes are.

Michael Krigsman: First off, yes, he is describing a use case, and sometimes your use cases can be broader, but nonetheless, as you just said, you still want guardrails. You don't want it

Paula Goldman: Right.

Michael Krigsman: to say, find all possible data, and if it's not available, dig it up anyways.

Paula Goldman: Right, exactly. And then also, you know, what you certainly don't want it to start making managerial decisions about. You found duplication, what do you do about it? That's a sensitive thing. You want to exercise judgment.

Michael Krigsman: Right. And along the way, so find all possible data, and along the way, if you see anything that needs fixing, just go do it and let me know after lunch.

Paula Goldman: Yeah, but I think the example is a good one because

Michael Krigsman: It

Paula Goldman: it's

Michael Krigsman: is.

Paula Goldman: not, it doesn't have to be in a particular domain or a particular department. It can be a lateral play as well, or a general play.

Michael Krigsman: And Mr. LogiRoot Runtime AI Governance comes back with a follow-up, and he says, is there such a thing in the Salesforce world, or maybe just in general, a version of that receipt that you were describing earlier, that audit trail that a customer can hand to an outside auditor who doesn't trust the software system, and that audit trail must hold up.

Paula Goldman: The answer is yes. So this is all data that if one wants to bring in an auditor, one can. And it's sort of in that format. I think it's interesting because that whole world, I think it is in a very nascent space. And I think there are certain standards and measurements around the success of AI that cut across no matter what your domain is. Like, is it accurate? Does it work for, you know, lots of different populations and so on? But then you start getting into particular measurements.

Or I should also mention security. Like, does it withstand attacks and so on? But then you also start to get in domain-specific measurements, which might be very different, for example, if you're working in sales than if you're working in HR. But I think what actually makes me kind of excited is I think we're going to start to go into much more of those domain-specific ways of measuring success, and that kind of auditing will become more precise because of that.

Michael Krigsman: Let's take another question. This is from Chris Petersen on Twitter. And Chris Petersen says, in terms of deterministic controls, including taste, brand, etc., where are they implemented, inside or outside the LLM, and how do they interact with token spend issues? He's been burned on

Paula Goldman: Yeah,

Michael Krigsman: bad outputs. So, thoughts on that?

Human goes first, then the AI

Paula Goldman: the controls that go into an agentic system for the enterprise in particular bring determinism into it. That's where you sort of say, but they're kind of about setting the box. Don't go outside this brand tone, etc. But I also think, and one of my favorite parts of the book is where we talk about how we actually design the collaboration and the work itself. And how do we ensure that the people that are interacting with it have the right moment to shine and to bring their judgment to bear.

So one of the stories that I tell was IKEA. And how they're trying to innovate on this idea of a couch and the idea of a couch being sort of like this boxy, cushiony thing that we're all used to. But they want to think of a new concept for a couch. And so they're like, okay, let's bring in AI. And surprise, surprise, it keeps reverting to the mean. It keeps coming up with the same cliché of a couch.

And they realize, well, actually, we have to do something that's called front-loading the brief, or something my team is starting to call human goes first, which is we need to challenge our own thinking before we give AI this tool or this task. And we're going to come up with ideas like campfire and gathering space. And that's how they came up with this idea of what they called Couch in a Box, which was this lightweight aluminum frame that could carry with one hand.

It was like 10 pounds, ended up in a museum exhibit. Those principles that are not just what are the controls that are built into technology, but how do we think about our work so that we're still exercising our human skills? They're very energizing to me. I think this is the tip of the spear about how we get the right results. So that was the answer to the first question, which made me forget the second part of his question. Can you remind me?

Michael Krigsman: Oh, he's asking about the tokenomics, the token spend costs that you're experimenting and you're burning your tokens up, which I can definitely relate to.

Paula Goldman: I think we saw this where we had sort of token maxing and then we had companies come back and say, okay, actually, how do I make sure I'm allocating AI spend to the right use cases. And I think what we're seeing is a number of tools that help you actually predict that so that you're not in the wrong space when you are, you know, you're not going over budget and that you're able to sort of predict at scale where it is you're spending money.

Michael Krigsman: These token issues have been killing me. I've been trying to figure out which models, you know, if I can't afford to run Fable all the time, which I can't, what can I delegate down? And trying to get the model to delegate down, it's just like the

Paula Goldman: Oh

Michael Krigsman: whole thing.

Paula Goldman: yeah. Well, because it's also a question of using the right model for the right purpose. You don't always need the most powerful model for something that's fairly routine. And again, it's really exciting, we're starting to see these systems that do that for you so you don't have to think about it all the time, but that you're finding the right model for the right thing. It's also a big part of the equation.

Michael Krigsman: Lisbeth Shaw on Twitter asks, can you give an example of a good use case? The use case can fail or be successful, either way, because both of those are illustrative and helpful.

Paula Goldman: Can the AI do this dependably and reliably? Does it have, you know, are the failure modes taken care of? Does it, is the way I'm measuring this, am I looking at kind of the overall business value of what's happening? So it may be the case, for example, that let's say you're using AI in customer service and you can deflect all your calls and have no one ever talk to a human. That may be great.

Or it might be that actually some of your customers actually want to talk to a person or they're upset that they don't get to. And so you might actually then start increasing your churn rate if you make that decision. So you're looking at it through a holistic business lens, not just the efficiency point, but more broadly, what is the customer value that's happening here? I think that's kind of the piece that's a little bit missing from the conversation right now.

Which is we're so focused on the tool, and not necessarily focused on the overall business outcome in which it's being deployed. And that's, I think, the reckoning, the idea that when we have the tools that are well calibrated, that we can better look at the big picture.

Michael Krigsman: Well, it's been so complicated to date, and I know Salesforce is working on a lot to simplify it, but this whole issue of AI agents has been so complicated with harnesses and the different models and so forth that it sort of draws you into the technology aspects, as opposed to making the technology easy, and therefore, you can focus simply on defining what's your use case and getting the business outcome, realizing the business outcome.

AI is simpler than it seems

Paula Goldman: We're not just using AI in one domain. It would be very simple if our whole conversation was AI and sales, and we're just, that's it. That's all we're talking about. But all of a sudden, because these models are very general, they can be used across almost any knowledge task. And so because of that, and because they're so powerful, there is a lot of conversation right now about how do we get this right? What are the tools good for? How quickly are they progressing? But I do think that will settle.

That is to say, you know, it's noteworthy that you and I are not having a conversation right now about the internet. Why are we not having a conversation about the internet? 15 years ago, 25 years ago, we might have been having a whole conversation about it and how novel it is, but we've sort of metabolized it. And it's not novel. We know how it works. And this is the moment of confusion that we're in right now with AI.

It is why I think there's, you know, a lot of excitement and also a lot of anxiety out there. But while I realize it's not necessarily easy to say, here are the, you know, the two things you need to do, and that's it. I do think it's actually not that complicated. We're just talking about good delegation, thinking through what we actually want the tool to do on our behalf, and making sure that it's not going outside of those parameters.

Michael Krigsman: Your example of the internet is so right. I remember 20 years ago, a company like Salesforce was talking with CIOs to convince them that the cloud is safe, 20, 25 years ago.

Paula Goldman: Yeah.

Michael Krigsman: And now, we don't

Paula Goldman: Yeah.

Michael Krigsman: question that at

Paula Goldman: Right.

Michael Krigsman: all. But, I think you're living in the future where you have metabolized this, to use your phrase,

Paula Goldman: Yeah,

Michael Krigsman: and the rest of us are still catching up and trying to sort through the noise from the results and sort of navigate through the static to achieve our goals. And so, it's confusing.

Paula Goldman: I get that. And honestly, you know, part of the reason that I started writing this book, you could ask, why would you write a book about AI? It's changing so quickly. You know, it's a very noisy space. And honestly, the reason that I started writing it was because I've been working on AI ethics for a while. My friends, you know, would ask me polite questions over dinner. Then all of a sudden ChatGPT came out, 2022, end of 2022. I started getting calls from these really accomplished people.

And there was a thread of, there was some anxiety in those calls. It was, you know, is this going to make us obsolete? And how do we use this tool responsibly? And while at first the fact that people were paying attention to AI was in some way gratifying, you know, I started to feel unsettled because it felt to me like there was this narrative of this godlike force that's totally uncontrollable and you couldn't possibly understand it unless you're one of like 200 researchers working in an AI lab.

And the whole point of me writing this book was to say, actually, no, you can understand it. It's really imperative that you do. It's not that complicated. It is about applying some of these common sense principles to how you leverage the technology, just as you would if you were spinning up a new team around something. And that confidence in our own ability to manage these tools is actually how we get the right results.

Michael Krigsman: Arsalan Khan has a question about AI ethics. And so, let me ask my question, and then I'll elaborate with his question, and we can kind of bundle them together because they're related. So, first off, Arsalan, thank you again for an interesting topic. So, my question is, what are the ethical and governance principles that you follow when designing

Paula Goldman: Yeah.

Michael Krigsman: this human-AI collaboration and the use cases. But Arsalan then says, if every organization is creating their own ethics about AI, wouldn't that create a hodgepodge of varying ethics? And, what role should a regulator, or third party, or government play, in all of this?

Society sets the floor, companies experiment

Paula Goldman: Number one, AI in many ways, there's lots of stuff that's new and powerful about this round of AI. Also, there's a lot of rules that already apply to it because, you know, like nondiscrimination rules, they apply whether you're using AI or not. AI, you know, there are new rules that apply around, for example, transparency that people need to know that they're interacting with an AI agent if that's the case, or an AI system.

And so I think that there are a basic set of things that cut across that are very important. Some are old, some are new. And then you don't actually want, you want those basic guardrails set kind of at a societal level. And then you actually do want experimentation and companies to be able to, and/or individuals to be able to then experiment and play with what actually meets their needs, within those parameters. And that's, I think, what we're seeing.

So I don't want to be in the position to tell my customer, I want to make sure that my technology, the technology that we're shipping, is trustworthy, that it's not going to go off the rails, that it's not going to, you know, violate laws and so on. And then I want my customers to be able to paint within those lines and to be able to ensure that they trust the technology and so on. So that is how I would respond to the second question.

The first question, remind me, was how do we set ethical guardrails for technology?

Michael Krigsman: Yes. Do you have a framework for how you think about the ethics in relation to AI?

Paula Goldman: We, like many companies, had a set of principles, have a set of principles around trustworthy AI that predate generative and agentic technology. And then we sort of had to update them a little bit for the specifics of AI, or this wave of AI. And so, for example, accuracy was always on our list, but because of this question of non-determinism and hallucinations, it moved to the front of our list because you need to be able to trust that you're getting accurate results from AI. But there are other things as well.

One of them is around transparency. We talked about that as a principle. We talked, empowerment remains one of our principles. It's like you want people that are using the technology to be able to understand it and get the right results from it and to set the right controls around it. So these principles that cut across the guardrails keep evolving, but the principles stay constant over time.

Michael Krigsman: Can you give us some advice for CEOs of companies when it comes to dealing with agents and this uncertain future and the implications of AI? I mean, on the talent pipeline? I mean, this is so broad, but what advice do you have for CEOs?

AI needs a clear accountability structure

Paula Goldman: One, I would say, think about the accountability structure that you have around AI. You wouldn't open a new store or hire a new person and give them no one to report to, no one in charge of the store.

What's happening with AI, I see, are these matrixed accountability structures where you might have an IT team that's responsible for basic standards around the technology itself, and then people that are closer to where the work is getting done in marketing, in HR, really pouring that knowledge into the use cases and setting the standards there. The second thing I would say is really give employees, employees deserve a seat at the table in shaping how AI is going to shape their work. Why is that?

Story after story that I come across when I am out in the field talking to customers about how people that are closest to the work actually understand how AI can help improve it and where some of the tripwires are. You know, so I was telling the story of a friend who went to go visit a factory, an auto factory where they were training workers to deal with robotics. And the instructors were deliberately people who were on, had been on the shop room floor.

And they came up with these apps around some of the interesting things they would see on the shop room floor and some of the failures and how you deal with some of those failure modes. That knowledge is really important. And too often, AI is something that is perceived as, you know, coming down from on high. But really, your people remain your essential advantage, and you want to be empowering them to innovate with the technology.

Michael Krigsman: Great advice! You know, we've had other senior business leaders on this show give the strong advice that CEOs gain a greater understanding of the technology. Obviously, we don't want them to

Paula Goldman: Yes.

Michael Krigsman: become programmers, but you're dealing with a technology, and it's a profound technology, so you

Paula Goldman: Mhm.

Michael Krigsman: need to understand. You can't just say, oh, let them deal with it. You have to understand it.

Paula Goldman: I agree with that. I think that applies to all of us.

Michael Krigsman: Absolutely. This is from Erica S., who says she doesn't know the details here, but she'd guess that deterministic controls sit mostly outside the guardrails, the validation, the retries. And she says, token spend is the real problem. You generate, reject, and then pay again. Any thoughts on that one?

Paula Goldman: When you're concentrating on use cases at scale, which is where token spend comes in, like for example, Salesforce, we use AI a lot in coding. Our engineers use AI a lot. When you use it at scale, you start to get a sense of where most of the money is going and what types of tasks they're going. And you can make those allocation decisions much more accurately because you have that data. And because going back to an earlier question, you have an audit trail built in.

So you're able to say, Which are these places? What, not only where am I getting the right results where it's worth investing in, but also how can I then fine-tune which models I'm using for which purposes so that I'm not overspending?

Michael Krigsman: Paula Goldman is Salesforce's first Chief Ethical and Humane Use Officer, and she's the EVP of Product at Salesforce. Paula, thank you so much for taking time to be with us. I'm very grateful to you.

Paula Goldman: Oh, this is really fun. Thanks for having me.

Michael Krigsman: And everybody, thank you for watching, and especially you folks who asked such great questions. Before you go, subscribe to the CXOTalk newsletter so we can invite you to upcoming shows. We have incredible shows that are coming up. So, go to CXOTalk.com right now.

Paula Goldman: Mhm

Michael Krigsman: Check it out and subscribe, and we'll see you again next time. Thanks, everybody!