AI Snake Oil Exposed:
Princeton Researcher Busts AI Hype

Michael Krigsman: AI is a mess of truths, half-truths, lies, and genius. So, what do we do? Today, our guest, on episode 867 of CXO Talk, just wrote a book on this topic. It's called AI Snake Oil. Arvind Narayanan is a computer science researcher and professor at Princeton University, where he's also the director of Princeton's Center for Information Technology Policy.

Arvind, I've been waiting to hear about AI Snake Oil.

Arvind Narayanan: AI snake oil is AI that does not, and probably cannot, work as advertised. There is a lot of genuine progress in AI, but many AI products claim to do things that, as far as we know, are simply not achievable.

Things like hiring automation software that claims to use videos of a job candidate talking and analyzing not even what they're saying about their job qualifications, but their body language, facial expressions, that sort of thing in order to determine their fit for a position. Much of this is going on in parallel with genuine AI advances. And we thought, my co-authors, Aayush Kapoor and I, that many people need a better way to tell apart what's real from what's not, and that's why we wrote this book.

Michael Krigsman: You're not talking about deception, or are you? Or are you talking about technology limitations? Give us a sense of that.

Arvind Narayanan: It's a mix of those things. In some rare cases, I think there is deception. One example is a company, DoNotPay, that got into trouble with the Federal Trade Commission for claiming to have built a robot lawyer.

There was no robot lawyer, and they further claimed that any attorney who wore an earpiece to the Supreme Court and used their robot lawyer to argue a case would be paid $1 million as a way for the company to show how amazing their technology was. I mean, it had to have been at least somewhat deceptive. They had to have known that it was a publicity stunt, because electronic devices are not even allowed in the Supreme Court. They knew they wouldn't be able to go through with this.

And to this day, a couple of years after that stunt, there is no evidence that they have built something like this. You can look at the Federal Trade Commission's complaint. It's full of juicy details of how they made things up. So, that's one category, but much more common is you take a kernel of something that works well, but then hype it up because that's what you think clients want to hear, that's what investors want to invest in.

In many cases, it's like centuries-old statistics, you know, regression, being rebranded as AI, and while it's true that these models can pick up some patterns in the data, they're not perfect oracles, and when you sell hiring automation software and make claims about video analysis, that's when it crosses the line for me.

Michael Krigsman: I'm in the enterprise software business, and every enterprise software product is now based on AI. Well, it's not really based on AI, but it may have some AI features, but you're not you're not talking about that, are you?

Arvind Narayanan: Not mainly. There is certainly a lot of that going on as well. So the things we're really interested in, in the book, and what we want to communicate to people, is some categories of AI are probably not going to work at all. It's not even worth buying the product.

In other cases, there is so much hype that you really have to evaluate for yourself how well the AI product is going to work for you. You can't just go based on the marketing materials, and it's not a matter of simple exaggeration. AI performance can be very dependent on our specific data set, the specific queries that we might want to put into it that in many cases, customers end up very disappointed and find that AI doesn't do what they thought it would do.

And then there's another category where AI might really work well, and this is less about the enterprise setting and more for society broadly, and we might want to push back on certain applications of AI precisely because it works so well. And a good example of this is facial recognition, and to me while there are some concerns about facial recognition making mistakes, the primary ethical concern is that it can be used for mass surveillance, especially by authoritarian governments. We know it has been used this way both in China and Russia, and it's dangerous because it works so well, you can pick out a face in a large crowd for instance. So, those are some things to be aware of. Those are reasons why we might not want to deploy AI or might want to push back against AI deployments.

Michael Krigsman: In your book, you make a distinction between predictive AI and generative AI. Can you tell us about that as a kind of foundation for our further discussion?

Arvind Narayanan: Absolutely, and this is where a lot of confusion around AI comes from. AI is of course not one single technology or application. It's an umbrella term for many kind of loosely related technologies and applications. So, ChatGPT on the one hand and generative AI overall is very different from "quote-unquote" AI that banks might use for assessing credit risk for instance.

So, the latter is a type of predictive AI. It's making predictions about the future, particularly a person's future in order to make decisions about them. Is this person going to pay back a loan? Is this person going to perform well at this job if we hire them? Is this person in the criminal justice system going to commit a crime?

And on the basis of those predictions, enormously consequential decisions might be made about those people's lives, such as pre-trial detention, that is denying bail, which might be months or years until your trial date. And so this is the kind of area where we're very skeptical, and the evidence shows that these tools are, you know, slightly better than random number generators, and that's very different from generative AI, which is about producing patterns as opposed to making predictions about what people might do in the future.

Michael Krigsman: So you're not throwing predictive AI and predictive analytics under the bus, because there are many also very positive examples of these technologies. So again, how do you distinguish what works and what is what you call AI snake oil?

Arvind Narayanan: Let's talk about some positive examples. If we want to use statistics and analytics in the hiring process, there are many great ways to do it. We can look at the data that we have about what helps people perform well versus not perform well and we might discover things about the applicants, but we might also discover things about our own workplace. Maybe a particular manager and their style of managing is a poor fit for a certain kind of candidate.

And in fact, those kinds of insights can be much more actionable, because you can change something in your organization that might benefit your hiring for many years to come as opposed to making decisions about individual candidates. So that's one thing, right? Analytics is different from automated decision-making. That's the lesson there. In some cases, I talked about banks, you know, predicting credit risk. Yes, those technologies don't work perfectly, but still banks do have to make those determinations despite those limitations. If they didn't assess risk at all, in lending, they would go out of business. So that makes sense to me as well, again, although there are reasons to be cautious.

Here's another example. The IRS might do audits, and they might use AI techniques to identify who might be at higher risk of committing tax fraud. So I think that's a well-motivated application as well. Here, the difference is that you're not making a life changing decision about someone. Yes, an audit is annoying for a lot of people, but ultimately, you only get punished, you only get fined if you have actually cheated on your taxes, so the moral calculus there is different as well. So both because of technical reasons and because of moral reasons, I think some applications of predictive AI are much more justifiable than other ones.

Michael Krigsman: Certain applications will have more profound impacts on the people involved, and they have no say. The targets have no say in those outcomes, and very often, unfortunately, there's little to no recourse as well when there's an automated decision.

Arvind Narayanan: I think a problem with a lot of our existing systems already, and the presence of AI in the mix, perhaps exacerbates that problem. It's already the case that when a job applicant is rejected, they get no feedback that can help them improve for future applications. And so, we talk about this a lot in the book. A lot of the time what AI does, is it exposes something that's already broken about some of our processes or organizations or institutions.

And I think those are opportunities to kind of use AI as a mirror to see maybe what's not working well and fix it at a deeper level as opposed to figuring out how to tinker with the technology itself.

Michael Krigsman: Subscribe to our newsletter. Go to cxotalk.com. Check out our upcoming shows. We have great, great shows coming up. We have some questions that are coming in on both Twitter and LinkedIn. So let's jump there, and the first question is from Arselan Khan on Twitter. He's a regular listener. He asks great questions, and he says this, "As AI is pushed more and more directly and indirectly towards consumers, should consumers have a way to opt out?" It's a policy question.

Arvind Narayanan: It depends on exactly what the application is. Opt-outs can be in various ways. So, when we're talking about generative AI for instance, many artists and other kinds of creators want to opt out of their data being used for training AI, and many creative people say that opt-outs actually don't go far enough, and there's a reason in fact to argue that there needs to be some sort of collective bargaining.

And opt-out is, if you think about it, in terms of an economic negotiation, is between one individual and one trillion-dollar company. The power asymmetry is so huge that it's you know, it ends up not very much in the artist's favor because opting out really doesn't benefit them to a significant degree because there are millions of other artists whose information is also used, and so it doesn't really hurt the AI model, and so the AI company has no reason to sort of be afraid of artists opting out.

And so that process doesn't really work well. So yes, that's one case where opt-outs, or maybe even something stronger, should be required. or you can think about recommendation algorithms on social media which have been blamed for a lot of things. The evidence there is very much still in progress, but even based on what we know so far, many people want choice over the algorithm, or being able to revert to a chronological feed. So those are a couple of examples where opt outs can be a very important intervention.

Michael Krigsman: There are profound policy implications here, because it's one thing if Netflix recommends the wrong movie, or Amazon recommends a product that you're not interested in. Something altogether different, if you're applying for a loan and the algorithm incorrectly identifies you as being a poor credit risk, and you can't buy the house, or the criminal justice system identifies you as being having committed a crime when in fact, that's not the case.

Arvind Narayanan: The answer to this should be contestability, which you already brought up. An ability to appeal a decision, an ability to have recourse. And when we look at you know, that that seems so obvious to say, why don't these systems already have this? And I think the answer to that is nuanced.

I think what is going on in a lot of cases is AI vendors are coming to these decision makers that are usually in a financial crunch, and want to save money on their decision-making processes and they're saying, "Look, you can automate your process. You can save a lot of money," but then, when mistakes inevitably happen, when people complain, the AI vendors are going to retreat to the fine print that says a human always needs to be in the loop. And so if you always have a human in the loop, are you really realizing the efficiency gains?

I do think there are ways to improve efficiency while also preserving the benefits of having human processes, but that actually requires innovation. So, in many existing systems, it's kind of one or the other. There's no way to have a sweet spot in the middle, but I do think we can get there.

Michael Krigsman: And of course, we all submit to the shrink-wrap license agreements that we, or the click-through agreements that we use. For example, we're streaming to LinkedIn, but how many of us actually read those agreements? they do provide outs for the technology vendors.

We have another question from LinkedIn, and this is from Edith Joachimpillai. she says, "Can you describe some of the reasons why people are so susceptible to this AI snake oil now? And what the average person can do to be better informed to avoid AI snake oil?" And she continues, "Also, we don't want people to be too fearful of AI adoption either." So, how do we encourage folks to adopt positive applications, and I'll just add, and therefore protect themselves at the same time?

Arvind Narayanan: I think it depends on the kind of AI. Let's talk about generative AI a little bit, which we haven't talked about much so far. Generative AI is a rapidly advancing technology, while at the same time there's also a lot of snake oil. For instance, AI for detecting when students are cheating on their homework by using AI. Unfortunately, as far as I have seen, those detection tools don't work. So, a lot of generative AI is very useful, so how can we encourage consumers to explore those while staying clear of the ones that don't work?

So what I'll say to this, is I think because of the way that technologists, tech companies, are portrayed in the media, there's a lot of deference to whatever tech CEOs are saying today. And I think we should avoid deferring to them so much. I think they are of course, very smart, no question there, but they also have vested interests to say whatever it is that's going to make them more money.

A good example of this is until two months ago virtually every tech CEO was saying they're going to scale these models to be bigger and bigger, smarter and smarter, perhaps all the way to AGI. And then around the time of the Neurips conference in November, the script suddenly flipped. Apparently now model scaling is over, and now they're doing something called inference scaling or test time compute scaling. So I'm not saying they were right before or they're right now, but the point is if without any new evidence you can suddenly, you know, completely flip your script, because now it's in your financial interest to do so, now they're, you know, some of them are saying this, in my view because they want to raise money for other purposes.

It clearly shows that we can't really trust their forecasts, whatever they claim about their own products etc. And so I think we should evaluate this independently. The good news with generative AI is that in my experience, a few hours of playing with these products will give you much better information about how it's going to work for your use case as opposed to believing what companies claim, or media reports, or frankly, in many cases, even peer-reviewed academic literature. So I would encourage you to trust your own evaluations. You know, maybe not to the exclusion of any other way of evaluating AI, but use that as the primary way in which you decide if generative AI products are useful for you. So I would encourage a culture of experimentation as well as a culture of having a little bit of skepticism about what is coming down to us from companies, from media, from researchers.

Michael Krigsman: Large companies, even small companies have a vested interest. I mean, just look at Facebook's recent about face in terms of the how they evaluate potential fake news. Facebook wants to sell users, attract users, sell their service, give their service away. So of course, there's a vested interest here.

Arvind Narayanan: That's absolutely right, and in fact, we have a whole chapter in our AI Snake Oil book talking about AI that's used on social media for content moderation, you know, deciding what content stays on and what doesn't, as well as for recommendation algorithms. And one of the things we concluded was that a big challenge that social media companies face is that they don't have the legitimacy in the eyes of the public to be the arbiters of the public square of all of the conversations that we're having.

And I think that's one reason why whatever decision they make is subject to fierce pushback from, you know, one side or the other, or perhaps both sides. That's been a consistent feature over the last more than a decade at this point. And adding AI to the mix. Yes, it might save them some labor in human content moderation, but ultimately it doesn't help that core legitimacy problem. In fact, if anything it makes it worse because people are treating AI as well with a lot of skepticism and when it's deployed by companies who are not very trusted, that only makes things worse.

And I think what we're seeing now is very consistent with that. Because of all of the backlash around the inevitable mistakes that AI makes, the role of AI has been dialed down, but also there's a lot of blowing whichever going whichever way the political winds blow. And perhaps, you know, social media companies are doomed to doing that sort of thing as opposed to making principled decisions because they're not trusted by the public.

Michael Krigsman: Folks, if you're watching, this would be an excellent time to subscribe to the CXO Talk newsletter. Just go to cxotalk.com and we will keep you up-to-date on the amazing shows that we have coming up. And right now, you can ask your questions.

Take advantage of this opportunity. If you're watching on Twitter, X, use the hashtag CXO Talk. If you're watching on LinkedIn, just pop your question into the chat. And truly, when else will you have the chance to ask Arvind Narayanan on pretty much whatever you want. So, take advantage of it. And we have a question from Greg Walters on LinkedIn and he says, "Because AI can see so much more than IMDb, for instance, don't you see AI having the ability to de-anonymize all data? I think the issue here is the extent to which data is anonymized, de-anonymized, and what that set of issues is."

Arvind Narayanan: There are definitely ways to use AI to try to breach people's privacy, to try to link their data across one website or app to another, one dataset to another, certainly. And a lot of my research when I was in graduate school, you know, 15 plus years ago was on exactly this topic, coming up with algorithms to show that there are these big privacy vulnerabilities.

Ultimately, the conclusion that I came to is that part of the solution is technical, but I think primarily the solution to privacy concerns, some of these privacy concerns, has to be social, economic, and legal. And yes, it's possible that AI is going to make it easier for bad actors to create these privacy violations, but generally a lot of the people who are doing this are commercially motivated. They're doing it because you know, they want to sell you more products or something like that.

It's rarely the case that it's a foreign nation-state adversary that is coming after a specific person's data and information. There are some of those people who are at higher risk as well, but for the majority of people it's a matter of having regulation out there that sets some ground rules, and removes the incentives for these deeply privacy violating business models. I do think we've made a lot of progress on that. There's no comprehensive federal privacy legislation in the US, but at the state level a lot has happened, and of course in the EU, there is there are things like the GDPR, which many people are concerned about with respect to its effect on innovation. So, those are two different approaches that these two regions are following, but overall, I would say there has been progress on the privacy front.

Michael Krigsman: And let me just mention, for folks that are interested in this international angle, we have a show coming up, I believe in April, with two members of the House of Lords who have been very focused on online harms, and the topic of AI is going to center very prominently in our discussion. So, if you're interested in that, check out the CXO Talk site and you'll find the, you'll find the date.

We have a really interesting question from Arselan Khan, and I think it follows from the earlier one from Edith and Arselan says, "How can normal people identify AI snake oil? And who should create the moral and ethical authorities? Is it the government, the profit seeking organizations, nonprofits, the public square in some way?" So it's two questions: How do we identify snake oil, and who creates the moral authority around, what is snake oil? What's acceptable, what's not?

Arvind Narayanan: Breaking down different types of AI, when it comes to generative AI, I think each of us is the expert on whether it works for us or not, and you know whenever I see on Twitter right, or LinkedIn or anywhere, just an everyday person who might not be an expert on AI, but is an expert in their own domain. They're a lawyer or a doctor or whatever it is, trying out some of these claimed AI products for their own purposes and posts their experience. Either it worked well, and here's here are the ways that they're productively using it, or it kind of works and here are some pitfalls, or it didn't work at all, and here are the reasons why it's very different from the way it was portrayed in marketing. I think those are excellent kind of evidence. I often find myself going to those kinds of experience reports. Again, personally I find that more valuable than either marketing materials or in many cases peer reviewed studies which have to kind of uniformly apply AI across every user, and so miss a lot of the nuances of the ways in which your AI application might be different from anyone else in another industry or even the same industry or different company etc etc etc.

When it comes to predictive AI, I think it's a lot harder. It can be harder for everyday people to do this, not because of a lack of expertise. That's not the issue at all, but because predictive AI is not a consumer technology, right? That's the big difference here. It's deployed by companies. You know, it might be an HR department, it could be the criminal justice system. So, we don't have access, right? And so I think one step that's really important there, is through regulation or by clients of AI tools negotiating with the vendors to insist that we have rigorous data on how these tools were evaluated that goes beyond a simple headline number like 97% accuracy and really digging into what data set was used, in what context was it deployed, and asking to see that data and judging for oneself.

In terms of who has the moral authority here, I mean I would say in that sense, AI is no different than the way we regulate really anything else. I mean it's regulation is something ultimately that flows from the collective will of the public, and that is channeled through various ways, the media for instance, in exposing certain problems, nonprofits in advocating for certain things, companies of course in leading by example, or in various other ways, and then ultimately regulators coming in and creating or enforcing policy on the basis of all of this.

Michael Krigsman: We have some interesting questions now coming up about industries. So let's jump to those. And the first one, I'm just taking these in order. Greg Walters asks about snake oil inside the "quote" education complex.

Arvind Narayanan: Yes, there's a lot of that. I do think there is a lot of potential as well, but even if we go back before the recent wave of excitement over AI, I think EdTech has been, you know, a graveyard of failed products. There are so, so many and we talk about some of them in our book and we have an article on our newsletter also called AI Snake Oil, which is titled something like 18 Pitfalls in AI Journalism and we go through how certain EdTech products were portrayed in the press including in the New York Times, and how a lot of the claims made by them were uncritically reported, and looking back a couple of years after those claims were made, many of those products are now dead.

So why does this keep happening? First of all, yes, there is potential. There are many products that are valuable. I mean as an educator, we use certainly a lot of tech products, and increasingly some of them do have AI, at least simple forms of AI. For instance, we use Gradescope for grading, which uses some degree of AI to cluster different student responses to make it more efficient for the grader to grade. It's not AI doing the grading, but it can help the grader. So those are examples of small but nevertheless useful ways in which you can do it. And there's a lot of research coming out around AI tutoring as well and again done right, I think that can be very useful. When I'm in learning mode, I often use AI myself. There are pitfalls but I do find it a useful technology.

All of that said, I think the fundamental structural reason perhaps why there is so much snake oil in this space, is that I think the key bottleneck to better education is not better technology, but kind of the social preconditions of learning. That's been my experience at least. For instance, 10 years ago, there was the assumption that oh all of these famous professors are putting their video lectures online and they're on Coursera or other online sources and so colleges are just going to become obsolete. You can hear from all of these experts directly on the internet.

Of course, it didn't happen. It turns out that the value of a college education is not the information that you get. That is of course freely available online, but again creating the social conditions in which learning happens. The motivation, the commitments that you get and the one on one interactions, all those sorts of things. And so when the bottlenecks are those kinds of social problems, putting too much trust in technology is almost inevitably bound to fail.

Michael Krigsman: So to what extent are the problems associated with predictive AI, a function of say immature technology, immature algorithms, immature datasets, versus vested interest and the bias that's created when somebody has a pre-determined goal in mind, whether it's to sell products, or gain eyeballs?

Arvind Narayanan: It's both. I think it's more of the latter, so the the incentives and those kinds of factors. But let's talk about the technology a little bit. I do think the technology is somewhat of a limitation, but not in the way that one might assume. It's not that improving these models is going to improve anything. That's not where the limitations come from.

The problem is that ultimately as long as you're taking a supervised machine learning approach, as long as you're just using a model that is built to match patterns, no matter how good it is at matching patterns, right? And using that to make decisions, which are ultimately about causal reasoning, right? So, you know, if we hire this person, how will that affect their performance? That's the causal question we're trying to ask, and similarly in healthcare etc. And so what we're doing in deploying AI, is we're reducing all of these causal questions to pure prediction questions. And that's a way in which technology today is fundamentally limited. There are lots of researchers working on this, including some of my colleagues here at Princeton, and there are ways in which we can integrate causal inference techniques into machine learning, and that can be a way forward. It's not going to be a panacea, but I do think it can result in better decision-making systems based on data than the ones we have today.

With all of that said, again a lot of the fundamental problems are not about technology. One example we discussed in the book, is software for predicting which students in college might be at risk of dropping out. Now, this is, I'm not calling this snake oil, it's well intentioned, so you can go help those students and help them do better, spend more resources on them, and I haven't looked into exactly how accurate this is, but I wouldn't be surprised if it's accurate enough to at least be useful. It doesn't have to be perfectly accurate.

The problem was how certain colleges were using this. There was one investigative report of a college, and we're talking about you know, the long tail of colleges which are under a lot of financial pressure these days. And the way they were using it is they were trying to use it right at the beginning of the semester when students had enrolled as freshmen in order to predict who might drop out in order to then encourage them to quit the program preemptively. So why were they doing this in such a student hostile way? It's because they figured that if the students quit the program very soon, they wouldn't count against their enrollment numbers and therefore their graduation rates would look better. Right? And so if you're, you know, even if the product is built with good intentions in mind, even if it works really well, if you're using it in a way that ultimately harms your decision subjects instead of helps them, that's not a technology problem.

Michael Krigsman: Yeah, you have the adaptation of the technology, the use of the technology and people put their finger on the scale and as a result you can end up with unintended consequences.

Arvind Narayanan: That's right. I mean, sometimes they're intended consequences but yeah very often they're unintended consequences.

Michael Krigsman: We have another really interesting question now from LinkedIn, from Shail Khiyara. He asks, "What role could academia play in holding the AI industry accountable for exaggerated claims and unfair practices?"

Arvind Narayanan: Think about the Pharma industry where you know, there have been a lot of exaggerated claims and unfair practices, and then you have the medical community, practicing doctors as well as academic medical researchers and there is a strong sense that while a lot of medical research is ultimately towards the goal of improving our medications, doctors are not automatically aligned with the industry. In fact, there are strong rules around conflicts of interest and independence is fiercely valued. There's a lot of focus on where funding comes from.

Now, in computer science academia, in contrast, of course, academia is much wider than that, but in computer science among the people building AI, there has been no such recognition. Historically, the academic field of computer science grew up as really as a way to help the industry, as a way to build more tech you know, prototype ideas and build proof of concept products that can then be adopted and commercialized by the industry. There is no wall and in fact, going back and forth between the industry and academia is highly valued. People have multiple affiliations. They perceive no conflict between them.

I'm not saying this is necessarily bad. I think there are reasons why it is this way, but I do think we need some strong subset of the academic computer science community whose identity is less about helping the industry and more about being a counterweight to the industry. To, you know, fact-check the claims that are coming from the industry. And historically, computer science has not had that culture. Now it is starting to develop and I'm proud to be part of a very small minority of computer scientists who play this role. Academia is of course broader. There are so many people in the humanities for instance, who are very critical of the tech industry.

I think that's really good and I think that kind of those critical voices can be much more effective if they're more informed by the technology. I think. I mean I think they're doing their best, but it can be even more productive if they're more technically informed. That can be in collaboration with with computer scientists for instance. So that's another way in which academia can be more effective at playing this role.

Michael Krigsman: Academia obviously has expertise, and one hopes that academics also have a more neutral perspective than the software vendors will have.

Arvind Narayanan: That's right. And again in most of computer science, that has not been the case so far. And in many other cases outside computer science as well, the people using machine learning to make advances in whatever fields, whether it's chemistry, political science, medicine etc, are often very prone to hype and exaggeration. They're not in it to make money, but excuse me, but you know, hyped papers tend to be more successful in the marketplace of ideas. So, unfortunately, a lot of those same bad incentives exist there as well. And I do think academia needs to police itself so that we can be, we can start to deserve our reputation for being more neutral and responsible, which I don't think we currently deserve.

Michael Krigsman: This goes to the heart of human nature and really has nothing to do with AI specifically, but it has to do with human ambition, human goals, and so forth. And as you said earlier, AI is written by people and so AI is going to suffer the same slings and arrows as anything else.

Arvind Narayanan: I think that's a fair observation. Yeah.

Michael Krigsman: We have an interesting question. I'm going to jump to Paula Rooney, a senior writer at CIO Magazine, and let's see if we can give her an answer that she can use in an article. Paula asks, "What are the top three misunderstandings or falsehoods spread by AI vendors that experienced technologists such as CIOs believe?"

Arvind Narayanan: I think there are a lot of things that CIOs might believe because they want to believe, even though, you know, theoretically, they should know better. And so one of those is claiming that this is, this is AI where it's AI agents, which is of course, the buzzword de jour, when it's in fact, traditional automation.

So one trend I've observed in the industry is a lot of companies, you know after ChatGPT came out, there was obviously a huge fear of missing out and so they all decided to make big AI investments. It's been two years now. Many companies, you know, boards and investors, are starting to ask where the returns on this investment are. What has all this AI done for the company?

And so what a lot of CIOs are incentivized to do in this situation is take traditional automation and kind of rebrand it as AI agents. I'm not saying AI agents are snake oil, but they're very much overhyped, at least now. And so that you know, kind of lets them fool themselves, but also in a way, fool the investors who have put in money into exploring AI use cases at the company. So yeah, that's that's at least one example of how CIOs, in some sense fall for AI snake oil.

Michael Krigsman: To what extent do you think that AI agents are the next coming of greatness?

Arvind Narayanan: In some sense, they're already real. So if we look at chatbots for instance, when the early chat bots were released, they were just kind of web wrappers around large language models. You take the, you take a large language model. I mean, you fine tune it to give polite responses, etcetera. And then you just pretty much let it loose on the web. So that's how chat bots started. That's not what they are today. They are full-fledged products. They have memory, they have the ability to write and execute code. They have the ability to search the web and find information. So many other things, you know, dozens of features that make them more useful than vanilla LLMs or large language models. And I think it's reasonable to call these features, many of these features, agentic.

So, in some sense, when we're using chat bots, we're already using AI agents. So that's one example of AI agents that are working. Another example I will give is ways to do better research online, not scientific research, not coming up with new ideas, but you know, compile a lot of information about a particular topic very efficiently, or compile big lists of things by doing, you know, dozens of different web searches. And tools that try to do this like Google Deep Research, for instance, I have found to be quite useful. That's very much agentic.

lisbthThose are success cases of agentic AI. Some agentic use cases in coding as well. So there are some small but important successes. I think the hype is around things like AI agents being a drop-in replacement for human workers. Frankly, I think that's a little bit silly. I don't understand how people fall for that when you think about your job or really anyone else's job. All the dozens, perhaps hundreds of little tasks that need to go into it in the course of a day.

You know, the idea that agents are going to learn to do every one of them just by being trained ultimately on text from the internet, I think that is very very implausible. The only way to train them to do these subtle tasks in these enterprise contexts, is putting them into practice in those situations and having them learn from mistakes and that's going to be really a years-long process, maybe a decades-long process and it's not going to be a matter of just training these agents in, you know, in a vacuum, and then letting them, letting them loose. So agents are not going to be able to automate everything that we do, but they are already doing some small but important useful things.

Michael Krigsman: We have a question from Tika Nagi on LinkedIn who says, "What is your take on super intelligence through AI?"

Arvind Narayanan: I think this idea of super intelligence that you know, that we're going to build this galaxy brain, it relies on certain assumptions. It relies on this assumption that it's going to be us versus AI. And when we look at the history of AI, that has not been the case at all.

As AI gets smarter, we're able to incorporate that smartness into our own workflows, and through our intelligence, at least for now, being far more general and more flexible, by incorporating AI capabilities into our workflows, it's actually increasing our intelligence. So, right now there's no reason to think that anytime in the foreseeable future, we're going to get into a worker versus AI, or you know, a human versus AI dynamic, as opposed to you know, regardless of how small the model is, ultimately it makes us smarter.

And so, the whole notion of talking about it as super intelligence, as opposed to a super useful automation tool, presumes that we're going to let it loose in a way that it's not controlled, in a way that it's not doing things for us, but kind of decides on its own what to do. And that's a normative choice and that's the key point that I want to make. Whether we build super intelligence is really a question not about the capabilities of the technology, but about how we choose to deploy it. And we can choose to deploy it in a way that it's not the super intelligence which will likely be harmful simply because it's so unpredictable, but rather, we can choose to deploy it in a way that augments human capabilities, again, regardless of how smart the AI gets.

Michael Krigsman: Do you ever get accused by people in the AI business and AI vendors of throwing cold water on all of the magnificent greatness that super intelligence, which is so close, will provide us? Do you ever get accused of that?

Arvind Narayanan: We have people who don't like our message from a lot of different perspectives. Some of them are AI boosters. Some of them are people who are very concerned about AI and existential risk and think that we're trying to minimize those risks. We're not trying to minimize the risk, we just have a particular views on what the policy responses shouldn't be.

And there are people who are very concerned about AI ethics and the whole capitalistic model of developing AI, and think that we should actually be much more, you know, vehement in our opposition to all AI, which we're certainly not. We're talking about specific types of AI not living up to their promises. And so in a way, we see it as positive that we have people yelling at us from different directions instead of all from one direction. So it seems like we're in some healthy middle ground perhaps and I don't mind being in that position.

Michael Krigsman: The healthy middle ground where instead of one side attacking you or the other side attacking you, everybody's attacking you.

Arvind Narayanan: That's right.

Michael Krigsman: So one of the things you discuss in the book is the, the nature of institutions and how flawed institutions can contribute, or do contribute to this AI snake oil problem. Can you talk about that?

Arvind Narayanan: So the classic example is hiring in HR. So earlier I was critical of some of these flawed products in that space, but I think the reason those products are able to be so successful is that anyone with a job application out there, you know, is getting hundreds or perhaps a thousand applications per position, and it's just not an option to manually read through all of those applications and interview people and so on. And therefore, when an AI vendor comes in and says, "we can automate most of this problem for you" that seems very, very compelling.

Even if we kind of know in the back of our minds that the product might not live up to the hype, it just feels very tempting to put our skepticism aside so that we can benefit from this efficiency gain. But the problem is, if you have an underlying situation like, you know, too many applicants per position, that's a problem with your institution or organization, technology is unlikely to fix it. Because what's happening now is that job candidates are also using AI to massively increase the number of opportunities they can apply to. So that's only creating an arms race, and I don't think more technology is going to get us out of this arms race.

So, instead, I think we should be thinking about reforming how we do hiring and there are so many great ideas out there. I know some software companies that hire, you know, not just based on CVs and interviews, but by working on a two-week or one-month paid project with someone so that they can much more deeply assess their skills and fit for the position. That's just one idea. There are many other such ideas. I'm really a fan of partial lotteries for instance.

The idea is that we can do some basic evaluation of someone's qualifications for a position, but beyond that trying to rank applicants is just an exercise in fooling ourselves because how someone's going to perform is so uncertain. The variation in people's performance in a job is not a result of one person being more competent than another. I mean, that explains a small part of the variation, but a lot of it is because, I don't know, maybe they weren't a fit for this manager and those sorts of things that aren't even determined at the time you're trying to hire someone.

So the best we can do is do some basic filtering and then and then select randomly, and if we were forthright about that, I think that would dramatically simplify the process and actually bring some clarity and peace of mind for applicants as well and could, you know, save us a lot of trouble in the hiring process without decreasing the quality of applicants.

Michael Krigsman: Arvind, you're describing one specific use case and how to improve that process. Of course, it's going to be very different depending on the industry, the process, what's what you're looking at. Is there a way to look at AI itself, how we manage AI, how we potentially regulate AI from a policy perspective in order to drive underlying changes that cut across all of these use cases?

And I, and I'm asking this because it has deep implications for policy efforts, whether overarching AI controls and policies will actually do anything in the face of this use case or application specific set of issues.

Arvind Narayanan: I think the majority of policy and regulation should be industry-specific, should be use case-specific, but I think there are a couple of cases where it makes sense to at least think through AI-wide controls. And then, one is the issue of releasing model weights openly. Should that be prohibited? Should the policy makers be neutral about that or should they in fact encourage it? Invest in publicly, you know, using public money, building and releasing open weight models? That is, of course, a very contentious issue. We lean towards the open side of things. We've written a lot about why we take that perspective. So that's one example.

And you know, the benefits of openness have risks, but I think the benefits will be realized across every single industry because it makes it much easier to take a model and customize it for yourself. It potentially lowers costs because you can run it on-premises, potentially betters privacy for the same reason, etc. etc. etc. So it can bring cross-cutting benefits.

Another one is labor. So if, if you don't mind, let me give a historical analogy. In the wake of the Industrial Revolution, living standards for everybody eventually went up. You know, it was kind of the best thing that happened, but for a good 30 or 40 years it led to horrible labor conditions because people migrated from rural areas to the cities, where safety conditions were horrible, work hours, you know, 16 hours a day, workplace safety was not there, there was no collective bargaining. And the modern labor movement grew out of that, right?

And so today, there is a concern that AI is now creating a massive reconfiguration of the relationship between capital and labor. There might be massive job losses. Even in areas where jobs are not necessarily lost. You know, AI might make the job involve much more drudgery, because maybe now your job is just data labeling for AI all day. And a lot of those jobs, we know how those work, those are outsourced to lower income countries, where people are complaining about really horrible working conditions. And so maybe we need a new labor movement for the age of AI, and that again can be really cross-cutting and might potentially help every worker in some way.

Michael Krigsman: So what you're saying is that at this moment in time, we are in this interim period between, we can call it the pre-AI and the mature AI, where the social issues, job displacement issues, and economic models have been worked out, and so we're in this kind of messy middle right now.

Arvind Narayanan: Exactly right.

Michael Krigsman: So this begs the question, what should we do? What are there? First, let's start with users, folks, business leaders, are there frameworks that we can use to evaluate what's snake oil and what should we be doing inside our companies about this?

Arvind Narayanan: One thing I can suggest is, I mentioned an article earlier called 18 Pitfalls in AI Journalism and that's something that's not just for journalists, but for anyone who you know, watches the news or reads the news or whatever and is getting a lot of information about new AI products. You don't have to look at our specific list, right, but have some way in mind of knowing which pitfalls to look out for, because some of these are just recurring issues.

You might see a claim like 97% accuracy with no context on exactly how it was evaluated which doesn't allow you to reason about whether that evaluation is actually applicable to your particular situation. So that's one thing. When you're encountering information about AI, what can you do?

A second thing is, what is the culture in your organization of experimenting with, piloting, deploying AI, putting guardrails around AI? There's a role for individual workers here. There's a bottom-up aspect, but there's also an important top down aspect, and from a top-down perspective, I think companies need to do a bunch of things. They need to set the right guardrails. Privacy and confidentiality can't be left to individual workers, those have to be enforced at a company level. And there are some more subtle issues as well.

So, for instance, Wharton Professor Ethan Mollick has written about how in a lot of cases when employees experiment with AI-based innovations, ways to make their workflow faster, they may be reluctant to share it throughout the company, because they're worried that now they're just going to get more work, and they're not going to get any credit for this innovation they've introduced to their colleagues in the company, right?

So that again has to be addressed in a top-down way. How are you giving people the freedom to experiment, come up with new things, and then reap some of the rewards of doing that? So, those are a couple of examples. There are a lot of other things, but I don't want to, yeah, I want to make sure we get to all the questions.

Michael Krigsman: And how about from a marketing perspective, because when we talk about snake oil, it seems to me that you're really talking about the delta between what is, quote-unquote, advertised and the expectations that that creates between that and the reality of the outcomes.

Arvind Narayanan: When we look at marketing, there are so many different ways in which things are overhyped, starting with just the decision to call something AI, right. And you know, there, there is such a huge incentive for that. So maybe on that one, the market will correct itself a little bit. There's you know starting to be a bit of a backlash to everything being called AI, so maybe people will stop slapping the AI label on everything.

But if you're going to call something AI, one really basic thing, or maybe two basic things, I would expect, is "Okay, which part of this is AI? What kind of AI? And what task is the AI being asked to solve?" Just, you know, being clear about that would bring a lot of clarity, so people can assess for themselves, do we, do we think this is even a task that AI is capable of doing? And then going a step further, and showing the evidence behind it. Right? I think those two steps will address really the majority of AI-related hype out there. But I know it's easier said than done, because there are so many of these misaligned incentives.

Michael Krigsman: Well, there's a lot of pressures inside software companies to get those eyeballs, to sell those products, and AI is this black box in many cases and so it's very, very tempting to indicate, even if you don't directly say that, hey, you know, we have this, this thing, and it'll solve your problems.

Arvind Narayanan: Couldn't agree more.

Michael Krigsman: What about policymakers? Where do you stand on government policy? Should it happen and what should it consist of, and how should we get there?

Arvind Narayanan: Yeah, a lot of people like to poke fun at policymakers or complain about them, but I'll stick my neck out and say that so far they're doing a pretty decent job on AI. Certainly, I have my complaints as well, but I do work with policymakers in some capacity, you know, on a weekly basis at least and I've gotten to observe them up close. And I'll say a few positive things and I'll say what are some areas where things are not working so well.

So it's not at all true that AI regulation is a Wild West. In 2024 alone, somewhere close to 1,000 AI bills were introduced in the 50 state legislatures in the US, and really dozens of them have passed, and in fact, many people are concerned about over-regulation. So this Wild West idea is not true at all. And I think a lot of AI regulation has been focused on how people use it as opposed to the technology itself, and in the US we generally take a sector-by-sector approach. I think that's broadly the right way to go about it.

Another area where there's a lot of concern is policymakers are not very tech-savvy, and that's very much true, but the politicians you see on TV are not actually the ones making policy, right? It's their staffers, it's agencies, it's their attorneys general and so on and so forth. And in those places, the amount of tech expertise has actually massively increased in the last couple of years because of the concerns around you know, chat GPT and AI risk and so on and so forth.

So, those are all areas of progress. I think there is still a long way to go. I think there are concerns, for instance, when you compare the US and EU, some people will say we're far behind compared to them. We don't have much legislation at the federal level, and there are concerns about that. Maybe this 50-states approach is not the right way, it just imposes too many regulatory burdens on companies. So those are all things to complain about. You know, things aren't perfect, but I think we have to start from a recognition that at a high level, things are going okay, it's not a disaster in the policy space.

Michael Krigsman: Arselan Khan says, "How to create an ideal AI-ready organization? Do legacy organizations or startups have a better chance at it?"

Arvind Narayanan: I don't know if it's legacy organizations or startups, but I would say, I think it really has to be oriented around your industry and your use case, right. I think you know, AI is often looked at as a magic bullet, but you have to start with what your problems are, right, and then look for how AI can help in your existing workflows as opposed to seeing it as a kind of one stop shop solution. So I would kind of flip it around. I would start from, "what are the problems you need solving?"

Michael Krigsman: How can boards of directors evaluate ethical, reputation, financial vulnerabilities in technologies like AI, which are highly technical and very rapidly changing?

Arvind Narayanan: The good news there is that while it's true that AI is highly technical and rapidly changing, when we think about what are the aspects of AI that gives, give rise to harms or concerns or reputational risks, those are known quantities, right. So for instance, when we were talking about much simpler forms of AI before, simpler machine learning models, bias was a big issue, and it's a big issue with generative AI models as well. Right? And we kind of know how to audit for bias, and it doesn't depend too, too much on the specifics of the model and there's a whole community of AI bias auditors, and they're working on more recent AI models as well.

And so, if we break it down not so much by the technology, but by what the harms and risks are, what are the ways things can go wrong? What are the processes we've put into place to minimize those risks? Then we're not so susceptible to the whims of the technology.

Michael Krigsman: And I'll also just mention here, we were talking earlier about academia, and there are academics who are really looking at this bias question and I'm sure that they can be of help as well, to to business people.

Arvind Narayanan: Definitely.

Michael Krigsman: And with that, Arvind Narayanan, thank you so much for taking time to be with us. I'm very grateful for your time. It's been a very interesting discussion.

Arvind Narayanan: Thank you so much for having me. I loved all the questions from you and the audience.

Michael Krigsman: And a huge thank you to everybody in the audience. Before you go, subscribe to our newsletter and go to cxotalk.com. Check out our upcoming shows. We have great, great shows coming up. Everybody, I hope you have a great day and we'll see you again next time. Take care.