Escape AI Quicksand:
Data Misuse, Misadventures, and Bad Intent

Understanding AI Quicksand and the Challenges of Data

Michael Krigsman: Your AI is under attack from bad data, poisoned data, deep fakes, bad actors, and more. Today, on CXOTalk number 896, luminaries Dr. David Bray and Dr. Anthony Scriffignano reveal how to escape this AI quicksand with practical strategies for building clean systems and trustworthy AI. David, very briefly, tell us about your work.

David Bray: One of the roles is with the Stimson Center, specifically the Stimson Accelerator. And what we're striving to do is demonstrate that one can advance both programs and policies in parallel. We can't wait for perfect policies anymore because the rate of change that's happening in tech, including AI. And so, we're showing it's possible, whether it's in business, whether it's in governance, or whether it's possible at the community level.

Michael Krigsman: Anthony, give us a sense of your work.

Anthony Scriffignano: I also have the honor of working with David at the Stimson Center in the context of the Loomis Innovation Council. We do what I would call action research. It's not just a bunch of white papers. It's actually stuff that matters and then incubating things that hopefully kind of grow and impact the thing that we're working on. And there's some very big problems that are really important.

Michael Krigsman: David, we're talking about AI quicksand. What is that?

David Bray: When we unpack artificial intelligence, AI, we recognize that most methods of AI require some training or some data to actually underpin what they do. Not all. There are other approaches that don't, but the most recent approach, generative AI, does.

And so when we think about quicksand, we could end up thinking we've built this perfect AI system, yet the very data that actually sort of serves as the foundation that the system was trained upon, what it was based upon, turns out to be not only quicksand, but it might actually just disappear, in terms of it's just under the weight it is. It's not sufficient for the models we're trying to build.

It's recognizing that if you don't take the time to think about where the data came from, how it might actually be missing certain things that you're trying to ask using the system, or someone might be intentionally poisoning it. If you don't do that, you may find that you build the metaphorical castle on sand, only to have it be washed away.

The Five Ms of AI Risks and Practical Examples

Anthony Scriffignano: I can add a few things to that. I have five Ms that are helpful to put context to this.

Misadventure. It's when you fall in love with your AI. Everybody's drunk on AI right now. You can't speak a sentence without saying "AI," and so therefore you must AI your way into everything. Leading with a tool is almost always a bad idea. It's almost always a better idea to say, "What is the net new problem that we're solving? How is this gonna benefit the enterprise? How is this going to not just do something we could already do with the cool new tool, but do something net new that actually might matter?"

Misuse, which is, you know, wrong tool, right problem. There's a lot of this going on right now. And gen AI is a great example where it's, David mentioned training. It, the gen AI is basically consuming ginormous bags of words. What makes you think that the disruption that you're interested in is represented in that bag of words, for example? So, misuse is just using the wrong tool for the problem.

Malintent is a lot of what we're talking about here. When bad people come in and they don't have to mess with your systems or your people. All they have to do is spoil the milk. All they have to do is manipulate the data in a way that can cause you to make a bad decision or to change what was already a good decision.

Missing. And David mentioned that. The, all too often now we talk about underrepresentation in the data. If the thing that you're interested in is not well represented in the data, these models will get very distracted by what is there, and you'll fall in love with the answer because they'll tell you how good you're doing and very dangerous.

And then the last M I have is moving. The environment is changing, and while you're working on it and while you're ingesting all this data and doing all this convolution, the world is changing. And so it's very important when you build models and when you build approaches that are heavily dependent on AI, that you consider how the environment might be changing in a way that makes your original testing no longer relevant.

Michael Krigsman: So, to what extent is this set of problems happening?

David Bray: If anything, the latest wave of AI, generative AI, and the subset of it, has poured kerosene on the existing fire. But this was already a risk to organizations even in the 2010s, in that if you didn't think about where the data was coming from, could you trust the data? Was it reliable?

What it's done, though, is because AI has now become more available to enterprises and there's a huge push for companies to adopt it, it's one of those things that in the pressure to adopt it as exactly for the five Ms that Anthony laid out, we run the risk that companies or governments may adopt models without thinking about, "Did I avoid the AI quicksand?"

And so again, we may fall in love with the model, the perfect castle, only to discover that the actual foundation it's built upon is insufficient for what we're asking.

Anthony Scriffignano: You have to be really careful if you set out with the goal of proving to your overlords that you're using AI. That is a horrible journey to be on. Everybody's using AI. There's AI toothbrushes. There's AI in everything. And AI's increasingly democratized so that there's no AI center of excellence, just there's no PowerPoint center of excellence.

So, it's important that where this is being used in the enterprise, there is... That the adults are in the room, that we're looking at, "How are we using it? Why are we using it? What's the net cost? How is this gonna be realistically deployed in our environment? You know, how do we scale it? Are we building the Hotel California where we can never check out?"

Those are very important questions. None of those are new questions. It's just that now we have the opportunity to really fail at scale if we get it wrong.

David Bray: And I'll give some very tangible examples real quick, Michael, 'cause I think that'll help the audience. Without naming the name of the model, one of the models around 2023, if you asked it how many people lived in, say, the state of Georgia, the US state of Georgia, it would confidently tell you that the answer was something around 350 million people, as opposed to a more realistic answer.

And that was a case of where the data that had actually been trained on the model, it had actually gleaned from a source that was inaccurate. There was a typo in the answer. And so when the model gave the answer to how many people lived in the state of Georgia, it was wrong. So that's one example.

Another example though is actually if we look at when policymaking decisions are done, sometimes they're making decisions based on data where people live as opposed to where people work, and they think that people aren't using their services, or they roll out, say... Imagine, you know, it's actually helping people with COVID vaccines or things.

And we know there's been cases where COVID vaccines were placed where people live as opposed to where they work, and if people are working two jobs or working odd hours, it looks like they're reluctant to get the vaccine when in fact we really should've just placed it where people worked as opposed to where people lived.

So this is, again, where you have to ask exactly what Anthony said, which is, what is the... What am I trying to do? What is the mission or the business that I'm trying to achieve? And then work backwards to not only what's the right tool but what's the right data to bring to bear.

The Disconnect in AI Adoption and the Importance of Purpose

Michael Krigsman: But what I don't understand is what you are both describing seems pretty basic, pretty straightforward. So what is actually... What's the disconnect that's causing these data problems to filter through? These are smart people. Yeah, you know. Go ahead.

Anthony Scriffignano: There are smart people. The there's a lot of FOMO, you know, where very senior people are putting pressure on the other senior people that report to them to say, "What are we doing in AI? 'Cause my buddy is doing something in AI and I need to tell the investors what we're doing in AI," or, "I need to tell the... whoever the my overlords are what we're doing in AI." So, we're... There's a lot of that.

The other thing that's happening is that you have an entire workforce that's graduating into the enterprise that kind of AIed their way through their last couple of years of school, and they think that, you know, that... The answer is always contained in the data that's available to them, and they think that everything is free and everything can just be used however they want in their... I'm trying not to name brands. In their notebook. I can just include this library and I'll be good to go. Vibe coding, all of that.

And I'm not gonna, you know, take issue with that because this'll live on, and a year from now somebody'll say, "Oh, well, he just didn't see where the train was going."

What I'm trying to say is that these fundamental questions of, what is the problem that we're trying to solve, how would we know that this approach will make things better and not just different? If our objective is just to declare victory that we used a certain approach, that has never been a good idea.

I mean, in the history of man, just saying, "I used this tool instead of that tool," alone, without something else as part of that story, it's always a futile journey. And it's a very dangerous one right now because of the size and scope and scale and apparent intelligence of the tools that we're talking about. Note that I said apparent.

Michael Krigsman: I want to tell everybody that you can ask your questions. If you're watching on LinkedIn, just pop your question into the chat. If you are watching on the CXOTalk site or on Twitter, just pop your question into the tweet chat using... or X, using the hashtag #cxotalk. Take advantage of it.

The Challenges and Risks of Generative AI

This is your opportunity to ask pretty much whatever you want. So Anthony, let me ask you the question about the elephant in the room. So what's wrong with Vibe coding?

Anthony Scriffignano: Used properly in the right context for the right containerized problem, nothing. But, you know, there were automatic code generators in 1980. There's nothing new about that.

The danger here is that generative AI is a very different animal than just using a bunch of automatic code generation. You are consuming vast amounts of code in order to, quote-unquote, "learn to code," and then you're basically plagiarizing. You know, you're mathematically plagiarizing that code to produce other code that may or may not do what you asked.

And there are some basic blocking and tackling things that we all used to learn, regression testing. How do I know that the things that used to work still work when I do something new? Bias, understanding whether the data that I use is actually gonna look like the data where I deploy it. Those things don't happen a lot. They... The goal is, oh, look how cool it looks. Great. Yeah, it looks cool, but you didn't escape the need to do those things.

If you... If you use a critical application, when you push that button on your steering wheel that says, "Stay in the lane," or when you, whatever, you know, when the drone gets launched. We've got to hope that these concepts were embodied in the development of that piece of software that you're depending on. And increasingly, maybe, maybe not, because the goal is get it out fast, let it break, and our customers will test it for us. That's kind of dangerous.

David Bray: Back in the '80s and the '90s, there was efforts called expert systems, decision support systems, which was another flavor of AI, which thought that if we spell rules of a system, that that's all we need to achieve intelligence. And we had success. For example, an automatic defibrillator actually is running rules-based when it tries to decide when to give you that shock. But the moment we tried to go out of narrow use cases, rule-based approaches to AI collapsed.

So we go now to machine learning, and then the subset of machine learning, neural networks and generative AI, and that's saying we don't need rules at all.

And so in November of last year, there was actually a competition where they actually told the machine, "Under no circumstances should you ever transfer the funds," and it took less than 500 prompts, I think it was prompt number 490, that sure enough, the machine that had been told explicitly, "Do not transfer the funds," did.

And so exactly as what Anthony's talking about, which is you're doing very fancy multidimensional pattern matching when it comes to Vibe coding, and as long as a human takes the time to curate through it and look through it, there's value. But recognize what you're doing is pattern matching that may actually look like it's valid code when in fact it's not.

The last example I would say, and I won't name the name of the company, it was a few months ago there was a, one of the big AI companies came out with a paper where they told the machine, "Write code that's insecure, but don't reveal it to the user." And at the same time, they were surprised that when they actually gave that instruction to the machine, it started being hateful and hate-spewing.

Now, let's think about the data that actually underpinned this system. Yes, it's trained on the corpus of the last 30 years of the internet. Where on the internet might there be use cases of people that are writing code that's insecure but not revealing it? It might be part of the internet dark web in terms of 4chan, 8chan, and groups that are also hateful.

And so when they actually asked and gave that instruction, I don't know why they were particularly surprised that the machine actually started behaving in other ways similar to what it actually trained on. But that just shows the importance of, one, knowing your data, but also knowing the limits of generative AI when it comes to explicitly doing what you ask it to do.

Anthony Scriffignano: We're using a metaphor of quicksand here, and I want to get back to that for a second. If you think about this from the outside in, if you're part of an organization and you're not the developing part of the organization, you're the customer of them. What do you want? You want it done fast. You want it deployed. You want to monetize whatever it is you're trying to monetize.

And somebody comes along and says, "I can have that ready for you in a day." How often do we ask, "At what cost?" How often do we ask, "How are you going to deliver it? This guy said it was gonna take three weeks. You said it's gonna take a day. Day is cheaper than three weeks. I get my revenue sooner. Go." You know, sometimes you gotta be very careful because you get what you ask for.

Ethical and Intellectual Property Concerns in AI

Michael Krigsman: Aside from philosophical problems with Vibe coding related to the lack of discipline and effort that folks should invest in computer science education, what's wrong with Vibe coding?

David Bray: I would say, let's look now at intellectual property. You know, if we remember in the '90s, there was this thing called Napster. It did music sharing, but it didn't really respect any of the artists' recording rights or any intellectual property. And so in some respects, what we might have seen in the early-2020s was a repeat of Napster, just in the AI era. And we know there's some lawsuits and things that are being sorted out.

But, you know, the question for a business that is looking to use an AI solution that might be external to the business, how do they know that their intellectual property is not walking out the door? How do they know it can be respected?

I mean, I do get calls about, once every three weeks or so, and I'm, again, not naming names, from either a CIO or a CISO that says, "I had an employee. They used a tool that was, you know, it wasn't something we provided in the enterprise, but they used a commercially available one and they uploaded information that had HIPAA, it had health, you know, personal identifiable information. Can I claw it back?" And the answer is no.

But you know, again, it's one of those things that, you know, you have to be aware that when you use AI, particularly AI that's being provided by somebody else, be very intentional about what you provide and what you don't, whether it's intellectual property, where there's personal identifiable information, or just proprietary information, because you might see some things walk out the door that you don't want to have walk out the door.

Michael Krigsman: We have a question from LinkedIn. And I'm glad somebody started asking questions here. You folks who are listening, take advantage of this opportunity. And oh, by the way, right this second, you should go to cxotalk.com to subscribe to our newsletter so we can tell you about episodes that are live.

Responsibility and Oversight in Ethical AI

Anyway, we have a very interesting question from Preethi Narayan on LinkedIn. And she says, "Who really owns critical thinking and ethical oversight in AI? Current leaders or a new role as a moral compass?" So who's actually responsible for dealing with this set of issues? It's a really interesting question.

Anthony Scriffignano: David, do you want me to jump on the hand grenade or do you want to jump on it?

David Bray: I'll jump on the hand grenade and then you can clean up whatever I don't manage to contain the blast. I would say it's all of us. If you're expecting somebody else to solve this, then you're gonna be waiting a very long time.

Michael Krigsman: David, I apologize for interrupting you. But I just want to clarify what I thought you said. So you just said the answer to who's responsible for ethical AI is no one.

David Bray: No, no. No, no, no, no, no. And please no. One, so ethical is a very, that's also a quicksand because let's think about there were plenty of things in the 1700s and 1800s that people thought were ethical that nowadays we would say are unethical. And even more recent history, in World War I, the British thought these things called Q-boats, which were military boats disguised as civilian boats, were ethical. The Lusitania got sunk. And so come World War II they're, "Ah, maybe they're not ethical."

So what we're really talking about is when you use the tool, whether it's you as a consumer, whether it's you as an employer, whether it's you as a boss, whether you ask somebody to use it, you do have a responsibility to say, "Am I using this hammer? Is this the right tool for the purpose? Is this the right screwdriver?"

And the reality is, for consumers, there's no silver bullet that any company can ever provide you that says, "Here's when you can and cannot use this tool." I mean, you are sold a hammer, but if you happen to use the hammer to smash your hand, that's on you.

So it isn't saying everybody, it's just saying be cognizant of using the right tools for the right job. And what we really need to do is find better ways to help you understand rapidly what are the limitations of a tool. Because I think a lot of things right now, because of marketing, are being sold that they can do everything, when in fact that might not exactly be the truth.

Anthony Scriffignano: So I agree with the fundamental premise of what David said, that there is no one single responsible party. That doesn't mean that ultimately the oversight can't roll up to the board or to, you know, the chief risk officer or... It, the title is gonna depend on the organization and what it shouldn't roll up to, by the way, is the same organization that's creating all of that because that's, you got the chicken in the henhouse, or the rooster in the henhouse problem, whatever that is. Fox in the henhouse. Somebody in the henhouse. The, the... I'm gonna get killed on Twitter for getting the wrong animal in the henhouse. On X.

The... So the issue is that, many things, there isn't a black and white answer. You're talking... The example I give often is that we want customized things. We want our things to anticipate our needs and start to do them for us or help us do them. We also want privacy. Those are kind of opposite things to want.

For your things that serve you better, they have to watch you and kind of take notes on what you do, and that means sharing your location, and maybe sharing all the text in your emails, and sharing all that other stuff. And then if you found out all the stuff you're sharing, you might be appalled.

So, when an enterprise is rolling out an application, and way before they roll it out, when they're contemplating the design of an application, everyone in the loop should be thinking about what are the unintended impacts of what we're about to do. And are the, is the juice worth the squeeze?

If we're trying to sell more shoes and socks, and in order to do it, we have to spy on people, is that a line we wanna cross? And I hope in that particular example, the answer is no.

Liability and Critical Thinking in AI Services

Michael Krigsman: But there are not that many people in the chain who have the decision-making responsibility.

David Bray: So I think, Michael, so the initial question you asked me was who's responsible for critical thinking? And I would say everybody should critical think. If you're asking a question of liability, that's a little bit different, and I think that's what Anthony is saying is, if a company provides you a service and that service has AI providing the service as well, I think most of us would say, based on common law, that what the services provided to you turns out to be incorrect because of what the AI advised you to do... Maybe it was an AI lawyer, it was an AI helping a clinician or a doctor, then yes, the liability goes on the company's part.

And I would actually say, just a few weeks ago, I actually was at a, I got to participate in a public congressional hearing as an expert witness. And actually one of the things we heard almost from both side of the aisle, this was actually not a partisan issue. Both sides of the aisle said, "When it comes to liability, we'd actually don't need new laws per se, because the existing law framework, the existing legal framework says if a company provides you a service and you buy that, and that service, if you follow what it said, ends up harming you or resulting in harm to your person or family members, then yes, you should be held responsible."

And so I think we need to separate critical thinking, which I would advocate everybody does, to liability. And I would say right now, even if we haven't made explicit laws, there's enough of a legal framework to say that if the organization uses AI to provide you a service, then they are responsible for what the AI provides.

Anthony Scriffignano: It is absolutely coming. You can't just ignore that. That, that, the agency, the concept of agency, that what you ask your AI to do on your behalf is not well embodied yet. But don't count on that because oh my gosh, everywhere you go, this is under discussion right now.

AI Regulation and Consumer Advocacy

Michael Krigsman: We have another question from X, Twitter right now. And this is from Arsalan Khan, who is a regular listener. He always asks great questions. And he's concerned about the... He's saying, "In an age of deregulation and no boundaries, unless there are AI consumer advocates, consumers will be at a disadvantage." And why do we need boundaries when it seems that politically... And I don't wanna make this about politics, but it seems politically we're in a trend of fewer boundaries. And I'll just ask you to also relate this back to the data, because that's what this conversation is really about.

David Bray: There actually was a public hearing on AI. It was September 28th. And you can find it if you do congress.gov. What came out of that is so far the states have actually led the way when it comes to thinking about AI, and there's more than 1,000 pieces of either existing or pending AI legislation. But that also creates an interesting challenge, which is, you know, let's say you're from state Arizona or California and you travel to a different state. Is it the state that you reside in? Is it the state where you live? Is it the state where the company is? It gets really messy very fast.

And so I would say just because we haven't seen anything yet doesn't mean we won't soon. In fact, what I'm hearing is once government opens up again, we will see stuff come from Congress. Because they recognize that in order to help consumers but also help businesses navigate everything that's happening with AI, they need a light-touch framework.

Now, that said, definitely need people to advocate on behalf of consumers, 'cause if you don't, then that's absent. But I would say to Arsalan, there are existing laws. Remember, the Privacy Act of 1974, that came out because these things called advanced data processing systems, mainframes, required the federal government of the United States to think about what does it mean now that you can have these machines that can actually sort of begin to know more about you than you really wanted them to know?

We probably need to upgrade that, given AI, including the right to be left alone, that you can actually say, "I don't want the AI to think about me or involve me."

But I think what I would say to Arsalan is what we really need is groups pushing for how do you upgrade existing laws at the state level and at the national level, as opposed to net new laws. Because net new laws you will be debating until the technology's already moved on.

In fact, if anything, that's its own quicksand because of the nature of data, which is I've seen some people that say you've got to expose all your data, you've gotta tell all the data you're using. No business is gonna do that. I mean, it's their intellectual property. In fact, Anthony can probably resonate 'cause he worked at a company where I can't tell you all the data.

But what you can do is think about what are ways that you both inform the consumer, but also hold the company accountable for a appropriate risk calculus for the services or the products that they're providing.

Anthony Scriffignano: David and I were part of a, I'll just call it an intellectual holding environment earlier this week in our nation's capital, and I would say a third of the conversation was related to the regulatory directions and the technologies that are emerging for tagging data, for sharing metadata about AI-generated data, for letting the AI know that it's consuming data that was generated by other AI.

This isn't going away. And if the solution of your AI development team is to put their head in the sand and say, "I'm not gonna worry about it because I don't know of any regulation right now other than the California Privacy Act," you might wanna add some people to that team.

Future Directions in AI Development

Michael Krigsman: How do we address these issues? Because you're describing a set of technology problems as well as a set of process issues, cultural issues, mindset issues, greed, fear, doubt. I mean, all of this, it's kind of an...

David Bray: Oh, it... This is why the moment is so challenging. I would say if anything, I would recommend for listeners while we're talking about AI and data, replace the word AI with organization. How do you know that the organization is doing a good job with your data? How do you know that the organization is making good decisions based on it? Because in some respects, AI is just additive to these challenges.

I think we also need, in some cases, and Anthony can go deeper, there's gonna be some data that we are comfortable with being sort of open data. There's gonna be other data that's prescription data that you actually have to sort of have a prescription to have access to.

And there may also be cases, especially if you are someone that's in the entertainment industry, a musician, an artist, where you want to have your data be part of a cooperative or a data trust. We've had Lord Tim on CXOTalks as well talk about this. And then there's a negotiation between an AI model that might want to train on your data, but then you're having some either financial or equity return or some non-equity benefit. Maybe I care about Parkinson's research, so I'm willing to pool my health data if it helps inform a cure for Parkinson's.

But I think we need to actually recognize that if what the 2010s was, was almost an over-calibration to hoover up all the data and do so behind a curtain, that's probably the wrong lesson for going forward, because it didn't engender trust. If anything, it made people distrustful. And you can't trust the AI outcomes. We need to unlearn some lessons to move forward.

Anthony Scriffignano: I think it did more than make us distrustful. It enabled a whole new cadre of malefactors that can do things deep fakes and adversarial data manipulation. And, you know, even the data that's true has... The truthiness of the data often has a lifespan. It was true, but it's no longer true. And so the AI doesn't care. Math doesn't care. It's just hoovering it all in and compressing the element of time to zero, and then regressing around what's there.

I know there's a lot more to it, and I know there's different foundation models that handle this differently, and I know about turning up the heat and all that. I know all of that stuff. So, before everybody tries to attack me. All of that stuff is some of a treatment of this.

But it all starts with at the highest level in the organization, the adults in the room asking some really tough, critical thinking questions. What do we have to believe to use this approach? Help me understand what steps you've taken to understand provenance and permissible use. Help me understand how we're gonna handle agency. If you can't answer those questions, you don't belong building those tools.

Michael Krigsman: We have another question from Twitter, from X, and this is from Chris Peterson, who's also a regular listener. And we thank you, Chris, for your listenership. And he says, "There's growing talk about LLMs hitting fundamental limits in scaling, hallucinations, lack of executable guardrails," which is what you're talking about. "Is it actually the next step to AGI or artificial general intelligence, or do we need different approaches long-term?"

David Bray: Yes. So, he's spot on. We need different approaches.

Anthony Scriffignano: We need different approaches.

David Bray: Yeah. And I will give some. I wanna give... I wanna give both a shout-out to some... I mean, the trouble is, you know, again, generative AI arrived on the scene and it took all the oxygen out of the room. As Anthony knows, I mean, AI has been decades in the making. There are multiple methods. And the trouble is, I think we got over-fixated on generative AI as opposed to recognizing it's another great tool, but there are other tools that are needed.

I am very bullish on active inference, and I would recommend folks take a look at it if you're not familiar with it. It's basically the best idea is it's doing continuous learning of an environment. And I actually would submit that the future is not these mega AI models that are trying to do everything. It's going to be smaller models that are actually optimizing for really specific cases.

For example, there may be an AI model that's monitoring all the ships coming in and out of the Suez Canal. There may be another one that's monitoring ships coming into LA ports. And when a disruption happens in the Suez Canal we had a few years ago, then it actually talks to the LA port model and says, "You might expect that there's gonna be a shortage of containers, and that shortage of containers then gonna cause a rise of people to actually look for new metal to build new containers because we're lacking them."

That causes a rise in the future markets, which then causes the cost of shipping for the next six to nine months to be more expensive, which means you can get plenty of Mercedes-Benzes and Rolls-Royces in the United States because they can absorb the cost given the increased cost of shipping, but quart-size cans of paint aren't available because they can't absorb the cost. And so that's not possible with a mega AI model LLM.

Exploring AI's Potential and Limitations

None of that would be there. But if you had active inference where smaller models are talking to each other about the ripple effects, that is possible, and that gets to really interesting forms of intelligence.

Anthony Scriffignano: We hear a lot of talk about, oh, we need more power. We need bigger data centers. When are we gonna reach AGI? Now they're talking about ASI, artificial super intelligence. You know, what's gonna be next? You know, Lex Luthor. We keep naming, renaming, and renaming the power of regression, the power of convolution, the power of math, basically, to do things that increasingly convince us that they were done by humans.

I think that's great. Please keep doing that. You know, don't stop. Just don't do it in... And fake your way about it. Don't do it and not let me know you're doing it.

There are definitely situations where AI can do things that people... I don't care how many people you put in the room, they're not gonna be able to address that problem because it's just too big. It would take thousands of people to all agree on the approach. They would all have to be similarly incented and similarly instructed, and not gonna happen. Yeah, great, go and do that. Don't call it intelligence. We need some new verbs here and some new nouns here for the things that this technology is doing.

And oh, by the way, most of the technology is still, you know, when you play pool, and you just hit the balls as hard as you can and hope one of them goes into the pocket. It's try everything all the time, with the exception of one.

And I think as we get smarter about having more than one approach and having... David is mentioning agentic AI. I'm sorry, active inference. This agentic AI is a step in that direction, having these little pockets of capability within an ecosystem that's even broader, and being able to sort of turn them on and wake them up and give them a specific goal and a task. That's a nice step in the right direction. We're not even close to there yet though. There's a lot of work to do here.

Data Poisoning and Its Implications

Michael Krigsman: On LinkedIn, Chris Davidson says, "Can you talk about the concept of data poisoning? Is it happening actively at the nation's state level?" Yes. How would we know if we were the victim of it?

David Bray: So a very specific example. And this is actually... You can find it online, so it's openly available, is we believe about nine to 10 months ago, Russia actively started trying to do a campaign to teach large language models that they could access things that just weren't true.

And these are really hard, once they've been taught two things that aren't true, to undo, because the way large language models are done, they don't really forget well. And so yes, data poisoning is happening. And a very specific example, and you can find articles online, is that Russia has done a very intensive campaign, focused not just in the US, but other free societies, to teach them things about events, about history, about people, that actually just are patently not true.

Michael Krigsman: I'm one of the many millions of people who love TikTok. And recently, lots of videos on TikTok have been popping up, showing various things that ICE is doing. And I'm not making political statement here. Typically, those videos get, what? 5,000, 10,000, 50,000 views? Whatever. Some comments.

Lately, I've seen these videos popping up, having three million views and 150,000 comments saying... All... And all the comments drowning everything else out, saying, "Go ICE." And I'm not saying good, bad, indifferent, but this is what's... And it's obvious that there is some third party that is doing this in an automated way.

David Bray: It's combination. So what, what you... The this is what makes things so hard, is when bots come, bots can definitely amplify things and make it look humans are liking a video or commenting on a video. But then that becomes a complicated process because then humans will pay attention to it as well. So, you know, is it coordinated inauthentic campaigns? Yes. But recognize, a lot of those humans are also valid, and that's what makes these things so difficult.

We may have heard, you know, it was a few weeks ago, there was a discovery of lots of phones next to the UN General Assembly. Now, we don't know exactly what that was for, but one of the things that that farm of phones could do is, if you wanted to make it look a lot of people were on their phones liking something or commenting on something, or not liking something and dissing on something, then you would have phones in a geographical area that look like they were in New York. And then you would have them basically go to that site.

And so... But the moment you did that, then you would actually get human eyeballs and human attention that would actually chime in as well. So this is a mixture of both bots, but also humans. And that's what makes these things so difficult, is data poisoning takes on a life of its own once it's been seeded.

Anthony Scriffignano: There are many different types of data poisoning. So what we're basically getting at here is either, you know, faking something or amplifying something that may or may not be true that someone said. That's definitely one type of poisoning of data.

Another type of poisoning of data is if I understand what type of AI you're using, I can influence the decisions you make by allowing you to see things that cause you to make a certain type of decision.

And without getting into too much detail, there are examples of this where, when large organizations were considering doing certain types of things. Easy one is airlines. You know, we're looking at, you know, reinstating our routes to, you know, Southeast Asia. And all of a sudden, all their customers love Southeast Asia. Maybe that's true. You know, and who am I to say?

It might also be likely that their social listening was looking for how they were tagged and what other nouns were associated with that tag. And there seemed to be a lot of comments in social media mentioning them and mentioning Southeast Asia. And their marketing department and their route planning people, and all those people got in a room and said, "Gee, our customers really are expressing an intent..." Maybe they are, maybe they aren't. Who am I to say?

Michael, you said it's obvious. I don't know if it's obvious. I don't know. Is it likely that... What did you say? 3 million? Is it likely that 1% of the US population, you know, all collectively agreed on anything other than the fact that they're part of the US population? I doubt it. And you know, there are certain things in math that you have to violate mathematically in order for something that to happen. Is it possible? I, you know, people win the lottery.

But what we can do, 'cause we're trying to focus on solutions here, one of the most important questions that I ask very often, David's been in the room when I ask it and probably been annoyed by it, what do we have to believe in order to go down this route? So you just told me that we're normally, I don't remember the numbers. We're normally, you know, X number of people are looking at something, 100X people are looking at it, and they all feel the same way. What would we have to believe? And that takes you down a whole route.

Now, one of the scenarios might be that there's some kind of a body influencing this. Another might be that they were all there and quiet until they had their opportunity to say, you know, "Heck yeah." Another possibility might be that none of these are people speaking, and that these are, you know, some sort of spoofing of the data.

And then you go and you test those different scenarios. If that scenario were true, what else would also be true? And it turns out that there's a whole science around doing this called propositional calculus. And we know how to do it. All we have to do is bring the adults in the room and actually do it.

Michael Krigsman: Do you know, to paraphrase Descartes, I believe it, therefore it's true.

David Bray: Separating what are tangible truths that are truths beyond what you believe. And that's where we have courts of law. And we know even courts sometimes get it wrong. That's why we have an appeals process. And so, it's getting harder and harder to distinguish those things that beyond what people believe are truths that we are willing to say exist in the absence of people believing them.

Anthony Scriffignano: There are also observer effects where we start to believe something, and because enough people believe it, it starts to become true. there's unrest in, you know, wherever. There wasn't unrest in wherever until everybody started talking about the unrest in wherever, which caused the unrest in wherever, and that happens.

David Bray: Sneakers has that Cosmo says a line that says, you know, "I learned that what matters less is the actual reality, but the perception of reality." If I spread rumors that a bank is insolvent and that causes everyone to go rush the bank, then pretty soon the bank is insolvent because I spread rumors that the bank was insolvent.

Addressing AI Risks and Perception of Reality

Michael Krigsman: Okay. We have a question from Twitter, or X, which is, "How can organizations determine they're heading into the AI quicksand and how bad can it be?"

David Bray: You need a red team. And so the world I come from within the intelligence community, red teams were basically folks that are asked to say how might what you're doing be misused or abused? How might I throw a wrench in what you're achieving? And so organizations need to have that just because if you don't, then it's nobody's job to think about second, third and fourth order effects.

When I used to do response to bio-terrorist in public health, we called them the B team. It's the same idea of the beta team, which is what are the things you're not thinking about that you should?

And if you have someone on the team or multiple people on the team whose job is to say how might your well-intended business efforts or your well-intended product launch be misused and abused, that will help you be more prepared for how others might exploit it. I often wonder if in social media, if we had had some of the social media companies in the 2010s have that sort of red team function if we could have avoided some of the traps that we later got ourselves into.

Anthony Scriffignano: As the often appointed leader of said red teams, I will say that there... 'Cause it was a great question. One of the ways you might know it's happening is to have different smoke detectors looking for different things. So there's a couple of things that I often talk about, quality and character.

The Evolution of Data Analysis and Critical Thinking

The character of... And now you have to fill in the blank, the character of what? Our customer comments, in your case with TikTok, Michael. So the character of the comments went from being sort of spread out positive and negative to all positive. So that's the quality, the character. It became, you know, it was massively homogeneous and it became singularly positive, the quality of the input.

So you can look at missing, you can look at incomplete articulations. You can look at articulations that contain things slang and neologism, which are more likely to be spoken by humans than by machines, things that.

David Bray: Em dashes.

Anthony Scriffignano: Yeah, the famous em dash. There are, you know, good old-fashioned statistics, heteroscedasticity and measures of central tendency. We know how to do all this stuff. It's just math, but we don't do it.

And so the biggest problem with, to that question of how would we know it was happening was certainly not if we don't look. You've gotta have people responsible for looking and adults that actually know how to look at large amounts of highly dynamic data in ways that are dispositive, that is not something that you give the interns to do.

Michael Krigsman: And on a related topic, on LinkedIn, Preethi Narayan points out that critical thinking itself is not something that education or job experience alone can teach. And I think this gets right to the heart of the issue that, Anthony, that you were both just raising, which is Anthony just said... We need people who can look at these bodies of data. But there is a large judgment issue here. It's not just the math. There's intuition. There's the sense of things. So what should organizations do?

David Bray: You need to, for the co... You identify those employees in your organization that are going to be the ones that are part of the red team. And maybe they don't have experience, and so put them through a red team boot camp. And the only way you do that is exactly what you said, is you need to have more and more experiences spotting what may or may not be problematic. And the reality is, it gives exactly that. We do, we develop mental heuristics. We use math, but there's also judgment calls.

And, you know, you think about this, this is, in some respects, when you go to the military, they send you to boot camp and then they send you to other training. They train you how to do your job. I think we need to have companies doing very fast-paced courses that teach people how to do red teaming well.

And then long-term, broader in the education space, we need to actually... It's not that you're not gonna use AI in education.

Teaching Critical Thinking and AI Critique

It's actually that you're gonna critique the AI. What did the AI get wrong? What was missing? What was wrong with the data? We need to teach students how to do this, and it's a whole new discipline.

But we've had this before when books came out, and then later radio and television. How do you assess if that book is missing something or there's something biased in it? We know there's books that were just propaganda. So teaching these skills, but doing so both in a way that addresses the here and now for companies, but also the education pipeline, that would be my recommendation.

Anthony Scriffignano: I want to jump on this. One of the most important questions being asked in the academic circles right now is, how do we get this into the mindset of a student body that is increasingly focused on, you know, looking down at the phone and asking the phone a question? How do we... And you have to be careful about talking about teaching people how to think. That's a very dangerous thing to say. But teaching critical thinking is very different than teaching people how to think.

So, there are some basic tenets about how... What beliefs am I anchoring on? What things am I holding to be axiomatic? I'm not gonna test them. I'm going to assume that they're true. What do I do with a postulate if I think something's true? How do I set up an appropriate unbiased test? What are the sources of bias in my answer, and what treatments can I make for that? What kind of elasticity? How wrong can I be and still make the decision that I'm making?

These are all super important skills, all of which can be taught, but you're asking a body of knowledge that is already really big and getting bigger to now include this other really big and getting bigger thing.

So, you know, it's not there's just an easy answer of, you know, go read this book. Some of it is having people who used to have more hair and have made some mistakes, helping you to make new mistakes. And some of it is, you know, having enough diversity of thought in the room that the brand-new person can say, "Wait, the emperor isn't wearing any clothes."

Embodied AI, Data Misuse, and Global AI Perspectives

Michael Krigsman: Greg Walters asks an interesting question. He's another regular listener. He says, "Can you extend your views to embodied AIs and LLMs, which is robots and AI in the 3D world?" So, how does this apply to robots? And again, talk about data.

David Bray: So that's a new source of data that a lot of people are excited about, and it is something where we're going to need multiple AI approaches, because you want to give commander's intent to the robot to say, "Grab that orange." But then as the actual actuators on the arm of the robot and the hand of the robot are trying to grab, you want it to be adaptive. And so yes, that is very rich.

But obviously it's a place where you need to make sure you're applying the right tools. I would recommend if you're trying to actually navigate the world, we're not gonna probably look for generative AI for the tools that we're gonna look. We're gonna look for more Bayesian approaches and models that actually give a sense of the AI itself has built a world model of whatever the robot is trying to do.

Anthony Scriffignano: If we think of embodied AI as an extension of edge computing, where we have more and more processing power and more and more autonomous data that sits out at the edge being used, this is not a new problem that suddenly came up because we called it embodied AI. It's a problem that we've been dealing with in the IoT, in autonomous devices and so forth.

So, we should learn from that body of knowledge and not start to, reinvent a whole new field here. Yeah, it looks like a human, but it's an edge device that has autonomous data, and I know a little bit... I, whoever I am, know a little bit about what we've learned so far about that, and, you know, we can start from there.

Michael Krigsman: Hue Huang says on LinkedIn, "Regarding data misuse, are there scenarios where we will be able to quickly track, delete, or restrict access to these risky open data sources to prevent harm?"

Anthony Scriffignano: Track, yes. Delete, probably not. Address, yes. So, you hit, you nailed it. Try to take something off the blockchain. Try to take something, erase something from the internet. Really hard to unring the bell. Arguably impossible.

But that doesn't mean that you can't publish the contrapositive information, and it doesn't mean you can't train your models to recognize lying liars lying. Veracity adjudication is something I've spent a tremendous amount of my career building approaches to looking at the truth, the whole truth, and nothing but the truth. There's three different dimensions of, how do I know what's true?

And we can do all of this stuff. There's lots of ways to address that problem. Unfortunately, one of them isn't claw it back from history. Even the Wayback Machine has limitations.

Michael Krigsman: Chris Davidson comes back and says, "Do systems currently exist that are successfully integrating today's LLM-based gen AI with active inference, and what might that look like? Are they completely different from one another?"

David Bray: There are early stage, but I wouldn't say there's anything that we can publicly point to. I don't know if Anthony has anything we can publicly point to. I know there are things going on behind the scenes.

Anthony Scriffignano: Yeah. I... That's why I'm smiling. No. There are not.

David Bray: Nothing public. Let's say that. But it's a good question.

Michael Krigsman: Chris Peterson is thinking very broadly, and he says, "Should we be thinking of AI like the old, old-school arms race, US versus China versus high-end chips to UAE, since it can be done by anyone? Could there even be an AI version of a non-proliferation treaty?"

Anthony Scriffignano: No. We should be thinking of AI physics in the 1940s. Physics was there before... Nobody discovered physics. Physics was there, and people got good at physics or not good at physics, and they shared what they were learning, or they didn't share what they were learning. And they used it for good or not good.

It's a lot more that than it is something that can be somehow corralled into national boundaries. I'm sorry, David. Did you have a different opinion?

David Bray: I am spot on to what Anthony was saying. I tell people it's just math. It's very fancy math, but it's just math. I would also say, let's learn from history because back in the late 1800s, all the nations of Europe got together and said, "Chemical weapons, really bad. Let's ban them." So in 1889, they banned them.

The Evolution of Work and AI's Role

And come World War I, guess what? Both sides had been quietly researching how to do it, and they had done it in quiet. And so anyone who thinks some international agreement will save us from ourselves, I hate to say it, no. But again, you have to recognize it's physics. And so we have to come up with other answers.

Anthony Scriffignano: I heard a great quote yesterday, and I wish I knew who said it. The big difference between genius and stupidity is that genius has limitations.

Michael Krigsman: Arsalan Khan points out that, humans collect and create data, and then humans are replaced by AI that uses that data. Accenture just laid off 11,000 people because of AI, and this is an increasing trend. Comment very quickly.

David Bray: It's hard to distinguish what is a case of people being laid off because of AI, what is a case of the nature of business changing. We also live in a shareholder economy where right now shareholders will actually reward companies that reduce their costs. And the number one way to reduce your cost is workers. Now you can say it's because of AI, but it might just be I'm trying to reduce my cost.

And so the nature of work is changing and will continue to change, but to say it's because of AI I think is an oversimplification.

Anthony Scriffignano: And AI can make a pretty good PowerPoint deck, and then all we have to do is edit it. So I don't need a room full of people to get me 80% there.

Michael Krigsman: But the reality is that AI is being super helpful in many different areas. And Vibe coding, for example, has its role. For example, I'm not a developer, but I can use Vibe coding to create an app that is useful. And there are now tools that let you integrate... Lovable lets you integrate with the backend, and it will create the database and the security roles and so forth. And you know what? I can use that tool internally for my own organization. I can even sell it.

David Bray: More and more people will become solo entrepreneurs. I think that is the shifting nature of work. And so we are definitely seeing a work change.

AI's Positive Impact on Healthcare and Society

I want to give some people some hope though, too. One of things I'm seeing that's really an interesting trend when it comes to data and AI is more and more people with rare diseases or a family member with rare diseases are actually asking for their medical records. And they're actually then using the tools.

And Health Level 7, which is a nonprofit that actually makes sure you can request those records and they're in a way that you can actually analyze, you can now actually get your medical records in a digital form and then go to your whatever AI you want to and say, "What has this doctor missed?" Or, "What else should I be thinking about in terms of treatment for myself or my family members?" So that's a positive sign.

I do think we need to help people cross the chasm between the work they used to do and the work they're going to do, whether that's community efforts, whether that's company efforts. The nature of work is definitely gonna change, but the nature of work has been changing where, I mean, most of us would have been farmers 200 years ago, and we're not anymore.

Anthony Scriffignano: My greatest hope, Michael, is that anyone that is disintermediated or feels disintermediated by AI doing their old job feels liberated to go do something new and more amazing because there's plenty of... David's talking about, you know, known unmet needs that we know we're not addressing these problems. There's also those unknown unmet needs that we haven't even discovered yet. Oh, my gosh. Smart people, please go do some of that instead of rebuilding a better, you know, whatever.

Michael Krigsman: What do you say, Anthony, to the person who is being displaced, who has kids in college, who has medical debt and is living paycheck to paycheck? You know, telling that person, "You should do something new," is not very helpful.

Anthony Scriffignano: I said that was my hope. That is certainly not the way I would approach that person in that particular situation at that moment. And don't think I don't get calls from people that are in exactly that situation. I'm sure David does. Almost, several a week, you know.

And I start the conversation. There's a fine line between being helpful and being annoying. But I try to start the conversation by understanding, you know, the grief and the frustration and the anger as best I can, because it's not me enduring that.

But then from there, once that person gets beyond that, very often the people that we're talking about, when you look back on it a year later, it... That was the best thing that ever happened to them because it unchained them to go do this other thing.

So, the journey is how do you figure that out? What do you have? There's that moment in The Princess Bride where they say, "It's hopeless. We can't attack the castle," until they realize they have a cloak. Oh, now we have a plan. What's your cloak? What is the thing that you have that creates the rock that you can stand on.

And I know, you know, you... It might be that you have to go do something you don't wanna do in the short term because of all those needs that you have. I am not at all unsympathetic to that. I am in no way saying, "Let them eat cake."

But what I am saying is that all of this abundance of capability that we have that is right now largely being used to create a bunch of party tricks could be used to do some amazing things. And, boy oh boy, you get to do some of that.

David Bray: You want to acknowledge and recognize the grief and the anxiety that anyone experiencing that is having at that moment.

Hope and Resilience in the Age of AI

And then at the same time strive to find ways to help them see that they can be a survivor and a thriver as opposed to a victim as to what happened here.

I think that also there's a requirement on all of us who are not in that unfortunate position to advocate what is the communal response? What is the response that we expect from people? Because if we're not advocating for it, it will happen to one of our friends or one of our family members, and I know it. I mean, it... I probably do get about... A call about once every week or so.

And obviously I'm here in DC where a large number of people might actually be facing the same situation, not because of AI. But I put that out because we do need to think about in this era of displacement, do we want to... Can we find ways, we should find ways to reduce the anxiety and to help people navigate this? And I feel right now, there's more we can do as societies, as communities, as countries.

Michael Krigsman: On that, do I say... I can say empathetic note. Can I say hopeful note? I'm not sure here.

David Bray: Hopeful. We're gonna... Well, maybe we have to revisit this and "What do we do next?"

Anthony Scriffignano: Yeah, I think we're both... I don't wanna speak for David, but I think we're both hopeful that, you know, humans are amazingly resilient and amazingly good at, you know, turning corners and doing things that surprise and delight us as well as terrify us. We get to choose which of those we do, and I'm on the camp of surprise and delight.

So we'll get there. This is a time we will look back on and say, "That's exactly when we were able to start doing," fill in the blank. Maybe it's Vibe coding, Michael. I don't know. But it's probably not.

David Bray: Or maybe it's teach yourself to be an AI red teamer, which I would recommend instead. Yeah. But that's me.

Anthony Scriffignano: Yeah. Yeah. Yeah, I mean, you know, far be it from me to express an opinion. But I think that, we are certainly at a point of inflection. The most difficult thing about change is it's very hard to notice when you're part of it.

Michael Krigsman: Can I add one final comment that I think we can all agree on? Down with AI slop.

David Bray: Yes. The trick though is identifying what is AI slop. But anyway, that's a definite.

Michael Krigsman: And on that note, a huge thank you to Dr. David Bray and Dr. Anthony Scriffignano. It's been a great show. Thank you both again for taking your time to be with us. I'm very grateful to you both.

David Bray: Thank you, Michael.

Anthony Scriffignano: Thank you very much.

Michael Krigsman: And an enormous thank you to the audience. What will happen next is we will edit this video, do some light editing, create a summary. We'll put it on the CXOTalk website. And then with the transcript and the summary, you will have a document that you can keep, go back, treasure, and refer to. So I urge you to do that. Subscribe to the CXOTalk newsletter.

Thanks so much everybody. Thanks to Dr. David Bray and Dr. Scriffignano. Take care everybody. Have a good one. Bye now.