Quantitative Marketing:
Data-Driven Strategies and Challenges

Michael Krigsman: Today on Episode #787 of CXOTalk, we're discussing data and intuition, the role in marketing. We call that quantitative marketing. 

What is quantitative marketing?

Our guests are two folks from the Columbia Business School at Columbia University in New York City. Oded Netzer is the vice dean for research and a professor and author of the new bookDecision#_Toc135932742#_Toc135932742s over Decimals. And Amy Jaick is the chief marketing officer of the institution.

Oded Netzer: For me, marketing is the part of any organization that is in charge of facilitating and enabling a successful transaction with the customer. Successful, by the way, could mean profit if it's for profit. It could be engagement, for example, if it's nonprofit.

The ones who are in charge of making sure that such transactions, relationships do exist, and now you want to layer on it the quantitative part of it. In order to create these successful interactions, I really need to understand and affirm who is on the other side. it's not executive other side; who are my partners, my customers?

For that, I need a lot of data and a lot of information about my customers and who they are, and a good understanding of who I am, what are my capabilities. That again requires data and analytics.

Importance of data and analytics in understanding customers

Specifically, the analytics is often done in order to try and understand this match, the match between the customer and the firm and what the firm has to offer and even changing what the firm has to offer to make sure it matches what customers want. Particularly in recent years (where a lot of the effort went towards) is getting a better and better understanding of the customers because the better and the more we understand the customers, the better we can actually serve them what it is that they want. That requires a lot of data, a lot of analytics, and a lot of quantitative marketing.

Michael Krigsman: Amy, what is the impact of this data-driven approach on your work right now?

Amy Jaick: There was a period of time where it was somewhat difficult to explain to senior leadership the value of marketing. The increase in the data that we had, the ability to show quantitatively and to connect an action that marketing has taken to a bottom-line impact, that's what a lot of the data has done. 

It has allowed us to be better marketers. We understand a lot more, as Oded was saying, about the audience. 

The power of personalization in marketing

We can engage in different tactics and strategies. Personalization is so important. We are inundated with material, and it is very difficult for us as humans to have that much flying at us and to be able to figure out what to pay attention to. 

Cognitively, if there's this influx of information, we have to pick. We have to sort. We have to think about what makes sense for us to pay attention to. What limited time do we have? And how do we react to that?

In mass marketing, sometimes you can get that benefit with your audience. But more often, what we're seeing is that the more personalized that the marketing is, the more likely that person who is otherwise inundated will take a moment, will stop, and will engage with you. 

The only way to do that is to be able to leverage the data that you have to say, "This is the right course for this person. This is the right path. This is the right next step for person A. But for person B, it's going to be a little bit different."

Every time that our audience or our consumer or customer has that experience, the more likely they are to come back because there's a benefit for them. They're getting what they need. They've seen that it works. They know it was worth their time. 

And so, I think data both helps us on the strategy side to develop better strategies. It helps us on showing results to senior leaders, to shareholders, to stakeholders. And it helps us to deliver a better experience to the customer. 

They're all wrapped up in one larger marketing effort, but I think those three components (and thinking about in those ways) is really helpful.

Oded Netzer: I've been teaching the marketing core for quite a few years here at Columbia Business School. It's probably the course that has changed the most out of any core course we teach. The reason is that the world of marketing is changing tremendously.

If you think, for example, about advertising, advertising, again, many of us have watched Mad Men, right? That's really the old days of marketing. But even up to the early 2000s, the center of advertising was here in New York on Madison Avenue.

If you think about what the three biggest advertising companies in the world are, they are not anymore in New York and they are not anymore advertising companies. They are Google, Facebook, and Amazon. These companies sit in Silicon Valley, and they sit in Seattle, which means they are technology companies, and with that come the quantitative aspect of marketing.

Evaluating return on investment (ROI) in marketing

Michael Krigsman: Amy made a very interesting point. She spoke about the enhanced ability to connect the marketing activities back to impact on the organization. Can you talk about that because that's what business leaders ultimately care about, right? We're spending a certain amount of money on our marketing. What is the ultimate impact, and how do we measure that? 

Oded Netzer: One of the biggest impacts of quantitative marketing on the world of marketing was this notion of ROI (return on investment). The famous quote, "Half of my marketing expenses, I will spend. I just don't know which half."

Then came the world of online marketing. Then came the world of digital marketing. And with this promise of, "We're going to see what people clicked on. We're actually going to see whether someone clicked on the advert or someone eventually even bought."

Finally, we can get an ROI on marketing. We know what is the impact of our marketing on consumers. 

I think it's true. We have moved a long, long way in the ability to show a return on investment in marketing. It's much easier now to have that conversation, and I think that marketeers that haven't taken that step have truly been left behind and still struggle within their organizations. 

I will also, though, mention that that promise may have been even too strong. We've now started seeing the pendulum shifting on that statement in the sense that the fact that I see where their customer clicked on an ad doesn't yet fully tell me whether it's a full return on investment. It could have been that this customer would have bought anyway, so we need sometimes more sophisticated quantitative techniques in order to truly ask questions like attribution or incrementality whether we truly have ROI on these. 

It seems as if we solved the problem and now we realize, the problem, we are way, way ahead, but there are still things to be done, which is where the world of quantitative marketing works today.

Amy Jaick: A few years ago, a marketer had to say, "I see a correlation, but I may not be able to show causation." Now, I think you're right. There still are numerous questions that we have. There are still challenges with attribution models. 

But I think we are much closer to being able to say, "It's not just happening at the same time, and isn't that nice." We're able to at least connect often some of those dots and say, "Well, I know that when my marketing spend goes up 13%, 27% of customers purchase more," or "My marketing spend goes up 13% and 50% more individuals entered this online store."

We're able to do it in a way that, if our attribution models are right and we have the right technology, at some point in the path we can say there was a direct connection and we know that it's actually going to make a difference. I think directionality is important, but directionality with some very specific models that show that relationship is what ultimately gets you farther in terms of your budget, in terms of being able to enter into new markets, and really buying into sort of the confidence of the C-suite, I think, which is super important.

Oded Netzer: I think one of the things that truly helped with that (particularly in the world of digital marketing) is the ability to run these A/B tests, right? 

Amy Jaick: Yes.

Oded Netzer: I mean it's very difficult to say, "Half of my customers are going to see one store, and half of my customers are going to see a different version of my store." My store in my store is very difficult to play with. In the online world, it became very easy. 

The role of A/B testing in quantitative marketing

I split my time between Columbia and Amazon. I spend some of my time at Amazon Advertising. 

The practice at Amazon Advertising is running these A/B tests. Let's try version A, version B, and when we do that, we truly can get to ROI. We truly can get to what is the difference between offering the customer version A and version B.

It's still easier to do in a digital environment where we can take... We have 100,000 visitors. We can split them in half or any way we split them randomly and show two different customers different offerings and see how it works. 

It's still a little bit more difficult in the offline world. But even there, we are making progress or thinking in a language of A/B tests, in a language of testing in order to measure ROI and in order to measure truly what customers want. 

Amy Jaick: I'll give you just an example of that. Several years ago, I was working at a large company. My role and my team's role was focused around telling investors the story of the success of the business. 

When we had our hypothesis about which creative would work the best to get them to engage with our financial results, I would have thought one thing. We A/B tested over a period of about a year, so that's four quarters' worth of data. 

We actually found that in telling the story with our IP, so the shows themselves, when we used that as a creative, investors – now, this isn't an audience that you would naturally think would pay more attention to a character on a show than a Financial Times headline – when we talked about our efforts and when we used the creative, when we used our IP, we found that the interest and the engagement in the information, the financial information about the company increased exponentially.

I think, to your point, that's an A/B test that (10 years ago or 20 years ago), one, you wouldn't have been able to run but, two, you wouldn't have hypothesized that. But when we just have the ability to continue to test and swap in and swap out, and we see very clear results in the data, it gives us confidence to both try new things. But then it gives us security in the decisions that we make because we know now that we can see that happening.

Michael Krigsman: Please subscribe to our YouTube channel and hit the subscribe button at the bottom of our website so you can get our newsletter and we can keep you up to date on upcoming live shows.

Data collection in marketing: Insights and challenges

What about the data? What kinds of data should marketers be collecting? What's the best way to get that data? Oded, can you take us a little bit behind the scenes of the kinds of models that you're looking at? Again, my real interest here is the impact of that data and those models on the real lives of marketers. Hey, it sounds like a reality TV show, right, The Real Lives of Marketers. [Laughter] 

Oded Netzer: We have a firehose of data, right? We have more data than we ever had before. It's still the situation that we don't always have the data we exactly need. We're sometimes looking for the right data. 

There is something we talk about in the book that you mentioned earlier that we call the certainty myth. There was this belief that finally, when we have all of this data, we'll get to certain decisions. 

Certainty is a myth, and we still make decisions under uncertainty, but we do have much more data than we ever had. Specifically, the data that marketeers value and cherish tend to be data that tells us about customer preferences, being able to measure and understand customer preferences. 

The reality and the reason why it was difficult (it maybe still is difficult in some ways) is because of heterogeneity because customers are so different. There are no two customers that are the same. 

I always tell my students, and I always find within my class these types of students, those who treat the Apple Store like a cult and those who could never be caught dead in an Apple Store. Right? The marketeers need to understand this, need to understand this often with limited data depending who you are. 

If you're Google or Amazon, you have a lot of data. If you are a media outlet, you may have less data about the customer and who they are. 

We tend to see, unlike—

If you go to data, for example, in finance, I can always go one more year and have longer data about stocks, right? I can also just do an analysis of the entire market (when it comes to stocks). 

When it comes to consumers, I cannot do data just on the entire market because of the heterogeneity that I'm talking about. I do need to understand each customer and their own preferences. 

And I'm limited with respect to the history because, if I'm talking about travel for example, if I'm Expedia, I only see you five, ten times. That's all I have. So, I need to work with that length of history of what I observe about the customer. That's where economic metrics helps or statistics. 

There is a tradeoff when it comes to data between how good your data is and how complicated your model needs to be. The better data you have, the simpler the model you can use. 

If we ran the A/B tests that I just talked about before, all I need to do is compare averages. The customer in the control bought that much. The customer in treatment bought that much. That's it; the level of math that a sixth grader could do. 

If I need now to build a model where I am again Expedia and I'm trying to understand customer preferences from this visit and the previous visit and so on, that's where I need a much more sophisticated type of modeling and type of analysis. 

Working with structured and unstructured data in marketing

The other distinction I want to make about data is the distinction between structured data and unstructured data. Until fairly recently, circa 2010, most of our analysis was done on what is called structured data. 

Structured data are numbers, data that comes in the form that we, by the way, generally think about data in Excel or in a table with numbers in it.

But if you think about the majority of the data we have (as business people, as marketeers), it's actually unstructured data. It comes in text, image, video, audio. A customer calls us to the call center.

We have data in terms of tens of thousands if not hundreds of thousands of ads that we can analyze. Ads are images and text. We have company reports. 

Different estimates, depending on who you ask, but 80% to 95% of data available for business is actually in the form of unstructured data: text, audio, video, and image. It is only since 2010 that we actually know how to analyze this data at scale. 

Unstructured data existed probably since the Ten Commandments were written on a stone. But really, the ability to analyze this data at scale came with machine learning methods type tools. And we're seeing more and more companies are using it. Of course, recently, the whole ChatGPT and generative AI is an example of leveraging unstructured data.

Michael Krigsman: Amy, you're a marketer. I'm assuming that you are not an expert statistician. 

Amy Jaick: Oh, no. [Laughter]

Advice to marketers on approaching data strategically

Michael Krigsman: And data scientist. [Laughter] And so, given what Oded was just describing in terms of the quantity of data, the quality of data, what can marketers do to take advantage of these important points but without having to be a deep-level statistician, for example? 

Amy Jaick: Don't be scared of data. I'm not a statistician, as you very correctly pointed out. I never have been, but I think, for marketers, when we talk about quantitative marketing or we talk about data-driven marketing, for some, especially individuals early on in their career, it sounds like they're going to have to be gathering the data sets. It sounds like they're going to have to be building the models. It sounds like they're going to have to be interpreting. 

I think we're very lucky in that we have a whole generation of individuals who understand data, who are data scientists, who are insights and research-driven colleagues. The more that we can develop very close relationships with those individuals as marketers, the better because most marketers I know aren't, as you said, statisticians. But what they do know is how to look at data—I know we're going to talk a little bit about this—and say, "Well, based on my understanding of the audience, this seems odd. Let's explore that."

They have a question that they know they want to answer, and they know that the data is an important part of that. And so, they can ask for help in building the model.

The first is you don't have to be the expert. Don't be scared. Don’t not use it. Use it. Find a partner. Find a colleague. Find somebody who is the expert. 

The second is to think about the data in a meaningful way. As Oded said, there's infinite data available, so it really is about what you are slooking for.

I know Oded is going to talk a little bit about this. When something doesn't feel right, ask more questions.

A good example of using numbers and qualitative elements happened a couple of years ago. I worked in a publishing company. It's very common for people to unsubscribe from subscription-based businesses, including magazines.

You build a model, and you know that churn will be X based on historical data. You know that there might be some macro or micro conditions that will impact that year's churn. You go out, and you acquire customers knowing this percentage is going to leave, and you need to have X percent of new people coming in and subscribing.

But as a marketer, it's not just about making sure that you can fix the revenue equation on both sides. It's also about understanding your customers. 

You have this model. We had this model. We know that X percent are unsubscribing. And so, then you have to dig deeper to think about why. What is it? What's the reason that they've chosen to cancel this subscription? 

We could get a little bit of insight in asking structured questions, and our hypothesis was, it was a value situation. Somebody decided that that product, that the magazine, was no longer worth what they were paying. 

It feels pretty simple, right? But we actually had done some brand research and had some qualitative components. What the qualitative components told us was that actually it had nothing to do with value. It had nothing to do with the product. It was about unfinish ability. 

People felt that this particular magazine was worth the price, had great value, but they couldn't finish it, and they would stack it up in a pile. They would say, "I'll come back to it," and they didn't. Every time they looked at the pile, it grew. And their guilt – so this is an emotional reaction – their guilt over not finishing the magazine was actually what was contributing to them canceling the subscriptions.

That insight is not insight you were ever going to get from structured data. It's not even insight you were going to get from some of your survey questions. It came exclusively through some qualitative conversations. 

And there were some solutions to that. Digital combated the problem a little bit because people can't finish the Internet. If you can't finish a digital issue of a magazine, you didn't feel as guilty. You can introduce newsletters and quicker bites of content and summaries and things like that. 

But I think the other important thing, and I want to really reiterate this, is that there's the structured data. There's everything you know. There's quantitative. And then there's this other qualitative element that is equally as important for marketers to pair together. 

Oded Netzer: I think it is important that we think about data in the most general sense of the word data. 

Amy Jaick: Yeah.

Oded Netzer: I mean a conversation with a person is data. 

I just maybe want to touch on one thing with respect to when I build a model, a statistical model, and I truly mean it. I would actually go to Amy first to tell me this model is right before I would go to a statistician to tell me whether they think my model is right because the true value comes from, let's say, it's a model that predicts click through on an ad, and the model showed 20% clickthrough rate.

A statistician wouldn't know to tell me that a 20% clickthrough rate on an ad is unheard of. Unless your ad truly offered money to customers, there's no one in five customers clicking on an ad. 

Amy Jaick: [Laughter] Yeah.

Oded Netzer: I needed marketeers. I need Amy to tell me, "I don't know what you've done wrong, but I can tell you this is wrong because it's impossible that customers, 20%, one in five customers would click on your ad."

Michael Krigsman: We have a couple of questions that have come up on Twitter. Actually, two folks, Chris Petersen and Arsalan Khan, who are both regular listeners of CXOTalk—thank you, guys—are asking independently essentially the same question, and that is, "Where do we draw the line for privacy in terms of data-driven marketing?" That's from Chris Petersen. Arsalan Khan says, "How and who should strike a balance between collecting data that can affect privacy and who decides what's private, what's not?" and also then you have the issue of biases.

Can we talk about that? Then we'll switch over to quantitative intuition. 

Oded Netzer: Privacy is an extremely important topic to think about, to discuss, and to realize that companies have a huge responsibility when we are collecting individual data. I think there is the issue of data being shared, and then there is the issue of data being used by the company.

Despite what actually most people think, most (at least the large companies) do not sell data. Now, I'm not sure it is, by the way, being done because of pure ethical issues or because data is just way too expensive to sell. The use of data by a company – again, think about the top companies like Facebook, Google, Amazon – is so important and useful for their own purposes that actually selling it would not be very wise. 

Generally, at least from these companies, data generally do not flow around. Generally, it does tend to stay within these companies. But even there, they should treat it with the responsibility that these data should be treated with. 

That's where recent regulation came about first-party data where you're allowed to use your own data, but you are not allowed to actually use someone else's data, which by the way, in and of itself, there is an interesting twist there. 

On the one hand, it sounds right. It indeed protects customers in terms of their data floating around. But it also increases maybe the gap between the top companies that have really good first-party data, their own data, and the smaller companies that don't have their own data. That's where they could have closed the gap by getting data maybe from someone else. 

I do think that we shouldn't let the cat guard the milk there. I think we do need regulation there. Europe is much better than the U.S. in that sense. 

We have made that mistake previously with social media where we let the companies guard themselves. We've made it with privacy, and I think we need to move there.

I'm happy to see that with generative AI, this conversation is starting already now and not once this is fully out there. Amy, I don't know if you have other thoughts.

Amy Jaick: Yeah. Going to an even larger question is who owns the data. We're now in a world where our behavior has been tracked for a very long period of time and there are reams of data. 

People probably know that I prefer baked macaroni and cheese over Brussel sprouts, but who doesn't? You know that data is gone. It sits with somebody. I hope I get an ad for macaroni and cheese. I will let everybody know [laughter] shortly. 

I think there's a larger question as we think about a next generation. Let's talk about kids, for instance, for whom maybe there's some data. There's not a lot. 

Who owns that data? Who should own that data? Is my child the holder of that data, or is it the platforms on which they're engaging? 

There have been these really large, high-level conversations around should I be able to own my data and I choose where, when, and how to monetize that? Or is there a tradeoff? I get to go and watch catch videos or whatever videos that I like and, in exchange, I'm giving somebody an insight. And I know that because I understand that's the way that a lot of practices work because I'm getting a popup that says there are cookies, and I am willingly engaging in a behavior.

I think there's both what is happening now and then how do we think about data, the value of data, and ownership of data, for a whole new generation. 

A little bit off-topic but, I think, an important conversation for any of us who are using data and whose data is being used and traded in some way.

Oded Netzer: We need to move to much more of an opt-in environment where at least we make the decision of whether to share our data.

What is quantitative intuition?

Michael Krigsman: Obviously, this is a crucially important topic, this issue of data privacy and data ownership. We're not going to solve that one now, but what we can get a better understanding of is, Oded, your concept of quantitative intuition. What is that, and why does it matter?

Oded Netzer: Quantitative intuition, I think this is particularly important in the world of marketing where, again, there is the more, if you will, unstructured data that Amy talked about or the conversations that we have. 

We tend to use our intuition regularly in our private lives. We talk about surprising things and interesting things. Sometimes we even call them gossip. But we are very weary of using our intuition when it comes to the world of business, to our work life. 

The idea here is not to just trust your gut. The idea here is, as you are approaching data, bring in your judgment into it. In fact, we have to bring in your judgment into it. 

On its surface, quantitative intuition sounds like an oxymoron. But I, in fact together with my co-authors for the book (Chris, Frank, and Paul Magnone), we believe that not only that they are not oxymorons. Particularly at the level of leadership, it's the only way. 

Incorporating quantitative intuition into business decision-making

The only way to make decisions is to combine the data with a good sense of judgment. When I say judgment, I mean intuitional judgment. This could come, actually, in at least three stages of the decision-making process. 

In the questions we ask, in being very clear about what problem we are trying to solve. More often than not, we are just going on a stray thing, "Oh, we have been collecting all of this data. There's got to be something interesting in it, right?"

We need to guide the process. To me, one of the biggest no-nos in data-driven decision-making is we expect the data to provide both the questions and the answers. 

We should ask questions. Then hopefully, the data can provide answers. 

In fact, we've learned it better than ever since November of 2022 with ChatGPT. We need to provide good prompts. If you want to get good answers, we need to ask good questions.

Secondly it is something you already talked about. How do we interrogate data? Exactly what I mentioned that you do need the context. 

As humans, we are very good in context. Particularly, if you are an expert in your domain, as Amy is in marketing, it's much easier for her to look at the model of very sophisticated analysis and interrogate it not from the P-value or from you didn't use this three-letter acronym or another, but from this number doesn't make sense. I don't know what you've done wrong, but I can tell you that it doesn’t make sense.

Finally, and a crucial place for judgment, is in the synthesis of the information. Generally, data and analysis will tell you the what. It would not tell you the so-what (what we need to do), what does it mean, and then now what are we going to do about it.

These are different components where we want to combine the quantitative together with intuition. 

Amy Jaick: I use the so-what and the now-what all the time (after you taught me that). Those two questions for marketers, truly, will change the way that you approach your data and your strategy, so thank you. 

Oded Netzer: In fact, we said that one of the reasons we went on this journey of quantitative intuition was, we were simply tired of meetings, of horrible meetings, meetings that go on the what.

Amy Jaick: Yep.

Oded Netzer: We'd be at the meeting, and someone shows data. Now we're going to spend an hour talking about the what. 

In fact, there is a persona we talk about in the book. We call them the Seemores. The Seemores in the organization are the ones who, in every meeting, have only one comment, "Can I see more data?" 

Amy Jaick: [Laughter] 

Oded Netzer: We could have meetings over meetings and postpone any decision forever by asking for more data but digesting and slicing and slicing again the data. Moving from the what, right? 

Amy Jaick: Yes.

Oded Netzer: From what's in the data to what does it mean. Right? What are the implications?

Again, this is a place where people are fearing a lot with ChatGPT and so on, with generative AI. 

What's left for us as humans? If all you do is the what, yeah, you will be replaced by a machine. But if your focus is on the so-what, on what does it mean, on the now what are we going to do, not yet. I'm not saying that machines will not get there at some point, but not at least in the near future.

The role of artificial intelligence in marketing: insights and limitations

Michael Krigsman: Arsalan Khan comes back, and he says, "As AI continues to be used more and more, do we really need marketing people if AI can understand the context around that data?"

Amy Jaick: Yes. [Laughter] Definitely, you do. You do. 

Michael Krigsman: [Laughter] 

Amy Jaick: I'm sure Oded has a lot to say as well. 

Where we are in terms of ChatGPT right now, I would say, it is a helpful assistant in many ways. Having used it myself, my team uses it, it gets you – I'm making this number up – a third of the way. 

But it's not replacing what we're doing. It's not replacing the knowledge that we have. And it doesn't always get the context and the nuances that are very specific to humans. Then even more kind of granular or smaller, there are more subsets of nuances amongst different audiences.

Now, you could argue all of that is data and, over time, maybe those things could be put into a formula. Maybe it could be understood. But I think we're still a bit far away from saying, "This will resonate because," or from understanding that word that means something in this context can be taken in an entirely different way in that context, and that context is problematic.

Yes. Is it helpful, for instance, for summarizing some research that you might have? Absolutely. 

Does it give you something to gut-check your approach? Can it provide more information? Yeah, it can. But we're not at the point where it could run a sophisticated campaign for you and get equal or greater results. 

Oded Netzer: Yeah, and I think, Michael, you really hit it on the head with the word "context." I think this is the key. 

The key is truly context both in terms of, by the way, how these methods have improved. The difference between GPT2, which was the previous version, and GTP3.5, which is ChatGPT, or now we're already at GPT4, is context. What I mean by that is in order to predict the next word or in order to interpret a word, understand a word, what these tools do, they take the previous that many words to understand the context of the word. 

The reason why – and it truly was, I mean ChatGPT was a huge leap in innovation over the previous versions – is literally because of context, because they use what is called 8,000 tokens. Think about 8,000 tokens, something like 6,000 words because it includes periods and so on, the previous 6,000 words to understand a particular word. The previous versions, for example, GPT2, used 2,000 words to understand any particular word. 

As humans, we are tremendous, actually, in doing that, in context. I'll give you two examples.

The first example is when I say the word "model." In fact, Michael, when you use the word "model," both of us understood that, in this context, we are not talking about the fashion type. We're talking about the nerdy type, right? 

Amy Jaick: [Laughter]

Oded Netzer: Because we have enough context to understand that, in this conversation, unless I'm going to give you a real clue with my previous words that we are talking about the fashion type model, we are talking about the nerdy type. Right? 

In fact, we don't even understand how good or how our brain actually understands context. I'll give you an example for that.

Think about a toddler, a 1.5-year-old, does not have the intelligence to speak yet. And we show them an illustration in the book, and the book has an illustration of a cat. And we tell them this thing does, "Meow." Then we show them an illustration in the same book of a dog, a book about animals, as we often read to our 1.5-year-olds, and we tell them, well, this thing does, "Woof." 

Then we show them maybe four or five times an illustration, different illustrations from different books and so on. The next day, we're going to take them to the street. They cannot speak yet. They are going to see a cat or a dog, a real cat and a dog. The first time ever, they see a real one, and they're actually going to meow and woof. Again, they can't speak yet, but they do it with five observations. 

Amy Jaick: Mm-hmm.

Oded Netzer: A few years back was the first time that machines went close to human level in detecting cats from dogs from images. They have done it because a researcher made available two million observations stacked by humans.

Amy Jaick: Two million. 

Oded Netzer: Of cats and dogs. How can a human do it with four or five observations and a machine needs two million? We don't understand how we do it with only four or five observations. If we did, we would have trained machines to do it because it would be very useful to have less data. I think that gives you the understanding of why, when it comes to the so-what and the now-what, we don't have enough data of similar so-what and now-what. 

Maybe if you are a physician and we have enough decisions of the given symptoms, maybe it works. But for a typical business decision, which doesn't come that often, it's unlikely. It's unlikely, at least not with the current tools and where we are today or even in the near future, that machines would have enough context, these 8,000 context but repeatedly with enough data, to truly go to that step. 

Again, it may happen at some point. Not yet.

If you want one place where it did happen already, and happened before generative AI, online advertising. The location of ads, of which ad I'm seeing when I go to Google or when I go to Amazon or Facebook, is almost fully automated (apart from the choice of the creative of the advertiser).

Amy Jaick: Yeah.

Oded Netzer: That has been already automated because the rules are fairly clear. There isn't, "Oh, so what does it mean?" Well, it means that this ad is likely to lead to a higher click-through rate than another ad. That's a predictive model. Machine learning can do that already. 

Michael Krigsman: How can marketers avoid being data myopic, which is to say losing sight of the fact that business decisions involve people, circumstances, we use that term context, and not just numbers? 

Amy Jaick: When marketing has a seat at the table and is involved in the strategic decisions, things like growth or new audiences or understand where value is derived for a company, that's when they can put together the pieces and they're not only seeing their small part. I think being part of having a marketer truly, truly involved at the senior level with a seat at the table who is just as involved as anyone else in the growth of the company is really important because it then becomes top-down.

You ask your team not to look at this slice. You ask your team to think about how it connects to something larger.

How to combine quantitative marketing techniques and intuition for effective decision-making

Michael Krigsman: Oded, how can organizations strike the right balance between relying on quantitative marketing techniques versus trusting their own intuition and experience (bringing the two together)?

Oded Netzer: Ask yourself. As you're looking at data, ask yourself what surprised you. It's amazing how this fairly simple, deceptively simple question often cuts straight to the chase.

You're either finding a mistake or you're finding an insight. Either way, you benefit.

In other words, don't trust your gut. Trust your doubts. Look for these surprises in the data.

Michael Krigsman: Amy, you're going to get the last word here. What advice do you have for marketers given everything we've just been talking about?

Amy Jaick: Your best friend is the person who can build the models. You need to be embedded in business decisions to drive true value.

And I don't think that ChatGPT will take our jobs. But if you don't know how to use it to make yourself more efficient or more effective as a marketer, it will. It's not a substitute, but it's an important complement that we all have to be using and working with on a daily basis.

Michael Krigsman: Can I accurately paraphrase that as saying, "Be part of the business. Understand the tools. Understand the data. And bring all of that together to make the decisions that rely on both the data and your experience with the business and with the context"?

Amy Jaick: Even better.