"Moneyball" for Movies:
Data and Analytics at Legendary Entertainment

Michael Krigsman: Okay. You go to a movie. You're totally enthralled and engrossed in the movie, and you think the acting is great. But, did you ever stop to think about the data, the analytics, the machine learning, the predictive analytics that go behind all of it to make it happen? Today on Episode #276, that's what we're talking about.

I'm Michael Krigsman. I'm an industry analyst and the host of CxOTalk. I want to say a quick thank you to Livestream, which provides our video streaming infrastructure. Those guys are great. They support CxOTalk. If you go to Livestream.com/CxOTalk, in fact, they'll give you a discount on their plans.

Now, before I say anything else, I need to ask you in the most heartfelt way to please tell a friend and like us on Facebook. Not only that, get them to like us on Facebook. For extra points, subscribe on YouTube.

Without further ado, I am so thrilled to introduce our guest today. Matt Marolda is the chief analytics officer at Legendary Entertainment, which is the blockbuster movie house behind some of the movies that you know of. Matt, welcome to CxOTalk and thank you for being here.

Matthew Marolda: Great. No, thank you. I appreciate it.

Michael Krigsman: Matt, tell us about Legendary Entertainment.

Matthew Marolda: Sure. Sure. Legendary, as you mentioned, is a producer of both movies and television shows. The types of movies we produce are large-scale, things like Godzilla, Kong: Skull Island, The Dark Knight series, movies of that scale, which are intended to be large, what people often refer to as tentpole movies that are big, global events all around the world.

Michael Krigsman: Okay, so you're producing these major films, and your role is chief analytics officer. I think, for many of us, the notion of what a chief analytics officer in this world, in the Hollywood filmmaking world, is probably kind of alien.

Matthew Marolda: Yeah. Certainly, yeah, it was for me. [Laughter] Yes. No, I can give you the genesis of it, where we started from, and what the objectives were. Initially, thinking back, this must have been back in 2012 or '13, in that timeframe. Our founder and former CEO, a guy named Thomas Tull, he came to the realization that there was an opportunity in our space in movie marketing and movie production, in general. That opportunity was to apply data analytics in a way that was different and unique than what other people were doing at that time.

It fell into two major categories. One category was very early in the sense that, as we were considering what films to produce, how to cast them, the release dates, things of that nature, there's an opportunity for data analytics to step in and provide input. To not take over the process, but to be an important kind of consideration along the way. Similarly, and on the other side, we also thought there was a real opportunity for us to be able to add efficiency to the marketing of the films.

The movies I described, I think folks might have a sense as to what they might cost. These are $100 million to $200 million production budgets, which is usually a pretty publicly known thing. Also, somewhat publicly known, but maybe less well-known, is how much we spend on media, which can be anywhere from $80 million or $90 million to $120 million, $150 million worldwide, so these are large bets. These are big bets, and we wanted to make sure that, before we entered in those bets, we were as informed as possible. Then as we took those movies to the world, literally, we wanted to be as efficient in the marketing as we could be.

Michael Krigsman: Your background was in sports.

Matthew Marolda: Yeah.

Michael Krigsman: Essentially, Moneyball. The idea was to apply those techniques to Hollywood filmmaking.

Matthew Marolda: Yes. In hindsight, it's funny because we didn't really know what we were going to do. [Laughter] We knew, at a high level, what it was. We knew we needed to use data and analytics to inform the process. The Bridges are similar in a certain sense. The first thing I said to our chief creative officer when I joined Legendary, and again these are two roles that could effectively be oil and water, creative and analytics. Those could be things that are opposing forces, basically. What I said to him is the attitude that we've had from the beginning from the creative side, which was that analytics, especially in sports, but the same with content, never produced a player, but all it tries to do is put the player in the best position to succeed. That was the attitude from the beginning.

On the marketing side is a little bit different. The marking side was how could we use data and analytics to gain a competitive advantage? On that front, what we realized very quickly was that there was a real opportunity in how we addressed our audiences, meaning the traditional approach, and this is still often the dominant approach for these kinds of movies is what we always call the spray and pray. Meaning, quite literally, spray the population with TV ads and pray they go to the box office. That works in a certain world, but maybe not even the world of today, but at some point it worked.

What we realized back four, five years ago was that we needed to be much more precise. It's a game of impressions, meaning how do we deliver the trailer, the TV spot, or the poster to the right people? Which of those things do we deliver, and in what format? Doing that in a very precise and individual way. What we've built are tools that enable us to, at individual levels, predict people's propensity, as we call it, their likelihood to take the action we want, which may be a trailer view. It may be buying a ticket.

That meant we had to use some very sophisticated tools and techniques that we had to build ourselves. We built up a suite of assets and capabilities that are all rooted back in AI. This was all when AI was not cool. [Laughter] This is a time at which AI was Skynet or something. It wasn't embedded really as broad as it may be becoming now. The reason why we knew we needed to go down that path and to use machine learning, neural networks, computer vision, all of that was the scale which we needed to operate was so massive that, without those kinds of tools, you're almost back to that spray and pray mode where you are quite literally taking broad guesses at large groups of people.

Michael Krigsman: Mm-hmm. This is really interesting. Let's start to decompose this.

Matthew Marolda: Sure.

Michael Krigsman: You've got your data sources. Your goal then is to have these very micro-micro-micro-micro-targeted--

Matthew Marolda: Yes.

Michael Krigsman: --can we say ad campaigns, marketing campaigns?

Matthew Marolda: Sure. Yep.

Michael Krigsman: Okay, so you're targeted these micro-campaigns. You need the data, so you have sources of data. You have these tools and techniques that you've developed. Then somehow, there's some link where you're now applying that to your ad platforms: Facebook, YouTube, wherever it is that you're going.

Matthew Marolda: Yeah. Yeah, so you've got the right ballpark. I'll go ahead and get more specific. The first step in that process for us is to really try to understand people. The best way for us to understand people is with data.

Now, what is unique [laughter], and I tell a story often. It's true, and it's not meant to be anything other than just factual. More or less, the first day I walked into Legendary, my first question was, "Where's all the data?" Again, coming from a world that wasn't connected to Hollywood, I didn't really understand how the dynamics worked, which was, we produce a movie, we deliver it to a distributor, who then hands it to exhibitors. Then the exhibitors or, ultimately, maybe an Apple, Amazon, or whomever, all the transactions, all the customer interactions happen at that level, which are too removed from us.

When the answer came back to me, "Oh, what data do you mean?" I said, "Anything on people," they came back with an Excel spreadsheet of about 50,000 email addresses. I realized at that point that there was a different challenge we had to face, which was, how do we actually get data on people?

I'll put that to the side for a second, but the principle, though, that we were taking wasn't data, necessarily. It was analytics. Our bet was not necessarily on getting the best and most precise data on people. It was, how do we build the analytic tools to take whatever data is available to us and use that to do our targeting. That is a recognition of a lot of factors that I think were true then, but even more true now: privacy issues, social platforms and how they share data and what levels of granularity they'll provide you, regulatory issues, all sorts of things.

The data will shift. What's available to you Monday might not be available to you the next, or new things will pop up that weren't there before. We knew we had to have data. That was table stakes, and so we invested a lot of money, millions of dollars, into data to acquire data on people, on content, unstructured data from social networks, everywhere we could find it.

That wasn't so much the bet. That was the table stakes. The real bet for us came at the next level, which was, what can we build on top of that data? With that, what we drove towards was these AI solutions. Meaning, could we take a billion or more email addresses and attach hundreds, if not a thousand or more, attributes to those email addresses that we created from, sourced them from, some kind of partnership to constructing them from unstructured data, meaning of text, image, and things of that nature? We produced a very robust picture of people.

Then once we had that robust picture, we needed to do something with it. It's inert if we don't act upon it. The next option to take is to use, effectively, that big table of data on people, which is not what it literally is, but that's a good visual of it, and create audiences from it and to make individual predictions. The first step in our process is to use our models. There are many different inputs into them, but to use them to hone in on who we think the most likely audience is.

It's not binary. In fact, we have three major categories that we kind of drill in on specifics. The three major categories for us are people we consider to be given, meaning they're going to watch the movie no matter what. They're wearing the Godzilla T-shirt. They've watched the Kong movie, from 30 years ago or even 10 years ago, dozens of times, that kind of person. There's a small number of them, but they're there.

There's a much larger number of people who will never watch, who are never going to consume this content. That's fine. We don't want to spend impressions on them.

Who we really care about are the people in the middle of those two groups. We call them the persuadables. The people who we can persuade by giving them the right piece of content or the right creative at the right moment through the right channel is key. Those are trite things now. People talk about that a lot, but we try to be very precise about it.

The first step we'll do is take that persuadable audience and define them sort of exclusive of the givens and the nevers. Then, within the persuadable audience, we will effectively score every single person. In the U.S., for a movie of a scale we typically would work on, it could be 40 million or 50 million people. They'll get a score from zero to 100, literally, 100 being very likely, zero being very unlikely.

Once we have that, where we can, we deploy media to them specifically and individually. A lot of people use the term onboarding. We might onboard them into, say, programmatic buy on websites, publishers that you would see on the sidebar or across the top. That includes social media. That includes search. That includes video like YouTube. Wherever we can find these people, we'll reach them, and so we'll launch these at the lowest granularity that that platform will accept. Sometimes it's small audiences. Sometimes it's individuals, but wherever we can.

Then, once that's launched, the next thing we'll do is take very small pieces of those audiences, so not only cutting into small microsegments, but now I'm taking even small subsegments of them to test. We call it calibration. We'll launch many, many combinations, hundreds or maybe thousands of combinations of subsegments and creative. That will give us an indication as to which of those segments will respond better to which pieces of creative.

Once we've done that, then we start scaling. Then we start actually applying more spend, and that will lead us to a more global kind of scale. At that level, once we've done that, where we can, and China happens to be the territory we can do this the best, we will actually try to measure conversion. Meaning, we will actually try to see who is buying tickets. Those ticket purchases will then feed into our models and enable us to be more honed.

What's interesting about our approach is we tend to do things that a lot of them are the opposite of what others do. A lot of folks will start to become narrow and them maybe even get panicked and go broad. We do the opposite. The closer we get to release, the more honed we're trying to get and the more precise we're trying to get.

Michael Krigsman: Right.

Matthew Marolda: We're trying to get to as close to the actual number of people that are going to buy tickets as possible.

Michael Krigsman: The platforms, this is really fascinating. The platforms that you're using then are what? Can you be specific about that?

Matthew Marolda: Oh, sure. Yeah, it would be, in no particular order: social media, so Facebook, Twitter, Instagram, Snap, all those platforms. It would include the Google platforms, which would be search or YouTube. We'll also do things programmatically, so we'll be able to target people across many different websites. Those are the major categories. We do try to do analytics to help us guide what we would consider being nonaddressable media, like television buys and outdoor ads. But, it's using the same concept of audience. We're just now deploying it in a more coarse way.

Michael Krigsman: You mentioned earlier that you're going down to the lowest level of granularity that these platforms will accept, essentially.

Matthew Marolda: Yes, that's correct. In certain cases, we can provide individuals and track them. That's rare, but we can do that. In other cases, it's these sort of subsegments that may be hundreds of people or a thousand or two thousand, something like that. In other cases, for example, a television buy, you're buying against the people who watch that show. We're making a prediction as to who we think are going to actually watch the show, but we can't actually precisely say, "Oh, these are the 700,000 people we want to get through this show."  We are taking a bet that they'll be watching, but we don't know precisely what they are.

Michael Krigsman: We have a question from Twitter, an interesting one, from Gus Bekdash. He's asking, "Do you apply this on a real-time basis or just to get trends? How does that aspect work?"

Matthew Marolda: It's a great question. The pace of these kinds of campaigns is very fast. What will happen is, for any given movie, the vast majority -- when I say vast, 80%, 90% will be spent over the course of about 4 or 5 weeks. This is where I think people who are just kind of awake and alive will see these massive sorts of media dumps out into the world. We knew that was the phenomena, and we knew that we had to be able to react very quickly within those timescales.

If this were an always-on campaign that ran over the course of years, it'd be much different. We do try to operate very precisely within that sort of very short window of four to six weeks. I would say our cadence for changes and adjustments are typically within a day or so. It's not real time in a sense of every minute or every hour, but once a day we're recalibrating and adjusting.

Michael Krigsman: You've had a set of tools and techniques that let you take the data that's coming back.

Matthew Marolda: Correct.

Michael Krigsman: You're selecting the data to work on, to analyze, and build models. Then you're running your campaigns. Then you're taking the data back from those campaigns and doing what with it to optimize?

Matthew Marolda: Sure. I'll just clarify the loop, but the loop you described is very accurate. We have a source set of data we begin with that creates audiences. Those audiences we then launch. We had to build our own technology platform. It was really crazy to scale. The data we were storing wasn't sort of serviceable by most data storage solutions, so we actually built our own data storage solution. Once we had our data storage solution--

Michael Krigsman: I don't mean to interrupt, but that's incredible.

Matthew Marolda: Oh, sure. Yeah.

Michael Krigsman: I mean I'm just thinking about all the tools you had to build. That's wild.

Matthew Marolda: Yes. Yeah, it's crazy. It was funny, going back to the beginning. I was a guy in a room. [Laughter] I had a checkbook, effectively, and we could have done a number of things. We could have built a mosaic of solutions.

What we found was that that didn't exist, and so we built out a team. Our team is about 70 people. Of the 70, we've got about half are some form of an engineer, whether they're data scientists or computer scientists, and we have people who have all kinds of disciplines.

We accumulated these people, and we built these tools because they didn't exist, and we couldn't find that solution. It's really that singular solution that goes from front to back. There were a lot of good point solutions along the way, but they didn't have the full integration.

The loop you described is a very logical loop, and that's exactly what we were trying to build toward, but we had a hard time finding the solution that would meet both the speed and the pace at which we were spending, along with the sophistication with which we wanted to spend. To go back to your loop, this data platform that we've created will suck in data from whatever sources we've started with, the initializing sort of data. Then it will launch the media out into the different platforms.

To your point, as the campaigns run, new data is being created constantly. That comes back into the system, enables us to calibrate and change dynamically, and then re-spend. It's sort of a virtuous cycle that continues.

Michael Krigsman: I think I know the answer to this question, but here you are. You're basically giving away your process, and you seem willing to go to whatever level of detail I ask, or the audience asks. And so, why aren't you uncomfortable sharing your secrets and your process with the world?

Matthew Marolda: Sure. I hope this doesn't come across as arrogant, but it's a hard thing to do. [Laughter]

Michael Krigsman: [Laughter] I was going to say, probably because how would you replicate it?

Matthew Marolda: Right. Yeah, there's a certain alchemy we had to create even to be able to pursue these goals. That alchemy included people. It includes data. It includes technology. It's a lot of things.

No, the process and the approach, I think a lot of things hopefully that I'm saying are sort of borderline initiative. Certainly, we're transparent about a lot of this. The real secret sauce is what's happening the next level or two down, the actual tools we have, the actual techniques we're using, the specific data types we have, and the ways in which we connect to these different places, some of which are ones that people could reverse engineer and kind of anticipate. Others may be more subtle. That's the level where it gets a little more, I think, proprietary. But, at this level, again, it's just a hard problem to solve.

We also, frankly, are these sort of data nerds, right? We like to share, to learn what other people are doing, and there may be ways in which we can learn from others by being open about what we do.

Michael Krigsman: My mind is kind of reeling with what you're doing. The targeting then, you must have developed real areas of expertise in each one of these platforms. You must know as much about these platforms as the developers do, essentially, I would imagine at this stage.

Matthew Marolda: I think a good way to think about it, and I appreciate the compliment, but a good way to think about it is we have a very specific set of goals and objectives, and they're very practically oriented. We need to have people buy tickets to movies or watch our TV shows, whatever it is. We are willing to invest the time and resources to best understand these platforms for that purpose.

I think, for those purposes, for the kinds of things that we want to do, and they're broader than just movies and TV shows. It's when you want to build brand awareness; when you want to build consumer intent. Those objectives are ones that I think are shared by a lot of industries. The platforms are trying to solve many, many more problems than just those specific things. Yeah, I do think we've spent as much time as anybody on trying to figure out how to take full advantage of those platforms for that particular set of activities.

Michael Krigsman: Understood. I want to remind everybody we're speaking with Matt Marolda, who is the chief analytics officer at Legendary Entertainment. Right now, there's a tweet chat going on using the hashtag #CxOTalk. You can ask your questions directly at Matt.

Matt, as you said, the general approach is one that's being applied by many different companies in many different industries. Do you ever have companies like, say, retailers coming to you wanting to license your technology because these techniques would work for anything?

Matthew Marolda: Yeah, it's interesting. This is not false modesty by any means at all. When we started back in 2012 and '13, I wasn't sure any of this was going to work. We had no idea. It was a gamble.

It started to work and pay off quickly. We were fortunate that it got us our own kind of ROI very quickly. For about a year or two into that process, we were relatively quiet about it. We didn't want to pat ourselves on our back or take undue credit because there are a lot of factors into any of these successes or even failures, frankly, that might happen with a movie or TV show.

That being said, folks did begin to notice. People, especially in our own industry, initially, saw things that they were surprised by. They saw patterns that you could observe but weren't quite sure what was causing the pattern. For example, people use what they call tracking in Hollywood, which is kind of like polling. It's sort of a national poll that assesses people's interest in a movie. Tracking has been used for decades to predict box office outcomes.

Our movies consistently outdid tracking, meaning the old ways of predicting the box office outcome were being broken very consistently by what we were doing. And so, that drew attention. For a while, we said, "No, we won't share this with anybody. This is a competitive advantage."

But then, what we realized was maybe not so much for the creative side, which is something we've held closer, but when it comes to the marketing, this is an asset and an engine that we've created that could be deployed for others. And so, over the last 18 months or so, we've been sort of slowly, but with some aggression, taken on what we call our partners, people who do use our tools and license them for their own campaigns, whether it's for other movies or TV shows. We've done things in CPG [and] hospitality as well.

Michael Krigsman: You mentioned the tracking. I know that one of the things that you've done is used biometrics to see how people respond to movies.

Matthew Marolda: Oh, yeah.

Michael Krigsman: Maybe you can tell us about that.

Matthew Marolda: Sure. The phases, I think of at least, for a movie, very broadly, there's, for lack of a better name, a greenlighting phase, meaning the point at which you decide whether to make a movie or not. Then there's a production phase. Then a postproduction phase, which means the editing and any other refinements you do to the movie after it's been filmed. Of course, this is from our point of view on how we look at it. Then the marketing of it.

In that postproduction phase, which I think is what you're describing or asking about, is the tools we've used to try to hone in on a movie. It's the same thing, I would say, at the greenlighting stage. By the way, we were never involved in the production stage at all. All we're trying to do is provide inputs back to the filmmakers and the producers that they can then make judgments and make decisions based on them.

The difference is, our goal is to provide it at a level of granularity that is really robust and unique so that they can be as best informed as possible. Biometrics is one path towards that. We have done things where we've taken movie theaters, and we've put several hundred people, maybe 300, 400 people in a movie theater, and put the wristbands on them. They're like a Fitbit that would track things like heart rate and others. We have an iPad on the back of a receipt to then capture a face to then grab facial emotion and other recognition sort of indicators. We'll tie that back to a second-by-second view of the movie and allow us to at least give these broad, but also very specific sometimes, comments about what's happening.

I would say we wouldn't discard other things, too. We will also use what people call sort of [surveys]. It's the cards, meaning the little card you fill out afterward. We still use those. We try to blend traditional approaches with the slightly more avant-garde ones because we think that there's value in all of them, especially when they're integrated.

Michael Krigsman: You mentioned the relationship to the creative side, and so I'm really wondering how. We don't think of the artistic process as being informed by predictive analytics. Is there a tension there? How do you work with the creative side?

Matthew Marolda: Well, it's certainly a collaboration, right? On the creative side, especially, we're trying to make sure we recognize kind of where we fit into the process. There are many, many things that the creative team is working on at any given moment on a movie - many. There's only a handful of them we can actually impact. But, just like you'd want to understand what an actor might be like to work with or you'd want to watch their previous work, those kinds of things are readily available to a producer. They can call and find that out.

Why not also understand what kind of audience the property might generate on its own; what kind of box office potential it has in advance; what kinds of talent might align well with that? Again, it's not to say that I'm prescriptively [saying] here's what you have to do and you're dumb if you don't. It's more like helping to either hone options and get a little more specific or uncover ones that they might not have thought of. Again, they have to use judgment and perspective to be sort of the final arbiter on how it all pulls together. But again, the tools we've tried to build are inputs in that process that can be very enlightening.

I think one of the things that at least I've learned even from sports into here, but certainly here as well, is there's a real premium on humility, so coming in with an open mind. An open mind helps in a lot of ways. It helps look at problems in a different way and to understand that there are maybe solutions that exist, but also helps in how you present the solutions or the findings you might have uncovered. Humility is important because we want to be able to be confident and clear about what the data analytics are telling us, but also humble in its presentation so we understand the broader context that it's been used in.

Michael Krigsman: How do you get folks who are on the creative side that, in their entire life, would never have thought of applying data and analytics and probably resist it? How do you get them to have that open mind and be receptive?

Matthew Marolda: Sure. Sure, so the attitude overarching, of course, is that humility, but that confidence, that sort of combination of the two. The real practical steps have been a learning curve, quite literally. Starting with things like here's a small sample of what's possible. Also, like a lot of things, it's about listening and asking questions. I'm trying to understand what is it they're trying to solve, or what's the riddle that they haven't been able to crack on their own? Going back and then building a tool that can help them actually get there in a way they couldn't have done through intuition or other more traditional means.

It's both the attitudinal thing, but also a sort of learning curve where it's dolled out. We've been fortunate to see that people become kind of voracious about it. They ask more questions and you get more in depth. It becomes a real conversation, which is exhilarating, I think, for everybody.  It's led to stars being cast in TV shows.

Even recently, there's a well-known actor right now who has a guest spot, basically, on a network television show because we used these analytics. Again, it's a great example where the producer came in with a list of, I think it was, 30 or 40 actors. They wanted to understand who had the best fit with the show, who could bring new audience but still have a consistency with the core of the show and be able to provide a ratings boost. The analytics helped us hone the 30 or 40 that they were considering down to 3 or 4. Of the three or four, then they were able to go and focus on the one they thought was the best, and they were able to bring him in. It's a process that looks like that where it's collaborative and communicative.

Michael Krigsman: It seems to work and, over time, then the creative side has become receptive because they've seen the benefit.  I'm assuming that must be the cast.

Matthew Marolda: Yes. It was two things, I think, and they are both human nature. One is it's a great way to be able to have a justification. It's a great way to be able to say, "Hey, we did all these things and here is hard evidence to support the decision." That's really helpful.

I think, also, they have seen the efficacy. Being the type of data nerds that we are, we would love to be able to do the test. You'd love to use a really basic sort of typical nomenclature. You want to do the A/B test. It would be great if you took actor A and actor B, created the same show, and saw which one was better. That's obviously impossible. That level of efficacy is hard to measure. The confidence that the decision-maker has to make that final casting call is much greater with these tools behind them.

Michael Krigsman: What about the ethical issues? You alluded to that earlier of the targeting. You've got so much data. How do you balance the privacy? You want to target very narrowly based on intense interests or subtle interests, but you don't want to be creepy and overstep your reach. How do you balance that?

Matthew Marolda: Ah, so there are lots of different ways in which we need to think about that problem. There's one you hit on, which is a regulatory one. To be very clear and without ambiguity, we have bright yellow lines around all that. We don't take anything that's anywhere near a risk.

That being said, that's an easy boundary to create. That's a simple thing. You would just do that by nature. That includes things like we don't scrape data. All the data we acquire is acquired legitimately. We have an agreement, an open relationship, and often a monetary relationship with the source of the data. Not only are we staying well within the bounds of the regulatory constraints, but we're also well within the bounds of the data itself.

Then there's another level I think you kind of touched on too, which is, how do you interact with a consumer in a way that doesn't feel creepy? We certainly are conscious of that. That's where the tools and the platforms we tend to use for these purposes, these addressable platforms we talked about earlier--social, search, video, display--those are ones where as far as the consumer of whatever they're watching on that platform are doing and they're getting these ads served to them, it's hard to discern whether that ad is anything special for them. It's just one of others. That feels like a very natural way. It doesn't feel creepy.

If we were to suddenly start popping up in places and addressing you directly, and you didn't realize that we were there, it's the kind of thing that I think would throw people off. We're well short of that. We're operating within channels that the advertisements are coming in a way that you would expect. They just happen to be very precisely directed.

Michael Krigsman: That's really interesting. If your data science is really working well, and you're operating on the right set of data, from the user perspective they're receiving advertisements that are natural, that seem organic, and that just flow.

Matthew Marolda: Right.

Michael Krigsman: Therefore, there's no creepiness factor because, actually, this makes sense.

Matthew Marolda: Yeah. In a lot of ways, and I don't think people always think of it this way, but that's the implicit barter we make with all those platforms. That's the implicit barter we make for having the ability to search the Internet for free is that there will be ads served to you. We're just kind of working within that ecosystem.

Again, it goes back to what I think I said a little earlier. The bet we made was always on the analytics themselves. We recognized, because of these shifting, even sometimes broader political -- like Europe was a great example where the regulatory issues there are very different than they are here, and people's view of data and even that implicit barter is different. We knew that those things would change.

We couldn't predict. [Laughter] Even for people who spend all their time thinking about these things, it's impossible to predict how those tides will ebb and flow. But, what we could predict was that they would ebb and flow. Therefore, we wanted to play at this other level above it where the analytics are, and then we will just adapt to whatever we can do. As things become more available, that's great. If they become constrained, we'll look for other alternatives. We'll kind of balance all those activities together.

Michael Krigsman: At this point, I'm assuming. I don't want to put words in your mouth, but I'm assuming that, inside Legendary, your team, your work must be kind of diffuse in a way throughout the whole organization, which is to say at every stage of the process your team is involved in some type of integral way.

Matthew Marolda: Yeah, often. It'll vary a little bit by project just depending on how things are going and where the need is. Some movies develop very quickly and nicely right through postproduction and don't need a lot of intervention. Others are a little more challenged.

We're always very involved in the marketing end. That's the part that's consistent. On the other side, there are movies that come down the pike that we don't need to do a lot of analysis on. We have a movie called Skyscraper. Of course, you want Dwayne Johnson to star in a Die Hard type movie. [Laughter]

Michael Krigsman: [Laughter]

Matthew Marolda: It doesn't require a lot of analytics. But, there are others that are much more carefully considered. It varies just a bit based on the specific project. Yeah, we touch all those parts, again with the real exception, of course, truly being like scriptwriting and production. That's an area we don't ever get involved in at all. But, the other parts in terms of greenlighting, some of the postproduction, and certainly the marketing, we're very involved in.

Michael Krigsman: Why not scriptwriting? I'm sure you could give insights into scriptwriters that would be really valuable to them. Why not?

Matthew Marolda: Yes, sort of. [Laughter]

Michael Krigsman: [Laughter]

Matthew Marolda: Again, we still believe that there are very creative aspects to all of this. This is a human endeavor, still. There are probably ways in which you could create artificial intelligence. I'm sure people are doing this now--we haven't spent a lot of time on it--to construct things more specifically.

I think what we do, I know, is that we do try to at least provide the same kinds of inputs I was describing earlier: thematic kind of observations or other signals we might pick up from prior movies that are similar and the audiences around those movies, and again very broad brushstroke type things. I can think of a showrunner, in particular, a very well-known showrunner that we have a show with, who is very receptive to those kinds of things. He's very opened minded about the data signals we would pick up that might help him shape the show, but we don't get into anything specific. We'll bat around a few big ideas, but he then takes those and puts his creative brain toward it and then figures out a way to solve it on his end. Yes, it's an interesting dynamic, for sure.

Michael Krigsman: In a number of different areas, it seems like you have constructed very careful boundaries like you were describing how you collect data, regulatory issues, the relationship to screenwriting, things like that. It seems like you're very careful where you go deep and, at the same time, you're equally careful circumscribing where you won't go.

Matthew Marolda: Yeah, that's true. We tried to establish a set of rules. I think it comes back a little bit to some of the questions you're asking earlier around how we get buy-in and adoption. By self-declaring those boundaries and making it very clear, the perspective we have and what we're trying to do, it makes the conversation much easier. It's human nature; as soon as there is any feeling of encroachment or someone trying to usurp someone else's role and responsibility, that really gets the dynamics misaligned.

The objective is to almost self-declare and say, "Hey, look. This is what we think we can do. Here's where we believe the impacts can be," and they're well short of anything that's near encroachment, right? It's something that's additive, supportive, and with again that humility kind of wrapped around it. Then that is a true thing, it's sincere, but it also has that byproduct of making people feel comfortable that we're not trying to go too far or push things in the wrong direction.

Michael Krigsman: There are a few questions. We only have about ten minutes or less left, and there are a number of different big areas that I want to ask you about. Very quickly, can you describe the composition of your team?

Matthew Marolda: Sure.

Michael Krigsman: Who do you need if you want to create this kind of magic?

Matthew Marolda: Yeah. We need the right people, for sure. I've said humility, I feel like, four times. It must be the word of the day for me. That's an initial starting point for us. We look for people like that.

Of course, we have other things we're really interested in, and so the specific skill sets we have accumulated. On a data science side, it's multidisciplinary. The person who runs our data science team has a Ph.D. in astrophysics. That's a discipline you wouldn't expect at a Hollywood studio.

Just like that discipline, there are people who have different backgrounds in social sciences like human decision sciences, or they are statisticians or econometricians. That's a whole category of people we have as data scientist folks.

On the software development side, we knew--and we talked about it briefly earlier--that we were going to have these very large data sets. We needed people who had the skills to be able to build these repositories to query and analyze data at remarkable speeds, to be able to even build the infrastructure and the thousands of servers we have running at any given time to support all that, to build the user interfaces that make it all work. Those were skillsets we were very specific and targeted on.

We also needed the other half of our team, basically, of people who are experts at applying these kinds of outputs into a campaign. That last group I just mentioned was by no means the last. In fact, we considered all three simultaneously because we knew that if the data science team and the development team built all the amazing tools they build but they were just shiny toys on a shelf, it was all for not. We needed to make sure we had a group of people who knew how to translate those tools into action. That creates the whole iterative loop we use to further develop.

Michael Krigsman: Here you are trying to build this team and you realize, for example, well, the storage systems that are out there are not going to work. You now have to go out and hire a bunch of storage system designers.

Matthew Marolda: Right. Exactly.

Michael Krigsman: That's insane! [Laughter]

Matthew Marolda: [Laughter] It is. Yeah. You're not the first to say that. It's actually probably a little insane, yes.

Michael Krigsman: Okay. I say it's insane with the greatest of respect and admiration.

Matthew Marolda: Oh, it's very practical. I think we were able to solve a problem we couldn't have solved otherwise, but it's crazy we went to that extreme, I think. Yeah.

Michael Krigsman: But, you had to do it. You really did.

Matthew Marolda: Yeah, we had to. We were forced to out of a practical need. Yes, that's right.

Michael Krigsman: Okay. I guess when you're keeping that practical need and really monitoring efficacy, "Does it work at every step?" then you can explore these new territories, actually make it work, and execute it.

Matthew Marolda: Yeah. I think there's another thing, too, which we were empowered from early on. A lot of people are punished, penalized, or even fired for taking risks and not making the safe choice. We were the opposite. I think we would have been punished or penalized for taking the safe choice and not taking a risk. We had the attitude from the beginning that failure was great. Failure is fine. It's a learning opportunity, but you want to do it fast and cheaply. [Laughter] That's the idea.

Yeah, we were willing to take a lot of [risk], and everything I've described was built progressively. It wasn't like we necessarily set out one way or the other. It's that we built from a need and grew it from there.

Michael Krigsman: You were layering.

Matthew Marolda: Correct.

Michael Krigsman: You build something, you layer, you experiment, you layer, and so forth.

Matthew Marolda: Exactly.

Michael Krigsman: Okay. In the spirit of trying to accomplish large topics within the space of just a couple minutes because we're really running out of time, where is the future of this? You're on the cutting edge, and so you're viewing. What's next, not 20 years out, not 10 years out; what's coming down the pike?

Matthew Marolda: For us, I think the thing that we think a lot about is two things. One is the increasing addressability of media channels, so the ability to get more and more precise. That feels around the corner. Whether that means addressable TV in that you can send an ad over whatever form of viewing you were doing, that's one thing for sure.

The other thing for us, which is always that holy grail--in a lot of industries this is not the case, but for things like ours--are conversion measurement is hard. Meaning, can we tell if someone took the action we wanted? As data becomes stronger and better there, that just makes everything better. Those are the two things that feel, in these sort of 18 to 24 months, or maybe even a little longer than that, but the 1 to 3- or 1 to 5-year range.

Michael Krigsman: The more narrow targeting then, you have the data.

Matthew Marolda: Correct.

Michael Krigsman: You have a lot of the data. The more narrow targeting then would depend on the enhancements of the platforms.

Matthew Marolda: Correct.

Michael Krigsman: Is there something about what Facebook is planning or YouTube, or Google that you should be sharing with us?

Matthew Marolda: No, no. [Laughter]

Michael Krigsman: [Laughter]

Matthew Marolda: No, I have no special knowledge. We just know the day is coming. Our goal is to be prepared for that day. The day that we're ready for is the day where we could find every single person we want to find, wherever they are, and however we want. That day isn't here yet, but it's coming, and that's the day we're preparing for.

Michael Krigsman: You're just anticipating that.

Matthew Marolda: Right.

Michael Krigsman: Given the evolution of these things, that's going to happen pretty soon.

Matthew Marolda: Correct. Yeah, it's a bet. I think it's a pretty safe bet, but it's definitely a bet.

Michael Krigsman: Okay. Finally, what advice do you have for businesspeople, and not necessarily in the movies, but businesspeople in retailing, in marketing, and whatever it might be who want to use data and analytics approximating for, say, CMOs in their marketing? What would you recommend to get in below the surface, but obviously not to the extent that you've taken it?

Matthew Marolda: I think, first and foremost, it's adopting this attitude. I think what I've seen, at least in a lot of organizations, is that data or analytics, or both, spring up as departments. They sort of become their own unit or something. It's almost like a reverse factitial effect. There never was a place for them before, so now they have to have a place, and they get created.

I think it's almost the opposite. Even though my title even has analytics in it, really it's a discipline applied to subject areas. I think the first sort of step is to be able to break that thinking that, "Oh, data is over here in this corner and people need to think about it as its own sort of discrete thing," and in fact, it's actually almost the opposite. It's a tool and a toolset that can be applied to something specific. That's just a mindset thing.

I think, more practically, data collection itself is hard. That's why we had to build what we built. If people can capture data and keep it at a level -- like, we don't throw any data away. We have innumerous amounts of data that may not even be any good, but we never know if we could use it or if it's somehow useful for us in the future. Storing data and keeping it is really another thing. Then having said that, this sort of higher level thing that, like I've mentioned a couple times that we try to do, which is, realize that it's about the analytics of the data, not the data itself. That's where the value comes.

Michael Krigsman: Okay. We have another question from Twitter. This is from Zachary Jeans. He actually is asking the question that I meant to ask you about what's coming down the pike. He's asking it in a better way. He asks, "What is a technical challenge that you will be facing during this next year that is new?"

Matthew Marolda: Hmm. There are so many. [Laughter]

Michael Krigsman: [Laughter]

Matthew Marolda: It's hard to almost pick. The ever-growing challenge for us is working through the shifts in the technology and data that's available. Part of it is just the nimbleness of me to have to be able to adapt. Those are typically the ones that are at the forefront for us.

It's not so much a technical challenge. It's the functional challenge, which is, we're always trying to solve the problem around measurement and conversion. Meaning, after all the work we've done, sometimes it's very hard to tell, did we actually get the person we wanted to buy the ticket? That's a challenge that we're always confronted with and we're always working toward.

Michael Krigsman: Okay. Wow! This has been a very, very fast 45 minutes. Matt, thank you for taking time to come talk with us today.

Matthew Marolda: Oh, no, I appreciate the opportunity. It's great. I thank you very much.

Michael Krigsman: You have been watching Episode #276 of CxOTalk. We've been speaking with Matt Marolda, who is the chief analytics officer at Legendary Entertainment. What an action-packed and very fast and interesting show this has been.

Everybody, come back next week; we have more shows. Don't forget to like us on Facebook. Thanks so much, everybody. Have a great day. Bye-bye.