Data Science and Predictive Analytics in Pharma

Michael Krigsman: You've got all of these people working together from different fields. How do you stop them from killing each other? [Laughter]

Bülent Kiziltan: Sometimes, you sacrifice yourself as the leader, I would say.

AI and data science in pharma

Michael Krigsman: Today, we are talking about AI and data science in pharma. We're speaking with Bülent Kiziltan, who is with Novartis (one of the largest pharmaceutical companies in the world).

Bülent Kiziltan: Yes, my title is Head of Causal & Predictive Analytics at Novartis. Our organization is called AI Innovation Lab.

What we do is we drive the internal AI innovation across Novartis and also position ourselves in the intersection of academia, tech, and the business units where we think AI innovation will really happen. This is what we do.

We are trying to build operational excellence with diverse talent coming onboard, engaging the domain experts from drug development and discovery so we can reinvent some of the processes, eat them up, and make them as low-cost as possible. We know drug discovery and development has been slowing down over the last five to ten years mainly because of the cost and also the slowing down of the process because scaling up the whole process has been very difficult. AI comes to the rescue, we hope, and a lot of pharmaceutical companies are investing in this area.

AI and data science, in general, can operate in one of two ways. One way is to be use-case driven. In those cases, they are providing the services to business units. The other case is where we position ourselves to be at the intersection of academia (where most of the interest and know-how is being produced between tech where we need that technological development and infrastructure to scale things up) and the business units.

Once we produce, know-how, bring it to novel applications, and are being empowered by our technology partners is where we're pushing the needle.

Michael Krigsman: What is your role at the AI Innovation Lab?

Bülent Kiziltan: I'm the head of causal & predictive analytics. Almost anything we do has to do with predictive analytics, data science, or machine learning. But in addition to those, we put an emphasis on causal learning and causal discovery, which is (we believe) the next frontier in implementing information extraction from data.

But also, in data science, typically how those teams are being perceived is they work with big data and, whenever there is a limited amount of data, the value proposition declines. We want to make sure that we cover the whole spectrum all the way from small data to big data because those terms are vaguely defined, and we don't have a clear way of quantifying small and big data. Me and my team, we've built those core capabilities where we can extract interesting information from limited data all the way to big data, as we call it, where all the data coming from different streams and modalities can be used and leveraged in forecasting.

Michael Krigsman: Can you give us a sense of the kind of strategies that you employ as you're thinking through these problems? Really, can you take us through the end-to-end way that you think about this issue of using data science and using AI in pharma, but also more broadly, because I think that will help give us some context?

Bülent Kiziltan: AI, in general, the operations have interesting constraints. For many, many years, we have been articulating what are the top parameters that are determining the success and failure of data science operations. Counterintuitively, we have seen that culture and leadership profile comes in the top two.

Until very recently, even though we are sharing our anecdotal experience, we didn't really justify our experience with data. Very recently, I have been a part of a global benchmark study to look at companies across the globe, thousands of companies that are small, big, come from different domains. And low and behold, we've seen that culture and the leadership are critically important in success.

Then we talk about AI in large companies versus smaller companies or startups. All those operational pipelines and priorities have to be very different.

In the startup space, the priorities, the timelines, the regulations are very different as opposed to larger companies. There is the constraint of a settled culture. There is inertia. There are different business units with different priorities, so there is a lot of "across business unit" talk in larger companies.

Then when we add another layer of sophistication of being in the pharma and biotech space, we start to talk about regulations, about data privacy, which is very, very important. We take those very seriously. But those pose additional constraints to the operations because data access is number one and data understanding is number two. We want to make sure that we leverage technology to lift those bottlenecks and barriers in our operations, so this is where we start.

Michael Krigsman: Well, you raise a very interesting point. You began with culture and leadership, and only then did you move into the subject of data. Nowhere did you even talk about algorithms.

Bülent Kiziltan: What we've seen over and over is the technology that we use for on-the-ground execution is somewhere between standard statistics and innovative machine learning – so, in between.

As I said, in a larger company setting we have to interact with domain experts on a regular basis. We have to work with different business units—with the engineering units, with the IT—and their technical competency comes not at the top in terms of reaching success, making the project work, and executing on the ground. We need to leverage all those different aspects of an end-to-end pipeline.

Then once we look into the data, more often than not, we start with small data and then, incrementally, enrich the information with additional data that we have. That is a herculean effort to pull in data coming from different streams and even augment the information with publicly available data that is coming from our collaborators and partners. Then we start talking about incrementally making our approach more sophisticated and benchmark as we go to more sophisticated algorithms and models to see whether it provides any value.

Then from the very beginning, we have to start the conversation with regulators and the units that will use that information. Is that going to be useful? Is that going to be sanctioned by the decision-makers? There are multiple angles to the whole data science and AI operations.

Michael Krigsman: Really, the technology aspect – technical competence, as you just described it – is, can we say, the enabler? It is a necessary, but certainly not sufficient condition for success in the work that you do.

Bülent Kiziltan: Yes, you summarized it correctly, Mike. It's definitely necessary to push the frontier in AI and execution on the ground. But without the right leadership and the culture, the value proposition of AI is only short-term.

If you want to sustain the value proposition for the long-term, you have to build a culture and a company around that. I think I am very lucky and privileged to be at Novartis because we have set the unboss culture – not as a choice, but it is necessary to make the impact on the ground, to reimagine medicine, as we call it.

Talent management and diverse teams in pharma

Michael Krigsman: Bülent, tell us then about the composition of your team. How do you hire? Who do you hire? What are the kinds of roles and, especially, what are the characteristics of the people that you're bringing in?

Bülent Kiziltan: AI innovation, especially data science, is a very interdisciplinary and multidisciplinary domain. So, we want to make sure that we attract talent from different disciplines who can bring in the value, the mindset that they bring from their own domain into our operations.

Certainly, data science and machine learning core capabilities are necessary, but we're open to all backgrounds. As you and your audience might know, I am trained as an astrophysicist and have done neutron stars and blackhole astrophysics for most of my career. But in that domain, I worked very closely with applied mathematicians, machine learning pioneers to bring in some of those technologies into the domain of astrophysics back then.

Extracting information from limited data is essentially what we've done. That is combined with domain knowledge in astrophysics and this is what we do today in the domain of healthcare, biotech, and medicine.

We are very cognizant of the fact that diversity is not only a choice, again, it's a necessity to think out of the box and innovate in the space of AI. Currently, we're obviously growing our teams and looking for that sort of talent that can bring in the core capabilities that are necessary with machine learning, but they can come from physics, mathematics, psychology. I had worked with people coming from sociology, economics – you name it.

Michael Krigsman: This notion of diverse teams, bringing people with different kinds of backgrounds from different disciplines sounds like it's foundational for you to accomplish your goals.

Bülent Kiziltan: Yes, absolutely. We certainly could operate and execute at an adequate level with just a homogenous talent pool, but we've seen over and over that diverse talent brings in really interesting ideas and that is the place where innovation happens. So, diversity is a must.

Michael Krigsman: We have a question from Twitter. This is from Arsalan Khan. He's a regular listener. He asks great questions. I love when questions come in because what an opportunity. If you're listening, man, you should be asking your questions and leaving your comments.

Anyway, Arsalan Khan says, "Everyone talks about culture, which is either an obstacle or an opportunity. If everybody knows this, then why is culture so difficult, how do you think about culture, and do metrics come into play?" It's the culture question and how do you manage that, since you said it's a foundational building block.

Bülent Kiziltan: It is. What we've seen, again, this comes from a lot of benchmark studies across the globe. Culture cannot change bottom-up, unfortunately, in companies. That's what we see.

It has to be empowered from the very top. So, unless you have a company that is conducive to data science operations, this is primarily an intradisciplinary endeavor. If the company is not data-driven, it takes only anecdotal experience into consideration in the decision-making process. It's very difficult to push against that inertia.

Certainly, it can be possible to run data science operations in companies that are very traditional. You have decision-makers that have been in the domain for 15, 20, 30 years, but they don't have an appreciation for data-driven decision-making whereas, in our company, I must say, again, we are very privileged to have that mindset in place.

We are evolving to be a data-driven company. All the decisions are data-driven, and this is why me and my team at my organization are cutting across Novartis, interacting not only with drug discovery and development but with all the units, including finance, including manufacturing – you name it. Top to bottom cultural empowerment is critically important.

Michael Krigsman: Talk to us about domain knowledge. I think it's obvious that you need to have the technical chops around data science, machine learning, and so forth. But what about the science domain knowledge, the biology, chemistry, and so on?

Bülent Kiziltan: We cannot operate without them. We are blind without domain experts.

Creating that interface is one of my responsibilities to bring teams together to interact and understand each other. The language difference is one of the obstacles, so what we do, as a part of our operations, is really listen in to the key questions that are coming from the domain experts and try to translate that into data science language so we can build the roadmap for the analysis, the data science operations together.

The cadence has to obviously be customized to the unit that you are interacting with so that there is a lot of cross-disciplinary interaction. It's one place that sometimes data science operations are not as successful. This is where leadership comes in to speak the language of both sides in order to build a successful roadmap by which you can execute.

Michael Krigsman: What are the characteristics that you look for as you're hiring?

Bülent Kiziltan: A diverse background certainly is critically important. The pharmaceutical domain has been known to be very conservative and requiring domain expertise from biology and medicine, which definitely adds to the operations, but we basically hire from all sorts of backgrounds that we can weigh accordingly.

If they come with a very strong machine learning engineering background, IT background really plays a critical role. Biomedical knowledge, whether they use some of the algorithms that we are developing or using, those are all going into the decision-making process.

I cannot tell you we are looking for this or that. We are looking at the whole interviewing process on a case-by-case basis.

Soft skills, again, are critically important, as I said, which is sometimes something that I emphasize when I give talks to students at programs when they're asking me what they should be investing into. They're expecting me to talk about Python (so I can learn some of the technology necessary). I say, "You know what to do in the technical domain, but something that you may not know is to invest in your soft skills," because communication with domain experts and being able to understand and listen carefully in order to really understand the problem is critically important.

I will say this to my friends who are technically inclined. Listening is not always our strong suit, so we really need to be listening in, understanding, empathizing, seeing the whole problem from a 360-degree angle in order to execute and be successful.

How to manage diversity 

Michael Krigsman: You've got all of these people working together from different fields. How do you stop them from killing each other? [Laughter]

Bülent Kiziltan: Sometimes you sacrifice yourself as the leader, I would say. I've been very lucky in that sense, but sometimes you go into an operation where certainly people come from different backgrounds, different cultures. Certain communication sensitivities might not be in place.

Then we, as leaders, have only soft power to gear the conversation in the right direction, but I've been very lucky in that sense. So far, nobody has killed each other. In that sense, again, it falls onto the leader to really gear the conversation, use the soft power to focus on the problem rather than making it personal.

Using AI and data science in drug discovery and precision medicine

Michael Krigsman: Can you give us some insight into the kinds of problems that you're tackling? Obviously, I'm not asking you to share confidential information, but help us understand the nature of the problems that you're solving or trying to address.

Bülent Kiziltan: Those are obviously the problems faced by the larger biotech and healthcare industry is, how do we customize medicine and treatment? That customization, at scale, we believe can only happen with the help of AI.

Precision medicine is one focus area for every pharmaceutical company, biotech, and healthcare company. Leveraging the technology that is developed in the domain of AI is critically important. It definitely will redefine that whole domain.

Other areas are where we develop and discover compounds and drugs, generative chemistry (to be more technical). It is an area where AI and machine learning is making an impact. There are lots of companies that are leveraging data science, AI, and machine learning to augment the development process, but also discover new compounds.

Again, previously those were done only in the lab. It was a painstakingly short and difficult process. With AI, we may be able to do that all in silico, on the computer. We can simulate. We can come up with a priority list of compounds, and then talk to our domain experts about whether what we find makes sense or it's totally crazy.

Michael Krigsman: We have a really interesting question from Lisbeth Shaw. She asks (on Twitter), "How will causal and predictive analytics impact the development of therapeutics and ultimately affect patient care?"

Bülent Kiziltan: Causal learning and causal discovery, as a domain, have not converged with machine learning yet. We are hoping to be at the frontier.

We're building collaborations, partnerships with the pioneers of that field in order to leverage that technology in discovering new causal relationships either at the compound level or at the patient level, so we can use precision medicine to customize the therapeutics, the dose – you name it. It will penetrate all parts of the development and discovery process when it comes to biomedicine.

Michael Krigsman: Is it fair to say that, broadly, the two areas that you're very involved in are precision medicine (using data science and AI) and then drug discovery, trying to uncover new molecules or design new molecules?

Bülent Kiziltan: This is the exploration part. Yes, we are very much involved in it. Those are our primary focuses. But also, we are empowering associates across the company to convert them into citizen data scientists so that they can use data in all their decision-making processes, not only in drug discovery and development but all across the company.

Michael Krigsman: Connect the dots of why you're involved in that. I don't want to shift the focus here too much, but I tend to think of that as an IT task as opposed to an AI innovation lab activity.

Bülent Kiziltan: Yes. IT is definitely very closely partnering to what we do. We couldn't survive without them. They are on the technology and infrastructure side of things, which is essential to scale up the technology.

But the brain behind what we plan to do and aiming to do has to do with machine learning, statistics (all the way from small data to big data) depending on what level of sophistication that we use. The brains, the engine if you will, around that car is data science and AI, whereas the infrastructure (the car itself, maybe the tires) comes in collaboration with IT so we can put the car on the road and we can drive it.

Michael Krigsman: You're thinking about the end-to-end process, not just about what you're doing but where you fit into the broader context.

Bülent Kiziltan: Yes, and without that vision, AI cannot survive in isolation. This is why some of the AI operations have failed to produce value in the long-term is because they have been isolated. They haven't built that cadence and interaction with different business units. This is where soft skills, strategic thinking come into play to sustain the value proposition of AI operations.

Challenges in drug discovery and precision medicine

Michael Krigsman: What are some of the challenges that you face with drug discovery, with precision medicine?

Bülent Kiziltan: There is a lot happening, right? Even though we have a very diverse pool of talent, it's sometimes really challenging to keep up with what's happening in the domain.

This is why one of the tasks that I have taken over and trying to contribute is to build that interface with those R&D-driven and academic institutions to understand the long-term vision and roadmap of that technology and where it's going. Keeping up to date with the direction of the technology has been very difficult, but I think we're doing a really good job in that sense as well.

Then on the execution side, again, there are always internal inertias, bottlenecks, depending on the domain that one is operating in. Certainly, pharma and biotechnology, we're not short of bottlenecks and pain points, which we are trying to address as one team.

Michael Krigsman: What are the challenges that you face and that your team faces?

Bülent Kiziltan: Engaging different business units, really it takes time to understand the problem. Translating that problem into a data science understandable problem and quantifiable problem takes some time, and it's sometimes painful.

I think we have reached an optimal point internally where we have an effective pipeline where we interact, convert, push the needle in terms of innovation, and execute on the ground. I would say we are very lucky internally.

Externally, what I see, in general, in AI operations is people have strong opinions. But we have to realize that, in a data-driven environment, those strong opinions have to be held weekly. Meaning, we have to update our opinions on our strategy and customize and adapt to incoming information. Sometimes, we don't see often that strategies are being adapted to new incoming information.

Michael Krigsman: Bülent, you mentioned big data and small data. Maybe elaborate on that for us and give us the context of where it fits into your work.

Bülent Kiziltan: AI, in general, is moving from a model-centered to data-centered operation, and strategies have to be built around what data is available. More often than not, specifically in healthcare (also in biotechnology and pharmaceuticals), we are somewhere in between small data and big data.

As a data science operations, if all your core capabilities are geared towards big data, you may become obsolete at some point because you have a hammer and you're trying to basically formulate every problem accordingly. Whereas, our operations and my team, we have core capabilities all the way from standard statistics, applied mathematics, basic statistics, to standard machine learning, to innovative generative learning, all the way to contrastive learning, even beyond. We are trying to have a broader view of extracting information from data not only with deep learning or machine learning, but we want to look at the whole spectrum of what information might be available.

More often than not, innovation actually happens around small data, how we extra information, and then implement, innovate, and augment all that information with bigger data with simulations (sometimes) or publicly available data. But it's an incremental process.

Michael Krigsman: Cindi Howson, on LinkedIn, points out that culture remains the top barrier to being data-driven. You've really elaborated on that. Very pointedly, how do you overcome that in your group?

Bülent Kiziltan: We are very lucky (in our company and within our group) we are data-driven. But what we see, more often than not, it's a journey on which we need to embark as a team and as an organization, always.

There is always mutual education, and we all need to keep an open mind and embark on that journey together whether it's digitization, a digital transformation journey, and then being data-driven. It's a cultural journey that all stakeholders have to be partners in.

How Novartis engages community around AI awareness

Michael Krigsman: We have another question from Twitter. We have a few questions. Diana McKenzie—who used to be the CIO of Workday and now she's on the board of a number of life sciences companies, and she's been a guest on CXOTalk—says, "Are you taking any unique approaches to educate and engage a broader team of associates at Novartis by providing access to and understanding of AI approaches?" In other words, are you engaging with the broader Novartis community about AI, what it can do, and how it works?"

Bülent Kiziltan: It's a journey that we need to embark on. This is exactly what I meant.

We need to and we are creating opportunities across the company to engage associates not only who are doing data science and bioinformatics, but the decision-makers as well. We have lots of one-on-one conversations with decision-makers on what we do, how we do it, our core capabilities that extend from small data all the way up to big data.

But we also have internal conferences, meetings that we engage. We have cluster talks. We keep up to date with the technology.

We have lots of opportunities internally. Those are critically important to be effective learners on that journey.

Michael Krigsman: I have a follow-up question for you. Why are you doing that? Why do people across the company really have to have expertise in AI?

Bülent Kiziltan: AI and machine learning are just a means to drive data-driven decisions and understand the risks that are associated with it. Namely, we can quantify appropriately the risk in the whole decision-making process regardless of what units we're talking with.

We are trying to reimagine medicine, provide therapeutics to patients. We want to expedite that process. We want to customize our therapeutics and doses. We want to help the patients and contribute to society.

All of those are intimately linked to the decisions that we make and the risks associated with it. Machine learning is doing a wonderful job in really quantifying and giving us a realistic projection and forecast of the risks that are involved. It helps us leverage that risk forecast in our decision-making process.

This is regardless of the subdomain. It can be related to manufacturing, finances, even biological research. But machine learning is one of those marvel technologies that help us scale up that decision-making process, ingest information, even combine information that are coming from different domains that are not necessarily homogenous, which is one area that only (as far as I can see) machine learning can help the decision-makers.

Michael Krigsman: It sounds like embracing the broader community helps your team remain grounded in not just the research problems, the abstract research problems, but also the down on the ground realities of delivering what will ultimately become products into the market. It helps you stay connected.

Bülent Kiziltan: Absolutely. Innovation can happen in two ways. One of them is developing those algorithms. But also, innovation, by definition, happens when there are novel applications. Being disconnected on both ends of the spectrum, I think, stifles the value proposition of AI.

Being grounded, executing at the same time, learning about the frontier, and contributing to the domain of AI, I think, is crucial. There is kind of a cycle where you contribute to the domain, you ingest some of that technology, you execute on the ground, you come up with novel problems and data sets, and go back to the domain experts and combine that information in order to keep the innovation cycle going.

Managing bias in data science and AI

Michael Krigsman: We have another question from Twitter. You can tell I really try to prioritize the questions that come from the audience on both Twitter and LinkedIn.

Arsalan Khan comes back, and he wants to know about biases in the data. How do you handle data that has inherent bias and can skew decisions, and bias on the team? The bias question, how do you address that?

Bülent Kiziltan: Yes, it's an active area of research, right? We engage the domain experts from different disciplines that try to address this particular problem. There might be sampling biases. There might be algorithmic biases. There might be data-driven biases.

Those are all things that we need to address early on. Once we come up with a forecast or prediction, we have certain steps that we take to make sure that we are not biased to a level that affects the decision-making.

But I must say it's an actively developing domain and I'm yet to see a strong, quantitative perspective and methodology that will help us address it. It has to be taken use case by use case, and we need to go (and we are going) step-by-step incrementally to address some of those issues. But it's a great question.

What are the differences between lab-based and AI-based approaches to drug discovery?

Michael Krigsman: How is all of this different from the traditional approach to drug discovery?

Bülent Kiziltan: Our colleagues have been working tirelessly in the lab trying to manually produce (working with chemists), trying to come up with new compounds, and then go incrementally and applying it to cells and see (under the microscope) how did it behave. Depending on the properties they want, it's a very manual process and very difficult to scale.

This is one of the issues which machine learning and AI are helping is scaling up and making that process faster. By doing that, we already are lifting some of the barriers in the compound production and discovery process.

What we also can do nowadays with technology is to come up with novel compounds within the computer and then come up with a list of compounds with predictive properties. Then talk to our chemists, domain experts, to really try (if it makes sense) some of those compounds. There have been companies that have demonstrated that those in silico produced compounds with predicted properties actually do exist, and so we are basically making the strides in that novel discovery space as well.

Michael Krigsman: How do you work with the lab-based folks who are working on using traditional methods? Is there cross-pollination? How would you collaborate?

Bülent Kiziltan: We have, obviously, meetings within the clusters. We exchange ideas. Depending on the use case, on the problems that we are tackling, we have those meetings, as many of them (sometimes too many) to tackle the problem, to understand the problem, and to push things forward. Yes, one-on-one interaction is still the only way to move things forward.

Advice to business leaders on managing AI teams

Michael Krigsman: Can you share advice, in general, to business leaders who want to apply these lessons and apply AI to their own business, whether it's in healthcare or pharma or not?

Bülent Kiziltan: Those AI strategies and experiences cannot be applied everywhere. Strategy has to be customized to the operations, to the priorities, to the culture, to the bottlenecks and barriers that an operation has.

We cannot have a strategy that we can apply to multiple places. A domain expert with experience has to go and analyze what the problems are, analyze the operational structure, and make recommendations accordingly. Take my advice with a grain of salt, but those are the things that are evolving and should be dynamically adapted to any company setting.

Michael Krigsman: How do you decide, and how should business leaders, in general, decide which problems are appropriate to likely have a solution, or you can use these techniques in order to make real progress? And, at the same time, which problems should you just stay away from?

Bülent Kiziltan: Data is the key. Understanding the data, what information is available is key in making that strategic decision.

Sometimes you can have a very talented pool of researchers that can go in, but if the information is not there, they don't have a magic wand to come up with a prediction and forecast that doesn't have uncertainties associated with it. Understanding what data you have available—and if you don't, bringing that data in, recording high-quality data—is step number one.

Michael Krigsman: We have another question from Twitter, and again from Diana McKenzie. She says, "Do you know if anybody has successfully patented an AI algorithm used to advance a therapeutic in discovery research or clinical development?"

Bülent Kiziltan: Yes. Patenting algorithms is an active area of discussion. I know of algorithms that have been patented, and I know companies have different preferences in that regard. Also, within the domain of AI, pioneers have been discussing what those patents mean, but there have been patented algorithms and the number is increasing as we are moving forward.

Michael Krigsman: Going back to the issue of choosing the problem, can you elaborate further on that? You said data is the key. What are other characteristics of a good problem to look at with data science and AI?

Bülent Kiziltan: We have to customize our approach according to company priorities, divisions, what impact area the team has a focus on. There are all sorts of parameters that go into the decision-making process.

Also, what type of talent you have, what your core capabilities are, how you are embarking on that journey. Do we have strong technological partners? Do we have the IT infrastructure in place?

Data is number one and then infrastructure, I would say, is number two. Then talking with domain experts to identify some of the high-impact problems is where I personally get very excited in engaging domain experts because we want to make an impact in society and we want to improve the lives of patients.

We know where we're lacking in the biotech and pharmaceutical space, so we then sit down and prioritize the key clinical questions to really understand, based on the infrastructure we have and the data we have, what are the questions we can tackle and really push the needle a bit to contribute to the society, contribute to better lives with our patients. Then contribute to the domain of AI, which is one of our focus areas as well, is not only focusing on our own domain but also contributing to the domain of AI in return, so we can be one community in pushing the technology forward.

Michael Krigsman: Data, infrastructure, and domain expertise, which then means that the choice of problem is dependent upon having a fairly elaborate infrastructure built already.

Bülent Kiziltan: Yes, it's a herculean effort, Michael. It's very difficult to appreciate from the outside, but building that infrastructure that talks to each other effectively, where data is streaming in from different areas, where there are lots of regulations in this space is a herculean effort. We're very lucky to have wonderful partners internally and externally.

Michael Krigsman: Bülent, as we finish up, any final thoughts about this domain, the domain in which you're working, that you would like to share?

Bülent Kiziltan: I used to have the chance and privilege to get together with the pioneers of the field. Just before COVID hit, we were hosting Yann LeCun, from Facebook AI, at Harvard to give a talk about energy-based algorithms, some of the work that they have been doing.

We were discussing how AI can use some of the approaches that we've developed in the domain in astrophysics and physics, at large. As years moved forward, I was expecting that the pioneers would develop an intuition about what model works better with certain data sets and problems.

What I have at least encountered and experienced is the opposite. He would say I'm paraphrasing. Most of the time where we find an innovative approach in architecture is an area where we expected the architecture to produce the weakest predictive power.

Things in the domain of AI and machine learning are not always intuitive. Some of the algorithms that are developed for big data sets actually can work and do work in the small data regime as well, but there is a lot of manual intervention that's required that sometimes make it impractical to improve.

There is a lot of cross-pollination between traditional statisticians, which has to happen, and the innovative machine learning researchers. Things are not always intuitive, and this a perspective to keep in mind when tackling any problem.

This is why I and my team are trying to build a core capability where we command the capability all the way from standard statistics, applied mathematics, all the way to the next generation of machine learning techniques. We go incrementally about addressing the problem, we benchmark along the way, and see where we can produce interesting information that our business partners or domain experts can use.

Michael Krigsman: Thus why you said at the beginning that the solution is not just technology but is this entire end-to-end chain of all of these pieces coming together (leadership, culture, communication) among all of these diverse experts, diverse team, as you described it.

Bülent Kiziltan: Absolutely. It's an ecosystem, Michael. This is why culture and leadership are critically important. We need leaders that can at least appreciate the intricacies and value proposition of the technology but also understands the sensitivities that are coming from individual units and also empowers us to be data-driven and engaging external partners in this journey.

Again, this is a one-team journey. We need to embark on that journey together. We produce value for each other and one plus one is greater than two (along the process).

AI (or any of the other processes) in isolation does not provide the long-term value proposition that we can produce together.

Michael Krigsman: All right. With that, I think our time today is drawing to a close. Bülent Kiziltan, thank you so much for coming back to CXOTalk and sharing your expertise with us. We really appreciate it.

Bülent Kiziltan: Thank you, Michael. It was great to reconnect with you and your audience.

But while we are on it, I wanted to ask you a question because you are putting this fantastic podcast together that we all drive value. We understand the different perspectives of business leaders.

I am pretty sure there are lots of technical intricacies involved. Would you mind sharing some of the challenges from your perspective?

Michael Krigsman: To be honest with you, every time I do a show, I think it's a minor miracle and I'm almost kind of astounded that the whole thing works. But, you know, here we are well over 700 of these and it seems to work. So, Bülent, thanks for asking. That's a long response, but that's what it is.

Bülent Kiziltan: [Laughter]

Michael Krigsman: [Laughter] Bülent, thank you for being with us today. Everybody, thank you for watching, especially those people who ask questions.

Now before you go, please subscribe to our YouTube channel, hit the subscribe button at the top of our website so we can send you our newsletter, and be sure to check out CXOTalk.com because we have great shows coming up. Take care, everybody. Have a great day, and we'll see you soon. Bye-bye.