Customer-Centric Data, Machine Learning, and the Internet of Things, with Adam Bosworth and Gary Flake, Salesforce.com

Michael Krigsman:

(00:17) …The smartest people that I’ve ever met in technology. Gray Flake and Adam Bosworth, who are both with salesforce dot com and I’ll ask each one of them to introduce themselves. So Adam why don’t we start with you. Tell us about your background and what do you do for Salesforce.

Adam Bosworth:

(00:44) My background primarily is building democratized platforms. I built one at City Bank for helping people manage their business for about 100 MIS in place. Build one of the first spreadsheets but with extensive programmability, Quarto, and went to Microsoft to help build the way to make it easy to build Windows and client server applications.

(01:11) Then did the same thing for the web. Went to BEA Systems to make it easy enterprise Java applications, went to Google and made it easy to build collaborative apps Google Box. And in the meantime I have been collaborating in helping marketing along, and Salesforce and finally came to Salesforce to help make an easy extensible platforms for this big mobile monitor world that we now live in.

Michael Krigsman:

(01:38) So tell us just briefly a little bit more of your role at Salesforce.

Adam Bosworth:

(01:47) I really have two roles; one is to make sure we are doing the organic innovation that we need to do particularly around platforms. And the other one is to specifically is to make sure that we build the right platform and product for what we call the IRT cloud, which is a platform for enabling customers to deliver literally billions or tens of billions events a day from their customers, and from their customers devices to a place where they can effortlessly adjust that, distil that, and use it, digest and distill that 360 profile about that as context to process all those events in real-time and cause the right business value to be generated from it to get the right things to happen, the right service events, the right opportunities and information to the right customers and partners to actually make their customers lives better. 

Michael Krigsman:

(02:41) Okay, and Gary Flake tell us about your background and what are you doing at salesforce today.

Gary Flake:

(02:50) So I’ve been working in machine learning, data science, internet related things, those kinds of areas for almost 30 years. My original background and my Ph.D.; is in computer science and in machine learning.

(03:07) In the 90s I spent most of my time in different types of research labs for the government, university, industrial labs, so I’ve became something of a student of failed R&D research models.

(03:22) I like a lot of people with machine learning background I made the transition to looking at a lot of data on the web in the late 90s. And post 2000 I had a lot of different roles in you know in some very important companies I was lucky to be part of the Chief Science Officer at Overture, which was the company that invented paid search. Yahoo required acquired Overture, and I founded and ran Yahoo research labs. I was a technical fellow at Microsoft and ran live labs, and I did a start-up. Now I’m actually the CTO of Search and Data Science at Salesforce where I run along with a phenomenal team the entire search infrastructure for Salesforce. Along the way I also wrote a book called the Computational Beauty of Nature in the 90s, which is a textbook over some complex systems and adaptation.

Michael Krigsman:

(04:16) So what we have then between you two is the intersection in a sense of Internet of Things as well as search technologies and analytics. So how did these intersect?

Gary Flake:

(04:36) I’ll jump in here and say each is the ying to the others yang in some sense. You know each is fuel for the other. And I would add you know machine learning, AI, data science to that mix as well.

(04:52) What’s really interesting about the current time that we’re living in right now is that we have seen finally this great convergence of data that is you know a side-effect of many different Internet services and soon to be accelerated even further with IOT. We have seen compute power through the ubiquitous deployment of cloud base infrastructure. And we’ve also seen the development or the refinement of different types of algorithms for making sense of these things.

(05:26) And so in combination, the compute infrastructure and the data becomes the fuel that makes machine learning and Artificial Intelligent much more productive, and in turn these machine learning and AI systems have an ability to produce massive quantities of data on their own and powering a new generation of devices and services to the operating of somewhat autonomously on the web.

Adam Bosworth:

(05:53) Yeah, I think that’s exactly right. As a concrete example I met yesterday with a large electric company. They have 33 million smart meters that generate power from multiple countries and in multiple different ways and modalities. And we’re busy focusing on how do we get the data in about what’s going on with those meters, and the generators, and the power. But what they need is as much as that they need to then apply data science for predictive and intelligent to figure out how the best balance that. How can they best see which customers will use what and when. And how can they best actually encourage customers to change their power consumption behavior in ways that would be more efficient.

(06:38) You know, everything that they asked me about was about data science, data analytics, and predictive analytics, and big data. And their company will be generating about 33 billion events a day for us pretty quickly in just running their meters and then growing from there.

(07:04) So we are seeing this very large amounts of data coming in, and yet the customers are really interested in saying it’s too much data to look up and it’s too much data to understand. I would take data science at a very large scale, and figure out what we need to start creating the kinds of benefits from this data that we want. I think it’s exactly correct.

(07:23) The change to a mobile monitor world has created a huge demand for the data science, for the predictive analytics that is necessary to try and control and manage this kind of information.

Michael Krigsman:

(07:37) Now one common thread that you just mentioned is the application of data science and analytics to solving customer problems, so maybe can you elaborate on that.

Adam Bosworth:

(07:53) Well I think Gary does a better job but you know just quickly what they want to do is every customer that I speak to literally without exception wants to predict behaviour in the part of the customer, employees, and partners in what would they likely do or what were they likely not do. What segments do they belong in, what are they likely to purchase, and what problems.

(08:18) They also want to predict behaviour on devices, when will this machine need to be serviced, when will this machine breakdown, what is the likelihood that it will have a failure. Is it likely that this thing actually needs more power or a different environment? So there is a lot of prediction that is about behavior, and there is a lot of prediction that’s about the hardware and physics.

(08:34) They want to do real time analysis in some of the events that they need to respond to really quickly in the event if something goes wrong. They want to actually then test alternative strategies, and what the all want, the holy Grail on this and it’s completely up Gary’s alley Ph.D. and everything he has done since, probably most importantly is they want automated self-improvement systems. They want a system where the more data they get, the better the system behaves and then more you know how it works, the better it does. So that at every time you’re trying something and that information is continuously making this system better and better. Gary you should take over because you’re more experienced in this.

 Gary Flake:

(09:15) Well that’s generous of you. But one thing I’ll add, I’ll make something of an analogy between the phase of development that we’re in right now and what has happened historically for example with transportation.

(09:30) So a couple of hundred years ago, if you wanted to cross the country there were you know very few options for you but you may have ended up doing something like stagecoach, where the person that was responsible for transporting you and helping you get across the country had a very high touch relationship typically with his passengers. And that path of being a stagecoach operator was arduous. It was difficult, you knew all of your passengers, and it had limited success.

(10:04) Later on, trains made their way, and the role of the conductor was such that the conductor may not actually know everyone that’s on the train, but they’re responsible for taking a bigger volume, a bigger mass across greater distances and at a higher speed.

(10:19) Then the airplanes came, and now we are doing a mode of transportation that no human, that there is any sort of natural analog for a human. Humans don’t fly. Humans don’t naturally in anyway move at that kind of speed. And now when you take a look at the latest generation of aircraft, in some ways what the pilot is doing is no longer flying anymore. They aren’t actually controlling something in the most advanced planes that are actuating the different portions of the wing, to changing the angle or anything like that. What they’re doing is they’re interacting with the plane through a metaphor. The metaphor is I want to go in that direction or that direction, and the system knows how to translate that into actuated movements of the physical equipment.

(11:13) So in that process, what has happened is that we’ve gone through a phase where transportation resembled something that we once naturally did. But now transportation is completely unlike how we as human actually naturally move. And the role of the pilot is actually very different. And as a result instead of merely being able to go twice as fast or three times as fast, we can travel now hundreds of times faster than what any human could do with transatlantic flights, vastly further, with more people than nature would have ever allowed us to do.

(12:01) Now connecting the dots between that matter further or analogy to where we are today, the person that is interacting through big data, data science, analytic or machine learning system in the past they would have to look at individual records, individual things. Just like a stagecoach operator you know, somehow having to negotiate over every single bump on the road.

(12:31) As things have become more automated, the ability to scale, to more density, to more pieces of data it has gotten even more powerful. But now with automation, machine learning, and analytics or riding on top of a larger system, the role of an analyst sitting on top of this information is no longer to look at every piece of data piece now. It’s actually to help guide system to yield an ever richer form of analysis, and therefore an ever richer insight. And this is the sort of thing where what the human is doing. This is no longer about the manual labor if you will. It’s much more supervisory in terms of teasing out hidden patterns, hidden segments.

(13:22) And in such and in that way what one person can do with a new modern analytic system is akin to what a pilot can do on an aircraft. We can now do things at a scale, at the speed, at a volume and velocity that no human could ever naturally do with the power of their own mind. But by working through a big data and machine learning system, we get a multiplier on what we can do and perhaps thousands and millions times more productive and more powerful than what a human alone can do.

Michael Krigsman:

(14:01) Adam, this notion of a relationship of the human to the data and to the technology, maybe you can elaborate on that, because that seems to be an indicator of where our future is heading.

Adam Bosworth:

(14:18) Well my favorite example and funnily enough very apt because this actually happened this week again is my wife has a monitored car and a mobile app for that car. And periodically and mysteriously this light comes on which is the engine light and she knows the car is monitored. And her point of view is whatever sensor and whatever computer that is inside the car knows it that the engine light is on should also be known by the car vendor. And then therefore they should know what’s wrong with it, how long it will take to fix it, which dealer can fix it, who has what availability and understand the schedule. And automatically before she ever sees the light have suggested to her that she needs to go there and get a loner and look at the car.

(15:06) And what you have there is an expectation failure because the expectation now everything is not technical, and if you are monitoring something you are predictably intelligent about it. If you are monitoring something or watching 24/7 and you are smarter than any human and scale in the way that Gary just said, and if anything is about to go wrong, it will go wrong. You have already figured out what to do and already put together a plan to help before you even find out as a customer that the light is on. Obviously that’s not her experience, her experience is 10 minutes on hold and the dealer says I don’t know what’s wrong with the car right now and I have to look at it, and take the computer out of the car. And her general attitude is well you guys are bozos.

(15:46) So the human expectations have been change profoundly by the concept of things being monitored. And they’ve been changed to say if you’re watching 24/7, then you should be able to be as smart as a human and scaled infinitely up across all of the kinds of learning that we just discussed. And if you don’t, you know, you’re not delivering to me in the quality of service and support that I expect. And that is a very interesting time we are in where as a human you’re now expecting these things. You expect your smoke alarm to tell you before the battery starts beeping and waking up your kid at two in the morning. You expect the car to tell you before the engine light goes on and you know we see these cute things and expect the car to park itself, and soyou can’t get out of the door if you do it.

(16:33) That expectation is actually moving faster than the software, not because the software can’t do it. It’s because the companies are very slow and taking advantage of this software. Gary is absolutely right, Moore’s law has been unbelievably powerful and taking these things, and where we want to go but  not necessarily cost-effective in taking that possible amount of scale to a completely different level.

(16:58) When I got to go to Google in 2004. You know I was just stunned by how could you type in this query against a billion pages, so I typed in, boom. Back came an answer until I went and looked at what they were doing. And really what they were doing wasn’t as much brilliance, but brute force. It was a brilliant application of brute force and Moore’s law, and the fact that memory and computer speed had both gotten you know massively cheaper, massively bigger, and massively faster than they have been when I started with computers.

(17:30) And people’s expectations have changed very quickly. When my son speaks through the echo now, if the echo doesn’t understand him, he gets annoyed. His expectation is that the echo should understand him. And you know, by the time he is my age I would be surprised if a human will be driving a car. In fact, forget my age, by the time he is able to drive a car, I would be surprised if anyone would be driving the car, and I would be surprised if he can’t tell the car the direction is kind of an expectation. It’s just moving at an extraordinary speed compared to you know what we have seen in the last two decades before that.

Michael Krigsman:

(18:04) So Gary Flake is that your job in a sense, is to meet these kinds of expectations for your customers, essentially help your customers meet these expectations for their customers.

Gary Flake:

(18:20) Well that’s a lot of void that you have just dumped on my shoulders. I have a lot of jobs; I wear a lot of hats. And the job that you specifically mentioned is something we all care about at Salesforce, so is everyone’s job not just mine, is Adam’s job as well. And you know I love how Adam puts the you know, the change in psychology that we’re seeing in people in terms of what is you know what our expectations are changing and we expect to interact with devices.

(18:55) I’ve seen it with my own kids as well. You know my two boys are of the same generation as Adam’s youngest, and I love watching how my kids interact with a Luxo with touch interfaces and things like that. And they just like Adam’s wife, and just like you know, my wife and everyone in my family we all have our ever increasing rising bar for what we would like to see.

(19:23) And I think that you know we are at this interesting crossroads in the history of computer science, and the history of machine learning and data, never has so much change so fast. You know, we’ve had a number of different historical milestones that you can site on one hand, and one might be the Renaissance, the industrial revolution, the invention of the printing press, the telecommunications you know, all of these different revolutions that have happened in other times.

(19:59) But what’s happening right now is simply unprecedented. I think in a large part what’s happened it can be characterized by the following kind of little pithy story. Back when I was working on my Ph.D. say you know, in the early 90s in machine learning, a lot of was used to say they are all networks in machine learning were the second best way of doing almost anything. And that was true, and that you could handcraft a solution that would be better than what a network or some other machine learning system could do.

(20:42) But the ability to take a certain toolbox like the machine learning toolbox and apply it across multiple domains and come up with something that was good enough for most applications that was powerful but it wasn’t revolutionary because again it was the second best way of solving almost anything.

(21:00) What has changed now and what is truly mind blowing to me is that machine learning approaches for a wide variety of problems are on their way to becoming the best solution for almost everything. And so when you take a look at lots of different minor revolutions that we’ve experienced. So for example you know, we went from this point where speech detection just didn’t seem to work and you know, that was the state of the world 15 years ago. It was clunky, it was awkward, and most people who tried to do the spoken word to text quickly got frustrated with it and didn’t use it.

(21:43) And now, because of the ubiquity of hand-held phones, and the data that has come from people just speaking in them, that virtuous cycle of clucky data, analyzing it, producing better algorithms and data solutions has yielded speech detection systems that are almost at the same level of humans in terms of their ability to tease out words and vocabulary, given a variety of different audio sources and a variety of different context.

(22:13) And so that’s almost magical to see that transition happening because it’s happening, not just in the domain of speech, but we are all seeing it in machine translation. We’re seeing it in self-driving cars; we’re seeing it in a wide variety of domains. And so while it is an interesting moment of time right now where we still can be surprised, our expectations can be – we can be a let down by our own expectations, because we expect you know the cloud to be smarter than it is sometimes. We’re at this threshold where the next thing that happens is, it’s going to have capability and be able to do things that will over deliver. It will be smarter than what we expected instead of you know, failing.

(22:59) So I think in a lot of ways I tell people right now is the most interesting moment in the history of the universe to be working in computer science, to be a programmer or working in the cloud in some way. And I think this particular window of time will go down in history as a fairly magical moment.

Michael Krigsman:

(23:22) And so Adam what are the implications of this for the things that you’re working on around Internet of Things and collect that data and analyzing that data.

Adam Bosworth:

(23:37) I mean the implications are that we need to do three things at once. We need to provide our customers with the ability to effortlessly listen to, digestive and respond with context to the data, that's step number one. Well Gary is absolutely right that machine learning is increasingly to become the best system. You need a fair amount of activity and a fair amount of data for it to work reasonably well and it’s nice to jumpstart that data.

(24:13) For example if the car vendor we’re actually going to start taking proactive actions then we’ll probably want to take a bunch of choices and figure out which is working best and learning from the outcomes of those actions.

(24:29) The second thing is we need to enable not just our employees, our data scientists but our customers data scientists. It is becoming a fundamental part of the data processing paradigm and I start with ingestion and I talked about how you do ETL and ultimately produce a 360 profile across this huge activity and this huge set of data coming in. but then I show this virtual cycle between the data science, both taking this activity and the data profile as of at the time of these activities occurred computing the things it wants to compute, and then updating the 360 profile with the intelligence that it can gleam with the proposals, the actions, the predilections so that it can figure out. And only then do you start to actually use segmentation and try and figure out who is doing what because these totals are essential to both influence what they do and to describe who should do what.

(25:30) So data science has become an integral part of the data processing cycle in a way that it absolutely wasn’t as recently you know at Microsoft in 99. And you can’t really talk about data processing any more without talking about this as being part of the cycle.

(25:48) And I think Gary said it accurately earlier with his analogy. It’s just too much data and too many responses and too many intelligent processes running at too large a scale to have humans handling each one. You know we are talking about things where a single one of our customers just generating billions of events a day to us. At that point you absolutely have to have automated intelligence deciding what to do. You cannot have a human look at this one on one the way in the stagecoach analogy worked, and say, well looks to me like you should you know clean your carburetors if you still have carburetors. Instead all of this has to be automated feedback or it just won’t scale otherwise.

(26:26) So you know I have nephews and nieces in college and all technical and one thing I tell all of them is make sure you study data science, make sure you study machine learning because largely speaking I completely agree with Gary that algorithms while important are increasingly less important and understanding the specific set of algorithms about how do you automatically learn from data which is so critical now. 

Michael Krigsman:

(26:49) We have a question from Twitter and Arsalan Khan is asking about how do you make this available to customers who are not data scientists, who just simply want to use this data. And so what kind of burden does that put you, and what do you go through thinking about the translation to customer use.

Gary Flake:

(27:16) I’d like to take a stab at answering that and I want to riff off of what Adam actually just said. It’s true that normal people, you know it’s not usually reasonable to be pushed out on our customers that everyone has to become a data scientist.

(27:38) What we’re seeing now is something that Adam alluded to and I want to give a concrete example of this. I think of this what I’m about to describe as closing the loop. It’s really the creation of the feedback loop as a fundamental part of the application architecture. So hold that in mind, just for a moment. And think back to one of the earliest may be the first time that you know you’d used your phone to do speech to text again. You are trying to dictate a text message or something else.

(28:12) In the earliest days of using text-to-speech on the phone, what would happen is it would make mistakes with a greater frequency than what we see today. And you would have to you know use your fingers to press on the word that was incorrectly converted and make the correction. And sometimes you would just delete the text and put in what it should have been. Other times, it’s got a little bit smarter and it would ask which of these elements did you mean.

(28:45) That little piece of UOI right there where not only are you taking the input from the user, not only are you giving an answer that is tentative, but your instrumenting the response from the user the feedback, that is a signal as to whether the output was correct or not or could have been improved or not.

(29:06) That break there was in some ways the critical architectural moment for a lot of these machine learning systems because it became part of the application itself to retrain it to correct the system. And people not only didn’t realise that at the time that when they made the corrections on speech-to-text systems they were actually becoming participants in a huge machine learning system that was making speech-to-text systems vastly more accurate than they ever had been before.

(29:41) Now you multiply that by the billions of people that have phones, that are speaking into their phones and interacting with then in that way, and suddenly you have a treasure trove of data the likes of which would have been impossible to have gathered merely a year earlier or five years earlier. That data of spoken words to the tentative answer of what the text should be and the correction given by a human times billions, that’s the sort of fuel that you need to power a machine learning system.

(30:12) So the answer to your question, what do we do and how do we make things much more accessible, well in the future I think what we’re going to find is that a lot of systems, whether that’s enterprise software or consumer or in the health care or other domains are going to have that property that you will interact with the system, it will give you an answer and you might feel compelled to correct it or refine it or give a thumbs up or a thumbs down on what the response was and that will be instrumented. And that will be instrumented times a billion, and by framing the application in that way it will be self-improving.

(30:52) And it will be improved not just by your own interactions – yes it will learn to mirror and build something of an implicit model of what you like and what you don’t like. So scenarios of personalization are possible. But when you multiply that across whole populations it will be learning the collective wisdom of all the people who are interacting with it.

(31:12) So it’s that kind of feedback mechanism that will become I think the default way that people come to interact and learn and improve, so that you don’t have to be a data scientist. You don’t have to be a Ph.D. in machine learning in order to get the benefit.

Adam Bosworth:

(31:30) So my answers a little bit different. I mean as I said my background is trying to democratize platforms, and you know frankly the current state of data science is one I wrestle with and I completely agree with Gary premise. In fact one of the core things we’ve done in the IOT cloud that we felt is that we made every possible outcome that comes out of every rule that you write you can signal basically whether or not you see a better or worse result. So we’ve created automatically the create signal precisely so we can learn from the application of those rules which one works better than others and apply machine learning to that.

(32:10) So to the extent we can automate the learning process we want to. But we have plenty of cases where the real problem is getting the data pulled together in the right shape, state, or structure so we can learn from it. And Gary and I have had numerous discussions within this company about to what degree can you automate that process, and to what degree is in auto, rather than an automated process. Not only other than that there seems to be pretty well understood ways. Once you have relevant data put together to try various learning systems and figure it out what is the best predictive model.

(32:46) But rather than the art of getting data into a reasonable state in the first place, and computers are so powerful now and so fast that there is you know there is a lot of discussion going on right now that you can try all possible permutations as part of the computing thing.

(33:00) What we’ve been looking at though is that we’ve been trying to structure the systems so that it is very easy for you to describe pretty much what Gary just said, which is I’m going to try something and I will describe basically based on an outcome whether this was a good outcome or a bad outcome. Then we will ask the systems to start picking the ones that work, and not picking the ones that don’t work, based on that outcome. And that is pretty easy for us to automate, and that’s because we take the burden off the consumer and essentially it’s training the systems as Gary said earlier.

(33:32) I mean a great example that we all have angst about is whenever someone on the customer service desk answers on something on fixing an automated response and says that’s the right response, he or she is training the system to the point where ultimately, you might don’t need the customer service desk. And that is actually the goal is to train that system to do at least a good or better job in answering 99% of the questions and so let the humans just deal withthe 1% that aren’t acceptable to that.

(34:04) But the actual business right now. It’s a little bit like a UX design. Right now there is no magic that we figure out for how you could just automate good UX designs. Why doesn’t design centric product designers become so prevalent as we move into this new age, and it’s very similar like you know we are still seeing that our customers are not having fewer data scientists; they are having way more. When I started doing this I expected that maybe five or 10% of our customers would have some data scientists looking at the data. At this stage I would say it’s roughly 85% and customers can afford that, but it pretty interesting to see that something like 85% of the customers we are working with already have hired a bunch of data scientists and they have them looking at the data. And they already have them working on this, because they have to, because they simply can’t afford not to apply this expertise to looking into data otherwise they can’t take advantage of it.

(35:03) So we’re wrestling with this, we’re wrestling with how do we democratize it. And where we can, which is mostly about automating the engagement patterns for human behavior and we are going to start and try to automate that. Where it gets harder like for example, recognizing that in a computer just like an engine will fail in a car, you know where the signals are very different from the actual outcome and try to correlate them isn’t easy. Right now we are telling them that they need data scientists and we are trying to figure out how could help our customers to do that more easily. But democratizing that is still something I think is still a little harder than what Gary said. The democratizing the process practice that, yes anyone who has been ever involved in any system that’s self-learning is training that system. But building that system for any given signal is not always so easy in my view.

Michael Krigsman:

(35:51 – 36:29) No audio

Adam Bosworth:

(36:29) I’ll take the first stab at this and Gary will be more articulate, so I would rather go first. You know the truth is I think what we’re seeing is the beginning of an age where enormous amounts of business processes, armed basically automated data artificial intelligence. And you’re right, they run automatically and almost scarily to take advantage of all these signals coming in, because it’s not possible to keep up with it algorithmically. And as Gary said earlier, and given enough signals, the data science is starting to beet the algorithms, and I first saw that when I was at Google and we were looking at clicks per rate on ads.

(37:11) Clicks per rate on ads is a startlingly easy thing to measure, and the data science started picking on which ads should be clicked on, and it began to be anyone else’s algorithm, and as hard as they tried no one else could catch up. And that was 11 or 12 years ago now, and watching this and going okay, that’s the future, you know. So the implications, to be blunt are you have to harness this stuff. If you don’t harness this stuff, it’s essentially like trying to bring your goods to market in a horse and buggy when other people are driving in trucks; you will not be competitive. You will not be up-to-date, you will not have actually been able to do good a job meeting your customer’s needs, where someone else can say I can actually learn from this data, and I can improve the system, and I actually use the customers in the way Gary described to help train that system. You’re just going to lose, and so the implications are that the business processes increasingly will be automated. And we hear a lot that what people are learning these days, which involves again other kinds of technologies where we understood it a while ago but simply wasn’t cost effective.

(38:16) But again, those systems people start to rely on them and they are going to rely on them for image recognition, they are going to rely on them for recognizing whether or not something is actionable or not, and those business processes are going to be baked in in very subtle ways and very concretely, and I’m going to hand it off to Gary. I had a gift I was rushing off to my wife for her birthday a few months ago, and I came home on her birthday had Amazon rushing to be there in two days. Checked Uber, my ride was where I was taking her out for dinner, and checked the table and the reservation was fine and I went to look for the gift and there was no gift.

(38:59) But I had and astonishing piece of email that just come in and said, your product was damaged in shipping, we are really sorry. We have rushed you a new copy and it will be there tomorrow morning. Now, what was impressive was the level of customer service was nice. What was impressive when in thinking that through and realizing that was completely an automated response. You know, there is finally a business process for figuring that was possible to rush that gift to me, and getting that gift rushed to me, and sure enough, at 10 o’clock the next morning they dropped it off. That business process almost certainly used intelligence in learning to figuring out the optimal strategy for me, the customer specific to who I am, and how much I buy from Amazon and taking the cost on the head to do that.

(39:43) And that’s not going to go away. Right, that kind of constant use of intelligence as opposed to some programme reading a set of rules, and saying here are the rules for my code and what we are going to do, here are the algorithms. You know increasingly it will not be algorithms; it will be data science and the only time it will be algorithms is when you’re jumpstarting a system, and you don’t have enough response to start doing that learning and you are just getting the data up and going. But it’s really priming the pump in my opinion. Gary.

Gary Flake:

(40:10) Yeah, I think Adam pretty much nailed it, I want to riff off of what he just said add spice it up with a couple of things. But he is the headline, here’s the summary. The pattern of what we spoke about before where you could design and application to be self-improving by having a feedback loop with the user. The user is giving some sort of indication as to whether the system is working well or not. That can be generalized and expanded to the whole of the company’s performance.

(40:41) And I think Google was really as Adam said was really the first company to really kind of master that pattern. But a little bit of information a little bit of an anecdote if you will about how that evolved. So I was the Chief Scientist Officer of the company called Overture who invented paid search. Paid search had the very first inclination of it was a very human intensive activity. So we at Overture employed human editors to review every single listing that went on. We had an editorial standard on what could be matched in terms of query to a listing that had met or exceeded. And the idea of introducing a fuzzy matching algorithm that had some intelligence on it was at that time not acceptable to our constituents, because they wanted to understand. They wanted transparency with how the system worked.

(41:38) So we created the business model, but right away we had painted ourselves into a corner. We weren’t allowed to really clearly innovate that in a way that would be you know less than transparent. But what we could do is we could do some things like – you know, people understood what a spell corrector was. For example at the time, we probably had 24 different variations of how people would commonly misspell Britney Spears as a search term. So you can imagine that if you could automatically figure out all the different ways that you could spell something like Britney Spears and mapping it to a canonical answer, that’s a good thing for the user to understand.

(42:19) So you know, it was around 2002 where we build a spell corrector of that nature. It has to operate at five mills second at its process of about 60% of all the queries around the world going to all the different search engines. That’s the kind of scope that we had. And that system was able to improve and five mills second could you are correct, the misspelling of Britney Spears and among other things. And what was amazing was was when we turned that on, that created tens of millions of dollars of found revenue for Overture and the company. Tens of millions at that time just from spell correction, which is truly kind of mind-boggling when you think about it.

(43:07) And so this was for us a real indicator that if we were going to succeed what we had to do was we had to make our system self-improving. We had to actually analyze the flow of the data and change the functioning of the business as a result. Now, I said earlier we kind of painted ourselves into a corner, and Google had the good fortune of arriving second into this particular race and they designed it right on day one to be a self-improving system. So they used click through rates as a way of accessing search quality rather than, say, a human editor. They had self-serve models for how the customers would sign up for advertising, or how distribution partners would sign up to show ads. The really automated the whole thing, and as a result whereas we had many humans in the loop trying to automate it, they did not have a human in the loop and they could actually do the whole thing algorithmically.

(44:15) And so what happened was that in a span of about 18 months, Overture was asleep at the wheel because they were acquired by Yahoo as we’ll know acquisitions, big acquisitions sometimes slow things down. Google had the benefit of accelerating that process and they went on to kind of conquer the world in some ways, the world of paid search engine as a result.

(44:40) So the bottom line here, and the answer to your question, what does your audience really need to think about, I think the answer is, how can you reframe, re-express your business to be more like Google, or to be more like one of these other canonical companies that is optimizing their very business, they’re very business model through the flow of data and information that flows through it.

(45:06) And so how can you get that feedback loop, how can you architect your whole business to an equivalent of that instrumentation of the user giving some correction to the system. That could be you know, a customer complaining, a customer sending a thank you, or any number of signals that customers give us explicitly and implicitly for how we’re doing our jobs. And using that to literally improve the performance of the whole company is I think is the future.

Michael Krigsman:

(45:39) We have just a few minutes left,, we’re just about out of time, so Adam may be some final thoughts from you on what we have just been talking about, and especially this implication on customers, their business models and the things that you were talking about, and Gary was just talking about. And then Gary will come back to you for final thoughts and that will be that.

Adam Bosworth:

(46:04) I think Gary picked one of my favorite examples, was when I got to Google I wondered how was it for that. There was essentially this universal magic spell checker that knew every first name and last name of everybody in the world. And went around and asked and it was really just brute force data. Every time you typed in anything and no one would click and then type something else and it did click and was it a possible misspelling. If you got a certain number it was a probable misspelling. If you got a certain ration you know it was a certain misspelling; didn’t even need to know machine learning. It was unbelievably efficient brute force.

Gary Flake:

(46:37)Statistics.

Adam Bosworth:

(46:38) Right and you know my jaw dropped because they had designed the system what Gary had just said, they had designed the system to be self-learning by counting. I mean this did not require LDA and this was you know counting and they had built this beautiful thing. And I can do no better in the last minute than what Gary just said which was to the extent you can design every system to have a feedback loop and the feedback loop tells you basically whether something is working or not working, and you can figure out or better yet what did work.

(47:13) My son asks the ZX to play a song and doesn’t understand and he asks again I am sure then Amazon is trying to then learn how could they have understood that song in the first place and if they’re not they should be.

(47:26) And you know that feedback loop is everything because if you can design the system to just count you know never mind do data science already you can do an extraordinary job compared to what you could do if you didn’t design the system that way. So I think Gary nailed it. That is the essence and it’s so simple to start with. The essence of what we’re learning is take your data, take your customers actions and use the combination to have a better level of service.

Michael Krigsman:

(47:53) And Adam just very quickly. Any advice that you have for your customers and for the people that are listening in relation…

Adam Bosworth:

(48:04) We tell them the same thing, we said look, it’s a whole new world and it’s like when the web came along and it was a whole new world. Pick some very simple things to start with. Take some subset of this signal that you care about, a lot of PSE is very fast and we try to help them build a very agile platform. Test them all, get them learning and learn as you go. Don’t trying boil the ocean, don’t try to rebuild your entire business at once. Pick the most simple and the most valuable thing you can think of and do that. Then the next month most valuable and simple thing, do that and tried seven or eight or nine riffs on that and just be super agile and superfast just as we used to be on the websites. Start learning how to operate in this new world because the sooner you learn the sooner you will get good at it.

Michael Krigsman:

(48:46) And Gary Flake you will get the last word here, what’s your advice and your distillation of everything you know and what you can share with your customers right now in about a minute.

Gary Flake:

(49:02) Well in terms of the outlook and the advice that Adam just gave for the immediate and near future you know, I don’t think I can improve upon that. But what I’ll do is I’ll make an attempt at extrapolating that a little bit further into the future.

(49:19) We talked about this virtual cycle, this feedback loop as something to embed within an application and also is something to embed within a business. There’s also a great deal of discussion and debate and concern around the future of artificial intelligence and the future of humanity and what this means.

(49:43) And depending on who we talk to, you’ll hear two very different stories. One is that we are on the threshold of an AI utopia, and we’re on another threshold of the AI apocalypse and so depending on who you believe you can either take a very optimistic or pessimistic view of the future.

(50:01) I think that the more likely path is actually a third path; that’s a little bit in the middle. And that is taking that feedback loop and applying it to one, one person. In the future we will be interacting with our environment, both in the raw in the sense of as we do today. But we will also be interacting in the environment through some layer that is effectively aesthetic with some of these technologies.

(50:38) And if you think about how that feedback as applied to one would operate, it could actually be something that is really revolutionary for one person in for what they can do and what they can be capable of doing.

(50:52) And so my whole thing is don’t fear the future. Don’t fear the AI apocalypse or the AI utopia. Instead I think there is going to be a world in which humans and machines are actually working together in a way that the combination is much more powerful than any one piece alone. And so to the doomsayers of the people who fear for humanity and say yes, AIs will become incredibly powerful, but there will always be one thing that’s much more powerful than an AI and that is the human working in conjunction with an AI.

Michael Krigsman:

(51:36) Okay wow, this has been a fascinating conversation with Gary Flake and Adam Bosworth from Salesforce. Gentlemen, thank you so much for taking the time and talking with us today.

Gary Flake:

(51:52) My pleasure.

Adam Bosworth:

(51:53) Thank you.

Michael Krigsman:

(51:54) It’s been great, you have been listening to episode number 171 of CXOTalk. Come back next week and will be talking with you Chief Technology Officer of Accenture. And thank you as well to Kathleen Obada and Michael Ellis from Salesforce for helping us get all of this setup. Thanks everybody, have a great week and we’ll talk with you soon. Bye bye.

Companies mentioned on today’s show

Salesforce                   www.salesforce.com 

Google                         www.google.com

Microsoft                    www.microsoft.com

Yahoo!                         www.yahoo.com

 

Adam Bosworth

Twitter            https://twitter.com/adambosworth

LinkedIn           https://www.linkedin.com/in/adambosworth

Gary Flake

Twitter            https://twitter.com/flakenstein

LinkedIn           https://www.linkedin.com/in/garyflak