Kirill Eremenko: 00:00:00
This is episode number 385 with Lead Data Scientist, Scott Clendaniel.
Kirill Eremenko: 00:00:12
Welcome to the SuperDataScience podcast. My name is Kirill Eremenko, Data Science Coach and Lifestyle Entrepreneur. Each week, we bring you inspiring people and ideas to help you build your successful career in data science. Thanks for being here today, and now let’s make the complex simple.
Kirill Eremenko: 00:00:44
Hello everybody, and welcome back to the SuperDataScience podcast. Super excited to have you back here on the show. Today, I’ve got a super amazing treat for all of us. I just got off the call with Scott Clendaniel. Scott is a lead data scientist there. He has a huge amount of experience in this space of data science and machine learning, and he’s always happy to give back to the community. This podcast is going to be an advanced podcast. It’s specifically going to be useful to you if you are an intermediate practitioner in data science or an advanced practitioner in data science.
Kirill Eremenko: 00:01:18
You’re interested in things like models, cross-validation, over sampling and things like that. With that said, here are some of the topics that you’ll hear about today. You’ll hear about Scott’s story and how he got into the space of data science. We’ll talk about fraud detection because it’s a big part of the financial services industry. We’ll talk about some specific examples of ways to detect fraud, including Benford’s law. We’ll talk about oversampling, the minority class, the multiplicity of good models, the tools that Scott uses on a daily basis.
Kirill Eremenko: 00:01:51
We’ll talk about data preparation techniques. Specifically, we’ll talk about target, mean and coding, and one hot encoding, what they are, which one is better, and why. Then we’ll talk about model drift, and we’ll discuss why models decay over time. Scott will give his recommendations on how often he checks up on models. We’ll talk about building populations to build to report. We’ll talk about some real world examples, cross-validation. Then we’ll cover off Scott’s advice on some of the softer skills like data science leadership, what it means to manage data science teams, how to best structure a data science team, the hub and spoke model for that.
Kirill Eremenko: 00:02:28
We’ll get Scott’s ideas and visions for what is coming for data science in the future. A very exciting podcast coming up ahead, I can’t wait for you to check it out. Without further ado, I bring to you Scott Clendaniel, lead data scientist at Franklin Templeton.
Kirill Eremenko: 00:02:51
Welcome back to the SuperDataScience podcast, everybody. Super pumped to have you back here on the show. Today, we’ve got a super special guest joining us, Scott Clendaniel. Scott, welcome.
Scott Clendaniel: 00:03:02
Thank you so much. I’m really happy to be on.
Kirill Eremenko: 00:03:05
That’s awesome. I forgot to ask you, where are you located right now?
Scott Clendaniel: 00:03:09
I am actually in Havre de Grace, Maryland off the Chesapeake Bay, about 45 minutes north of Washington, DC.
Kirill Eremenko: 00:03:16
Havre de Grace, Maryland.
Scott Clendaniel: 00:03:20
Absolutely. [crosstalk 00:03:23].
Kirill Eremenko: 00:03:23
Havre de Grace is the name of the city.
Scott Clendaniel: 00:03:26
Yes. Port of mercy, I believe, is the loose translation.
Kirill Eremenko: 00:03:32
Very interesting. How did you end up there? Have you been there for a long time?
Scott Clendaniel: 00:03:38
Actually, we just moved here. We had an opportunity because my job allows me to work remote to be able to change location. My wife as a children’s librarian just got a position up here near Cecil County. We just moved here, and we’re pleased as punch. It’s really pretty.
Kirill Eremenko: 00:03:57
All right, tell us… Before that, where were you located before that?
Scott Clendaniel: 00:04:03
I was actually born in Baltimore, Maryland, lived there for a long period of time. Then I was in Delaware, Pennsylvania, and a few years in Honolulu, Hawaii.
Kirill Eremenko: 00:04:15
Amazing. I’ve always wanted to go to Hawaii. What is Honolulu like?
Scott Clendaniel: 00:04:20
Honolulu is really a fantastic city. I really loved living there. Obviously, a lot of tourism, but the people there are just so warm and inviting, really had a good time there, a big fan of Hawaii.
Kirill Eremenko: 00:04:38
What Island is Honolulu on?
Scott Clendaniel: 00:04:39
Oahu, which is the main Island. About three quarters of the population lives on that island. There’s the total [crosstalk 00:04:48].
Kirill Eremenko: 00:04:47
Wow. I heard there’s a great restaurant on Maui, and it’s called Mama’s Fish House, I think. Do you know that?
Scott Clendaniel: 00:04:56
No, I don’t know that one. I haven’t been there, but I visited four of the eight islands while I lived there. Maui is very pretty. Each island has its own personality, which is fun.
Kirill Eremenko: 00:05:10
What’s the differences in personality?
Scott Clendaniel: 00:05:10
Well, Kauai is the garden island, so it tends to be much more laid back. It is probably the greenest of the islands. That’s great. The big island has all kinds of different topography. It’s probably the only place in the world where you can go snow skiing and water skiing on the same day.
Kirill Eremenko: 00:05:28
Wow.
Scott Clendaniel: 00:05:29
Each island has its own unique factors to it. Maui’s a lot of fun too. Oahu is where most people go. That’s where Waikiki is. That’s probably the most popular of the group.
Kirill Eremenko: 00:05:41
Oh, fantastic. Very cool. The mountains are tall enough for skiing?
Scott Clendaniel: 00:05:46
Only on the big island. When I say snow skiing, I’m not talking about Aspen or Vail, Colorado. I mean, you can get about that, but you can at least tell people, “Hey, this is fantastic. On the same day, I went water skiing and snow skiing.”
Kirill Eremenko: 00:06:02
Amazing. Okay. Got you. Well, Scott, it’s a pleasure to have you here. We’ve got a lot to go through. For those who are listening and maybe don’t know, I posted on LinkedIn just, “Hey, Scott Clendaniel is coming to the show in 24 hours, and post your specific asks for advanced data science questions.” In that 24 hours, there’s now 56 messages in there. Thank you very much for taking the time to really answer to some people. There’s been a lot of cool discussions and [crosstalk 00:06:33].
Scott Clendaniel: 00:06:32
Absolutely.
Kirill Eremenko: 00:06:34
I really want to go through [crosstalk 00:06:36].
Scott Clendaniel: 00:06:35
Very few of them were from my bill collectors, which I really appreciated.
Kirill Eremenko: 00:06:41
Gotcha. What I want to start with though, so we’ll definitely get to those, and there are some really cool advanced machine learning questions, model and cross validation and things like that. Before we get there, give us a bit of a background around yourself. Who is Scott Clendaniel?
Scott Clendaniel: 00:07:02
Sure. Actually, I have a bit of an unusual background. My undergraduate, my MBA are both in strategic planning. They’re not in either statistics or computer science, which makes it relatively different from a lot of other folks in the field. I was in financial services for a long period of time, and all the way up through vice president of consumer lending at Bank of Hawaii. Unfortunately, I had a family emergency where my ex-wife at that time decided to take my son, and so I had to leave.
Scott Clendaniel: 00:07:33
She had indicated to my son who was only three years old at that time that I had actually gone, because I was packing his toys in Hawaii. I was like, “Why did you do that?” Anyhow, so I got a phone call from my three-year-old son one day. He said, “Daddy, are you done packing my toys yet?” It was the worst thing ever. I was like, “Gosh, I’ve got to figure out how to give up my career in financial services and do something else.”
Scott Clendaniel: 00:08:00
I thought to myself, “I wonder if anyone’s interested in this machine learning artificial intelligence stuff. I wonder if I could do that.” For the next 16 years, I became a consultant. To all those folks out there who are trying to break into data science, if I can get past that, you can get past whatever you’re facing currently. I encourage everybody to give it a shot.
Kirill Eremenko: 00:08:23
Wow. Wow. What a story. You just packed your stuff, gave up a vice president position at Bank of Hawaii, and moved back to mainland U.S. How did that go for you?
Scott Clendaniel: 00:08:36
That was rough [inaudible 00:08:38], but it opened up a world of opportunity. It also gave me a whole new perspective on what’s important in life and what isn’t, and also gave me a lot of focus on persevering and problem solving and how I could add value to others. It was a tough thing to go through, but it provided a lot of opportunities for me later on in life.
Kirill Eremenko: 00:09:03
Wow. Wow. Amazing. 16 years past. You were consulting in this space. What happened next, and where are you now?
Scott Clendaniel: 00:09:16
Absolutely. Morgan Stanley actually recruited me to be their first full time data scientist in their Baltimore office. That was great. Unfortunately, they had a situation with some internal fraud, where… I can share this because it was on the front page of the New York Times. Someone walked out the front door with $11 million. They suddenly changed my role to focus exclusively on internal fraud, which wasn’t really what I wanted to be doing at that point in time.
Scott Clendaniel: 00:09:46
I wanted to stick with machine learning. I was recruited away from Lake Mason, and had been there for two and a half years. We just got purchased by Franklin Templeton, so I’ve been trying to help build functions for those organizations.
Kirill Eremenko: 00:10:02
Just to give a bit of background on the Franklin Templeton, this is one of the world’s largest global investment firm.
Scott Clendaniel: 00:10:11
Yeah. Once the transaction is completed, which will probably be about the time this podcast is released, it’ll be approximately $1.4 trillion in assets making us the sixth largest in the world.
Kirill Eremenko: 00:10:23
Wow. That is crazy. As a lead data scientist at this company, what is your role like? Do you actually look at how to invest this money, or are you looking for fraud, or is it like a broad scope of locations? What exactly do you do?
Scott Clendaniel: 00:10:41
Most asset management firms have separate groups who do the actual portfolios of investments, so I’m not involved in that so much. I help working on other types of business problems like optimizing sales, trying to help in profiling customers and opportunities, occasionally some small pieces of fraud detection, and actually trying to educate the organization as a whole on best practices and analytics and trying to make sure that we can meet the academic component of what’s going out in the world versus the real world, and trying to bring people up to speed, do a lot of training for folks, helping them form their own business plans and helping them build their models. It’s our reaching.
Kirill Eremenko: 00:11:28
Gotcha, so quite a broad scope of applications, but not specifically the investment management. Interesting. Very interesting.
Kirill Eremenko: 00:11:40
I hope you’re enjoying the podcast, and we’ll get straight back to it after this quick announcement. This announcement is going to be a bit tough for me because it is about my own book, so please excuse the shameless plug. However, I do believe in it so much that I want to get the word out there. This book is designed in a way to get anybody and everybody up to speed with data science, pretty much everything that is important that is needed to get going.
Kirill Eremenko: 00:12:05
The unique proposition of this book is that it doesn’t require coding. There’s a lot of books out there on data science where you need to sit in front of the computer and code in Python or R. This book, you simply take and you can read it on your lap, in a car, in a plane, in your backyard, on a couch. You can read anywhere. There is no coding in the book. It focuses on intuition. If you’ve taken our course, then you’ll like those intuition tutorials about how an algorithm works and why rather than what the code behind it is, then you’re going to love this book.
Kirill Eremenko: 00:12:41
It’s going to be a great way to solidify that knowledge. It covers pretty much everything in a data science project lifecycle from asking the right questions to data preparation, to machine learning, to visualization and finally presentation. Pretty much everything you need in your career is covered. If that sounds exciting, check it out. It’s called Confident Data Skills available on Amazon. It’s the data science book with the purple cover, and please enjoy.
Kirill Eremenko: 00:13:11
Well, on that note, I think let’s dive into these questions because there’s quite a lot to go through that, and I think that’ll take us-
Scott Clendaniel: 00:13:23
Let’s go.
Kirill Eremenko: 00:13:23
All right, awesome. I’ve gone through the questions. This was on LinkedIn. I’ve sorted them in order of… We’re going to start with the most advanced machine learning, AI, deep learning stuff first.
Scott Clendaniel: 00:13:34
Fine. No pressure on me. That’ll be great. Okay, go ahead.
Kirill Eremenko: 00:13:38
Here we go. Vighnesh, if I’m pronouncing that correctly, Vighnesh asks, “Can you give us a brief about the real world applications of data science in the investment industry? How do you approach a particular problem in this space?”
Scott Clendaniel: 00:13:58
Sure. I think one of the big components in any field is actually trying to make sure what is the business problem and defining that first before you actually define your modeling approach. In investment management, there are a bunch of different problems where data science can be applied, and also, financial services have been involved in advanced analytics since at least the 1960s, so it’s a great field to be in. A couple of examples would be fraud detection. How do you tell whether a particular transaction is someone spoofing, whether they’re the real person or not?
Scott Clendaniel: 00:14:31
There’s developing portfolios in which assets should be in there. You can look at time series forecasting of how an individual investment is going to perform. You can also look at things. One of the problems I’ve been working on a lot recently is sales optimization, so how does a financial advisor look at the broad palette of customers and potential customers, and figure out who should be prioritized in terms of what their needs are and coming up with a product recommendation on what would fit their needs?
Kirill Eremenko: 00:15:05
Gotcha. It ties in well with what we just spoke about that there is a broad scope of applications in business problem. Gotcha. What specific AI technologies have… This is Matthew. Matthew asks, “What specific AI technologies have changed the investment industry, and which do you predict will shape the industry in the next five years?”
Scott Clendaniel: 00:15:28
Sure. I think the development of additional algorithms to be available to us has changed AI quite a bit. Deep learning, perhaps not as much as others, but things like xgboost and algorithms that allow for ensembling have really helped the industry quite a lot. Also, approaches in terms of anomaly detection for fraud detection, they’ve been huge contributors as well. Those are probably the changes in AI that have impacted the most.
Scott Clendaniel: 00:16:00
Also, the growth of open source have made it very difficult for organizations to say no. There was a time many, many decades ago where people would say, “Oh, no, I’m sorry. We can’t do anything without a software package that costs $90,000.” Now, they can’t say that anymore. I think that actually had probably the biggest impact on the growth of AI overall.
Kirill Eremenko: 00:16:23
Gotcha. Gotcha. In the next five years?
Scott Clendaniel: 00:16:27
Next five years, I think there are going to be huge opportunities in terms of predicting credit performance and also fraud detection. Those are extremely difficult problems to solve, and having the more advanced AI technologies, I think, are going to continue to help in that arena, especially fraud detection, because it keeps changing. What appears to be fraud in one given quarter may look very different the next quarter, because the fraudsters are always adapting and changing their approaches.
Scott Clendaniel: 00:16:58
So you need to have a technology that allows the models to continue to grow over time. You can’t just pick a point in time and say, “Okay, we know what fraud is,” because it won’t be the same next quarter.
Kirill Eremenko: 00:17:09
Gotcha. Before I forget, I wanted to say that Scott is sharing his comments today on behalf of himself and not on behalf of the organization that he works at. These are just opinions at the end of the day, our opinions.
Scott Clendaniel: 00:17:27
It’s all my fault. I want to make myself really clear. Everything I say is my fault. No one else’s.
Kirill Eremenko: 00:17:32
Thank you. Thank you, Scott. Speaking of fraud, do you know what the size of this problem, globally or in the U.S. is per annum?
Scott Clendaniel: 00:17:47
I don’t have recent statistics on that, but it runs into the several billions of dollars. The challenge is the fact that you not only lose the profit on the given transaction that would come in if it turns out to be fraudulent, but you lose the entire dollar amount. In financial services, your inventory is actually the dollars that you manage. If you have a fraudulent transaction, you lose every bit of that inventory along with any type of profit you would have made.
Scott Clendaniel: 00:18:18
It runs into many, many billions of dollars, so it is a huge issue. It’s also really complicated because the fraud rate, the percentage of transactions that are fraudulent tend to be very low, but its financial impact is ridiculously large. It’s a real class and balance problem.
Kirill Eremenko: 00:18:36
I did a quick search. The global fraud market size for fraud detection and prevention market size is valued at $17.3 billion. As you said, lots and lots [crosstalk 00:18:51].
Scott Clendaniel: 00:18:51
That’s just the market to stop the fraud. That’s not the product sales. You’ve got a real clear picture of how big of an issue it is.
Kirill Eremenko: 00:18:58
That’s a good point. I only know one… saw quite… Intuitively, I know what… I’ve read about one specific broad algorithm that I could confidently explain, and that’s Benford’s law. Are there any algorithms that you can share with us?
Scott Clendaniel: 00:19:23
Sure. Actually, I’ll make a recommendation of something to be careful with. There has been a huge amount of press about using anomaly detection for fraud detection. That is very helpful. It does have some pretty severe limitations though, and that, a given fraudster is going to try very hard to not look like an anomaly. In other words, to some extent, the data is actually fighting against you. The fraudster is trying to look as similar as possible as they can to the mean of any given transaction.
Scott Clendaniel: 00:19:57
They’re actively fighting you to not look like an anomaly. The problem is the false positive rate on anomaly detection is enormous, and it’s very difficult to fight against. Just using anomaly detection, regardless of how sophisticated the version that you’re using is, tends to have some severe limitations that you’re going to come up against, so just be aware there should be one tool in your arsenal. It shouldn’t be the be all and end all.
Kirill Eremenko: 00:20:28
Gotcha. Anomaly detection is one. That’s fantastic. I’ll share Benford’s law. Benford’s law is more of a aggregates tool. It doesn’t look at individual transaction. It looks like as a whole. The way we were taught it at Deloitte is that if you take a… I might be paraphrasing what they told us back then. Again, I’m speaking from my opinion as well. You take all the transactions on all of the dollar values on a balance sheet of a company.
Kirill Eremenko: 00:20:59
Just take all of them. Mix them all up. Put them into a bag, and then look at the distribution of the first… Is it the first? No, of the first digit in all of these transactions. What is the first digit of all of these amounts? What’s the first digit? How many number ones are there? How many number twos are there? How many number threes are there? The leading digit, and Benford’s law says that the distribution should be… They should be 30% once, 17% twos, 12% threes, 9% four.
Kirill Eremenko: 00:21:40
It drops off as you go further. It’s an intuitive thing, and that is something that is really hard to fake, right? If you’re faking a balance sheet and you’re making up numbers. You don’t think of that distribution in mind. You might make your numbers look really believable, but overall, when you take the distribution on the first digit across all the numbers that are going to fall on Benford’s law. That’s how a qualified expert can tell that, “Hey, there’s something going on here.”
Scott Clendaniel: 00:22:10
In forensic accounting, that becomes really important. That is definitely one of the warning signs. There are a couple of interesting things about Benford’s law in addition to what you said. I think you gave a great explanation of it. Benford’s law seems to apply even if you change the base unit. If it’s not a decimal system, if you used a base eight system, you will tend to see very similar patterns. The percentage of digits will change, but I just find it amazing that even if you change the base number that you’re using, that it tends to show up.
Scott Clendaniel: 00:22:45
It’s great for things like reviewing the balance sheet, just as you mentioned. What becomes tricky is the fact that when you’re dealing with consumer transactions, for example, the actual transactions, you don’t apply as much for the trailing digits. Everybody wants to charge $1.99 or $1.95. They don’t tend to charge $1.03. You actually have the opposite problem there, so you have to be careful. Benford’s law is extremely powerful when you’re looking at a whole collection of numbers given in one instance such as the balance sheet. It becomes much harder when you’re trying to look at individual transactions.
Kirill Eremenko: 00:23:24
Awesome. Awesome. Thank you. Well, there we go. That’s two techniques in fraud detection. Speaking of fraud detection, we have a question from, again, Vighnesh who says, “While working on fraud detection problems, most of the times, we come across imbalanced datasets. Can you please put a light on how to overcome such problems or how to resolve it?” Maybe to start off, what does it mean that the data set is imbalanced?
Scott Clendaniel: 00:23:52
Sure. This is typically referred to as class imbalanced. In other words, if I’m trying to do a classification problem, and let’s say that I’m trying to do fraud versus non-fraud. If you look at the distribution of how many transactions are fraud and how many transactions are non-fraud, your fraud rate tends to be relatively small down into the tens or hundreds of a percent. The problem is if you try and compare the fraud transactions versus the non-fraud transactions, a lot of algorithms are going to choke on that unless you adjust the balance of the dataset so that you can have more of a 50/50 ratio between fraud and non-fraud.
Scott Clendaniel: 00:24:33
Otherwise, what the model’s going to do is go, “Let’s see. I’ve got 1,000 transactions. 990 are not fraud, and 10 are fraud. I’ve got it. They’re all not fraud.” It’s going to be right 99.9% of the time. It’s just going to mess everything up, so you need to make [crosstalk 00:24:52].
Kirill Eremenko: 00:24:51
It’s going to have fantastic markers.
Scott Clendaniel: 00:24:54
Yes. That’s amazing. I’m done. I just said there is no fraud, and I’m going to go home and have lunch. That doesn’t work out too well in the real world. What you do is you tend to over-sample what’s called the minority class, so in this case, the fraud transactions. I might take most of every fraud transaction I can get that is in my training set. I’m going to compare it against an approximately equal number of non-fraud. That means that the model can’t just arbitrarily say, “Okay, everything is not fraud.”
Scott Clendaniel: 00:25:27
That’s the technique that I use the most. There are other techniques that you can use including doing all sorts of complicated things with synthetic data or smoke techniques or those types of things, but the over sampling the minority class, lots and lots of the fraud and much fewer of the non-fraud has been the technique that worked the best for me. It’s also very simple.
Kirill Eremenko: 00:25:50
What’s the drawback? What’s the potential danger of using this technique?
Scott Clendaniel: 00:25:57
Part of the problem is you may not have enough fraud cases to use in the first place. You may have such a small number of records that you may not be able to use that in its purest form, but you’re definitely going to want to move your sampling as close to a 50/50 ratio as you can.
Kirill Eremenko: 00:26:17
Gotcha. As long as you select the other ones, the non-fraud ones at random, you shouldn’t have any bias in your model because you didn’t use all of the available random.
Scott Clendaniel: 00:26:29
Correct. Use of random becomes really important. Some of the experts in the group who have classic statistical background can talk a lot more about random versus non-random and that there is no true random, but there are all sorts of techniques you can use to make sure. For most of the work that I’ve done, I just use random functions, Python or SQL. That has worked pretty well for me, and I haven’t run up [inaudible 00:26:55] stations.
Kirill Eremenko: 00:26:57
Awesome. Speaking of Python and SQL, what kind of tools do you use in your day to day?
Scott Clendaniel: 00:27:06
Python, Spark are the most common. Because I am older than dirt, I started doing this all the way back in 1986, God forbid. I actually grew up using gooey tools like SPSS, which is now by IBM, Salford systems, which is now owned by Minitab, lots of other gooey tools. There’s actually a free one, especially if you’re just starting in the field and you don’t have developer background, called Orange Data Mining, which is included in the Anaconda distribution. Those were a couple places that you can start, but eventually, it tends to turn into a lot more Python as a starting point, and probably Spark if I’m using a distributed system to build.
Kirill Eremenko: 00:27:55
Gotcha. You said Orange Data Mining.
Scott Clendaniel: 00:27:58
Orange Data Mining, yeah. It is not the prettiest program you will ever encounter. The interface looks at least about 20 years out of date. Don’t be fooled by how the interface looks, because there is a lot of power underneath it. A lot of people get turned off. They’re like, “Oh, this doesn’t look cool. This doesn’t look like something that was built for Apple. I’m not going to use it.” I would say that’s a mistake. They’ve actually done a really good job of creating a gooey interface to sit on top, and then underneath, they’re primarily as Python. [crosstalk 00:28:29].
Kirill Eremenko: 00:28:29
Is this primarily for building models or fraud detection?
Scott Clendaniel: 00:28:36
Any type of model.
Kirill Eremenko: 00:28:38
Gotcha. Awesome. Awesome.
Scott Clendaniel: 00:28:40
I’ve been really happy with it. People laugh at me when I show the screenshots, but it actually works pretty well.
Kirill Eremenko: 00:28:46
Gotcha. What kind of models have you noticed work the best with fraud detection? We’ve got a huge range. K means clustering. We’ve got Naïve Bayes. We’ve got logistic regression. We’ve got xgboost, and so on. What would you say are your go-to models? When you have a fraud-detection problem, what’s your first, second, third choice?
Scott Clendaniel: 00:29:12
First of all, I’m going to throw out an old theory that I hope people look into, which is essentially called the multiplicity of good models, which means if you have the right data and you’ve prepared it the right way, all sorts of algorithms are likely to give you a very similar, positive result. If you haven’t done the data prep correctly, you will start to see wide variances. That being said, ensembling techniques of any type would be my favorite, probably the most common being XGBoost.
Scott Clendaniel: 00:29:44
Also, as a data prep technique, I recommend target meaning coding. That can be extremely helpful. In terms of the final technique, I always recommend ensembling because each algorithm has its own strengths and weaknesses. If you’ll ensemble a group of different models together, you’re likely to end up with a better result. The final model is usually logistic regression based on the inputs of the XGBoost and other types of techniques in the family.
Kirill Eremenko: 00:30:15
Wow. Thank you, very, very detailed and advanced. This data, perhaps, I think, you mentioned, target meaning code.
Scott Clendaniel: 00:30:21
I got really passionate about this stuff, so I may bury you in detail, and I apologize.
Kirill Eremenko: 00:30:24
That’s okay. That’s okay. That’s okay. I want this to be an advanced discussion, well, advanced learning for me.
Scott Clendaniel: 00:30:33
That’s fine. That’s fine.
Kirill Eremenko: 00:30:34
I wanted to ask you this data prep technique, target, mean encoding. I don’t know it. I haven’t heard of it before. Could you tell us a bit about it, if you can just explain what does it do and how [crosstalk 00:30:44]?
Scott Clendaniel: 00:30:44
Sure. Let me give you a really simple example. Let’s say that we are an auto insurance company, and we want to predict whether the old myth that red cars always cause more problems than others. I’ve got a red car, a blue car, a yellow car, a white car and a black car. I’ve got five different car colors. I want to encode into my model this categorical variable, so rather than use the original values of the colors in the variable, you use a transformed variable, so a new variable. Instead of recording the car color there, you actually put the claim rate for each color.
Scott Clendaniel: 00:31:29
If a blue car has a 0.1% claim rate, you put 0.1 at any time it’s blue. If it’s really red, you’ll use 0.2% any time red shows up. In other words, you convert the original categorical input into its actual target mean. What’s the mean rate that this issue is going to come up with? You use the transformed variable as opposed to the original variable. It tends to work better than one hot encoding, which you would usually use. Also, you can use it in pretty much any algorithm, including algorithms that only take numerical inputs like the original XGBoost.
Kirill Eremenko: 00:32:10
Wow. Wow. That’s awesome. I hadn’t heard of that one. I’ve heard of one hot encoding, but still, do you mind refreshing my memory on that, please?
Scott Clendaniel: 00:32:18
Sure. One hot encoding, so for each of our five car colors, we’re going to take the original variable, and we’re going to move it out of the dataset. We’re going to create five new variables. One variable is going to be, “Was the car color white, zero, one?” If it’s white, you can put one. If it wasn’t white, put zero. When you have a second column that says, “Was the car color blue?” The third variable says, “Was the car color red?”
Scott Clendaniel: 00:32:42
You can run into issues with that, and that you can end up with a sea of categories, and you can overload your model with way too many inputs.
Kirill Eremenko: 00:32:55
Gotcha. Gotcha. You also got to be careful of the dummy variable trap there, right?
Scott Clendaniel: 00:33:00
Absolutely.
Kirill Eremenko: 00:33:00
You have to have one less than the original number of categories. Gotcha. Very interesting. Thank you. That’s very exciting. Let’s move on. Now, let’s talk a bit about models. Sonam, Sonam, I hope I’m pronouncing it right, asks, “What are the parameters to look out for and or test to perform that indicate model drift while monitoring a production AI or machine learning model?” To start off, what is model drift?
Scott Clendaniel: 00:33:33
Model drift is a silly name, but basically, it means that the performance is going to drift over time. All models tend to decay over time, and different models decay at different rates. There are a couple of things that I’d suggest with this. Number one, assume that your model is going to decay over time. Don’t guess it’s going to decay. Just assume it’s going to decay, and set up a schedule of when you’re going to retrain any model you have.
Scott Clendaniel: 00:33:59
That’s the first component. Plan for obsolescence before you put the first model in. That’s very important. The second thing is if I use what’s called a population stability report, which sounds like some bizarre sociology experiment, but essentially, what it’s doing is to say, “My data that I started off with in one period, how similar does it look to data that I’m looking at right now? Are those two populations similar or not?”
Scott Clendaniel: 00:34:27
When the population stability report comes in and says, “Hey, wait a minute, your data starts to look different,” you definitely need to retrain your model, and that the world that the model is trying to represent has changed. Therefore, the accuracy of its predictions has changed. Just assume it’s going to happen. I get irritated with folks who just want to go out there and say, “I have created the world’s greatest model, but it’s based on data from seven years ago. And I don’t know why it’s performing poorly.”
Scott Clendaniel: 00:34:54
Come on, seriously? If you put in your model, test it constantly, and use that population stability report to keep your eye open to see that the world’s changed.
Kirill Eremenko: 00:35:04
Gotcha. A couple of questions here. First one will be, “Why…” This might be a naive question, but I’m just tempted to ask, “Why do models decay over time? Why do they never get better? Why is it that way?”
Scott Clendaniel: 00:35:21
Trick question. They decay over time because they represent the world as they knew it, and the world changes. The model does its absolute best to represent the world as it saw it in your training set. When the world changes, the type of data that shows up in your training set may drift. Let’s pick an easy example. Inflation, prices tend to rise over time for lots and lots of things. If your original model was based on prices from three years ago, the prices now look different, so the model needs to adapt to reflect that change to come up with representation of what happens today.
Scott Clendaniel: 00:36:02
That’s why all models tend to drift. However, that isn’t necessarily a bad thing in that you may have learned more information over time. You may have a larger data set to look at. You may have found new variables that you didn’t see before. You may have new algorithms you want to test out. Actually in the end, you can end up with a model that is more accurate than the last model was at its peak. Over time you actually can get better. That’s one of the things that’s exciting to me.
Kirill Eremenko: 00:36:32
Once you update it, of course, right? If you leave it alone, it’s not going to get better.
Scott Clendaniel: 00:36:37
No. I wish.
Kirill Eremenko: 00:36:39
Gotcha. In your answer on LinkedIn, so it was really inspiring to see that you went through and answer to everybody, you do a huge part for data science.
Scott Clendaniel: 00:36:52
Tried. If I missed anybody, send me a message.
Kirill Eremenko: 00:36:55
Awesome. You mentioned that six months is your magic number to look at. Why is that?
Scott Clendaniel: 00:37:05
It is completely arbitrary. I’ll tell you why. If you try and make it annual, it tends not to happen. In the real world, organizations are like, “I don’t have it.” We did it last year or whatever. If you make it six months, you keep it top of mind with everybody. Six months is the outer limit, and then if the population stability report says, “Wait a minute, the world looks different, or the performance of the model tends to fall up.” You’ve got a fraud model, and your fraud rate keeps inching up.
Scott Clendaniel: 00:37:36
Either those two events, you look at it in a shorter period of time. But if you actually build that into the calendar on the front end, and also explain to your stakeholders, “Model drift is a thing, and you need…” Modeling is a process. It’s a process of learning, and it’s adapting to change as information changes. Just bake that into the schedule from the beginning, and you’re going to be in a lot better shape.
Kirill Eremenko: 00:38:00
Been there, it was in an organization once where they didn’t look at the model. Some consultants delivered a segmentation model. They did… oh, no, prediction model, prediction in terms of who will churn, who won’t churn. It didn’t look at it for 18 months. When we had to look at it…
Scott Clendaniel: 00:38:21
Ouch.
Kirill Eremenko: 00:38:22
Its accuracy dropped from 78% or something like that down to 49, so it was better to flip a coin.
Scott Clendaniel: 00:38:32
Well, let me throw out a made-up word for everyone. That’s nonstop optimization. Instead of optimization, if you’re constantly nonstop trying to optimize, you call it nonstop optimization. Senior management loves phrases. I’m kidding. But if you take on that theory that I am always going to be improving my model, I’m always going that it’s an organic process, that it’s something that you put in as a regular business process as opposed to the snapshot one time event that’s going to fix all our problems. I think you’ll be in better shape.
Kirill Eremenko: 00:39:06
Great. The population stability report, what do you look at there? Do you look at means, distributions? What are the [crosstalk 00:39:13] or maybe-
Scott Clendaniel: 00:39:14
It’ll actually give you an indicator on a scale of zero to one and on how much things have changed.
Kirill Eremenko: 00:39:21
It’s like it’s a library in Python?
Scott Clendaniel: 00:39:24
Yeah. For example, if incomes on the original data set were $62,000, and now it’s $140,000, something’s wrong. It also helps you to figure out if your data stream has been corrupted. In other words, let’s say the model works fine when it has accurate data, but somehow, something has happened, and now the data isn’t as good as it was before, you can then go in and say, “Okay, wait a minute. Something’s wrong. The population looks different from what it was before.”
Scott Clendaniel: 00:39:54
Maybe we’ve got a problem with the database. Maybe we’ve got a problem in how the data was collected. Maybe we’ve got a metric versus English units issue that suddenly popped up.
Kirill Eremenko: 00:40:06
Gotcha. Anybody can go and download this population stability report, or everybody needs to build their own version of it?
Scott Clendaniel: 00:40:14
Yes. The formulas are out there. I can’t recite it off the top of my head, but if you just type into Google population stability report, it can give you a walkthrough of that.
Kirill Eremenko: 00:40:22
Awesome. Fantastic. Speaking of data collection, there was a question from Santosh. How do we check stability and consistency in the process before using the data it generated from model building? I understood there was a bit of he meant one thing, and she first answered another question, and then he answered the second one. Let’s start with the first one. The first one was like, “How do we check for stability and consistency in terms of the data collection process?”
Scott Clendaniel: 00:40:52
Sure. Usually, this is done sort of further upstream before when we get data. This is usually done by the folks who are doing your data ingestion or your ETL process on the data upfront. The point he’s making is extremely valuable. It’s as simple as garbage in garbage out. If the data you’re putting into your model has flaws in it, your model isn’t going to work. One of the things is actually sit down with the people who are the stewards of that data.
Scott Clendaniel: 00:41:20
How is this collected? How often? How long have we been keeping track of this? This is also one of the reasons why data visualization is so important. See if the data makes any sense before you start loading it into your model. His point, I think, was data scientists are so excited. They want to have a model. They want to have results. They want to use their area of expertise. They want to pull the algorithm out of the quiver and start shooting.
Scott Clendaniel: 00:41:47
That is a problem if you haven’t checked the data consistency upfront. I will give you a real world example from a client who shall not be mentioned, where the original data stream, there was some type of corruption when data was migrated from one database to the next database. If it was a dollar amount, they literally physically typed the dollar sign in the value. If there was a comma, they type the comma. If there was a decimal, they type the decimal.
Scott Clendaniel: 00:42:16
At other points in time, this wasn’t true, so it was just the raw number. This is why you really need to understand your data. To his point, you need to make sure that data seems to make some type of sense. Sit down with the steward of that data to make sure that you understand what you’re dealing with before you get too far down the pipe.
Kirill Eremenko: 00:42:34
Awesome. Gotcha. Then you also talked about your answers, something that intrigued me. You said that if for some reason, the data is corrupt, then cross validation of the model should also fail. Basically, we could probably use cross-validation as I understood as a indication, whether there’s problems [crosstalk 00:42:58].
Scott Clendaniel: 00:42:57
Absolutely.
Kirill Eremenko: 00:42:59
Tell us a bit more about that.
Scott Clendaniel: 00:43:01
The great thing about models and cross validation is it’s virtually impossible to come up with a great cross validated model based on bad data, because it just won’t work.
Kirill Eremenko: 00:43:14
What is cross validation?
Scott Clendaniel: 00:43:15
When you’re taking cross validation, you’re taking the original data set, and you’re dividing it into what they call folds. Let’s say we’re going to take your dataset. We’re going to break it into five folds. You test the first model built on four of the folds, and you leave out the fifth. On the second one, you may use folds two through five, and you test it on the first. You want to make sure that those results look very similar.
Scott Clendaniel: 00:43:38
If they don’t, you’ve got a problem in the model, so you either average the results of the folds, or what I tend to do is definitely do that first part, but also go back and say, “What is weird about that one fold that doesn’t seem to be working very well? Why is the performance off for this version versus that version?”
Kirill Eremenko: 00:43:56
Gotcha. Is the data in the folds randomized? Before you select defaults, you randomize them.
Scott Clendaniel: 00:44:03
Absolutely.
Kirill Eremenko: 00:44:04
Gotcha. Just to recap, let’s say we have, I don’t know, 100,000 or 500,000 records. Then you break it down to five groups of 100,000 each. In the first version, you train the model on the first four groups of the data, and then the fifth one, you test it. In the second version of the model, you train it on, say, the first, second, third, and fifth groups. Then you test it on the fourth. Then you train it on the first, second, fourth, and fifth, and you test it on the third.
Kirill Eremenko: 00:44:38
You’re always shifting this window. Ideally, you should be getting the same results throughout, similar results.
Scott Clendaniel: 00:44:48
They should be really similar. Also, your final model should take the average of each prediction.
Kirill Eremenko: 00:44:54
Gotcha.
Scott Clendaniel: 00:44:56
There are some folks who just use cross validation for testing hyper parameters and things like that. If production allows me to be able to use all five models and take the average of the scores, that’s what I like to do.
Kirill Eremenko: 00:45:15
If you have corrupted data or probably errors in the data, if you randomize the data before you do the five bands, wouldn’t that corrupt records? Wouldn’t they distribute equally across the five bands, and therefore the model would still perform identically, but still there would be-
Scott Clendaniel: 00:45:37
Well, it would be identically terrible performance. You’re not just looking to see if it differs among the five. If my area under the curve is 51%, I’ve got a problem. I need to be really aware of that. To your point, yes, they will look similar, and they will look similarly awful.
Kirill Eremenko: 00:45:59
Gotcha. Gotcha. Could you do the cross validation without randomizing first? Then you would more likely have one of those bands that is definitely underperforming.
Scott Clendaniel: 00:46:12
It’s no. I would not recommend that because you lose… The real advantage of the randomization is the fact that it eliminates or greatly reduces the chance that you’re just looking that a few records are off, or that you’ve got outliers or that type.
Kirill Eremenko: 00:46:29
Gotcha. Gotcha. Awesome. Thank you. Next one was… This is an interesting question. This one is more about complexity of machine learning models. What is your experience… This is a question from Desmond. What is your experience with the complexity of machine learning models and alpha or out performance to the most complex models XGBoost, neural nets yield the highest alpha, or are there other factors that yield higher alpha? For example, type of data feature, engineering, et cetera. First of all, what is alpha?
Scott Clendaniel: 00:47:14
Well, I’ll tell you what. I am going to do what all good interviewees do. When they don’t have the pure understanding of the answer, they change the question. I’m going to treat this as a question on over fitting as opposed to the pure definition of alpha. I’m going to regard over performance as a function of over fitting the model, which means that it’s basically memorizing the data. As I said, I am older than dirt. When I went to school and studied math, they used to do this weird thing where they would give you the answers to the odd numbered questions in the back of the book, but they wouldn’t tell you the answers for the even number of questions.
Scott Clendaniel: 00:47:54
What would happen is if you just tried to memorize the answers, you’d only be 50% right. Models tend to be very greedy. If they get the chance to memorize the answers, they will do it. What you want to do is to make sure that it’s very difficult for the model to actually do that. Otherwise, what it’s learned is what the answers to the specific records that you gave it is as opposed to identifying the true patterns that can be applied to other folks. If I went and I got a suit, and it was completely custom fit but for somebody else, it’s not going to fit me very well.
Scott Clendaniel: 00:48:32
It’s over fitted to the wrong person. I want to make sure that my model generalizes well. I want to make sure that my model applies to all kinds of different people. Over fitting is a big problem. The more complex the model, such as deep learning, if I’ve got a 600-layer deep learning neural network, and I’ve got 10,000 records, I’ve got a problem, because in many cases, it’s going to try and memorize the data itself as opposed to learn the patterns. Algorithm can be a component.
Scott Clendaniel: 00:49:03
You can also have data leakage issues, where it’s memorizing the answer because it’s actually included in the original data set. The ways to avoid that or an answer to the question, algorithms can be a problem. The data can be a problem. There are all kinds of things that can lead you down that path, ways to get around that or to use very robust validation methods including cross validation to try and eliminate the possibility of models actually memorize the answers in the back of the book.
Kirill Eremenko: 00:49:35
Wow. Wow. That’s fantastic. I just got transfixed on your answer. [crosstalk 00:49:43].
Scott Clendaniel: 00:49:43
Oh, sorry.
Kirill Eremenko: 00:49:44
That’s awesome. Thank you. Definitely an important point to look at. At what stage would you say people should keep that in mind when they’re building a model?
Scott Clendaniel: 00:50:01
From the beginning. Part of what we as data scientists, we tend to be really tempted to jump in and build them up. I’m a model builder, so I want to build a model. That might not be the first thing that you do. You really have to have a solid design in terms of what your model building process is going to be. Coming up with the validation strategy should be one of the first things you do because you’ve got to segment your data out into what’s going to be test, what’s going to be training, what’s going to be validation or cross validation before you start the modeling process, or you’ve already contaminated the experiment, so to speak.
Scott Clendaniel: 00:50:39
In your process design, before you sit down, that should definitely be a component that you look at.
Kirill Eremenko: 00:50:47
Gotcha. How much time should be spent on designing the process versus implementing the model?
Scott Clendaniel: 00:50:54
Well, if you come up with a standardized process in terms of how you select variables and your randomization, you can actually bake it into the process. Once you do it once, you shouldn’t have to repeat it a whole bunch of times, but you should definitely be very robust on your first model and see what components you can redo. Randomization should be something you should be able to do in every model in a specific script. You shouldn’t have to reinvent the wheel every time.
Kirill Eremenko: 00:51:21
Gotcha. The follow-up question from Santosh was he said, “Give an example.” Let’s say I want to build a forecasting model for daily pizza sales, and say I have data for the past model.
Scott Clendaniel: 00:51:36
After my own heart, pizza sales.
Kirill Eremenko: 00:51:40
My question is unless the processes that drive for pizza sales are consistent, we can’t rely on the data. For example, there was a change in employees every month, the change in tools being used and so on. On the other hand, if the pizza store is started just a few weeks ago, it might not have reached this stage because… The main thing is how do we know? If there were changes in the process, you have data for a whole year, but then there are changes in the process, employees or how we do sales and so on. Can we still use that for modeling?
Scott Clendaniel: 00:52:15
Sure. What I’ll say to that is you’re absolutely right, very important point, but it is very rare that a modeler is ever going to have a perfect data set to start with. The trick is, “What is my decision making process now, and can I improve it with the model, and if so, by how much?” You never get to perfect. You try and get to a perfect representation of the system for your pizza prediction, but every organization has employee turnover. Every organization has some of those elements in play.
Scott Clendaniel: 00:52:50
That’s why you also have to be very careful with your validation strategy to make sure that your model holds up on data it’s never seen before. That’s why keep banging that drum. The question is, “Is there something that I can learn from the data that I have right now using standard validation procedures?” If I can, and I increase the performance of my decisions, if I make 10% better decisions, does that help my business? If it does, the Teddy Roosevelt quote about, “Do what you can with what you have where you are right now.”
Scott Clendaniel: 00:53:26
Make sure that your client, whoever, is going to be using your model understands. This is what I think I know. This is what I don’t think I know. This is how much I think I can improve performance with the model over where you are right now. Then you work with a client to say, “Is it worth the effort? Is the juice worth the squeeze, so to speak?” That’s why you need to work with your client as opposed to just turning this into some academic exercise off in an ivory tower.
Kirill Eremenko: 00:53:52
Nice. Fantastic. Scott, thank you. Those were all the advanced questions. We’re moving on to the world-
Scott Clendaniel: 00:54:02
Okay.
Kirill Eremenko: 00:54:03
You did well. You did really well. We’re moving on to more soft skills and predictions and forecasts for the future. A question from Muhammad, “What is the difference between a data scientist and leading a data science division?” Basically, what is the difference in skills required to be a data scientist and to lead a data science division?
Scott Clendaniel: 00:54:33
To lead a data science division, I think you need the skills of a data scientist plus a couple of other things. One strategy, making sure that you’re leading the appropriate goals across all modeling projects, not on your individual modeling projects and management. You need to be able to work with people to coach them to get the absolute best performance that they can achieve, not just what you can do best, and a lot on what you’re working on right now.
Kirill Eremenko: 00:55:01
Are the people skills required for a data scientist such as communication, presentation? Are there still people skills? Are they different, and how are they different to the management people skills required for a lead data scientist?
Scott Clendaniel: 00:55:15
I don’t think that they are different. I think the problem is people skip over them altogether. The quality of management in general in most organizations can be somewhat appalling, not just for people who manage data scientists, but for people who manage any type of group. I think it’s a real chink in the armor for all types of organizations. I think the difference would be that you may communicate to data scientists in their own language.
Scott Clendaniel: 00:55:43
You must be able to speak their language and be able to establish their trust and be able to work with them to get them to their highest performance. If you were dealing with an accounting team, you need to be able to speak the accounting language to be able to help them reach their highest performance. It’s not so much different skills. It’s applying the right skills to the type of game you have, I think.
Kirill Eremenko: 00:56:07
Gotcha. I know you’re passionate about leading data science teams. Why are you passionate about that?
Scott Clendaniel: 00:56:15
Because I think so much can be done outside of just the algorithm, and I think there has been such a push, especially in the past 10 years, on the type of algorithm you use. Algorithm isn’t necessarily what’s going to lead you to the best performance. I’m going to steal a story from Stephen Covey. He said that pretend you had a bunch of folks, and they’re trying to lead a trail through the jungle. They’re like, “Okay, we’re going to have Fred in the front of the group because he’s really good at dealing with a machete, and we’re going to have Michelle. She is the absolute expert at machete sharpening, and she’s really good at that part, and such and so forth and everything else.”
Scott Clendaniel: 00:56:53
The leader is the one who climbs the ladder up and shouts down, “Wrong jungle.” You got to be able to change. You need to be able to figure out if you’re in the right jungle or not. A lot of managers are not terribly good at that, and so you need to have that holistic view in addition to the expertise of data scientists.
Kirill Eremenko: 00:57:17
Fantastic. What advice do you have for data scientists or advanced data scientists who wants to become leaders or who wants to become data science managers?
Scott Clendaniel: 00:57:32
It’s understanding a fact that all data scientists need to come to grips with, and that is that data science is not about data. It’s about people. Let me explain what I mean by that. You’re trying to solve people’s problems. You’re trying to help people. You’re trying to communicate with people. Whether you’re in accounting or data science or medicine or nuclear physics or social work or anything else, it’s about people. A lot of people come into our field unfortunately, because they don’t really like dealing with people. They like numbers more.
Scott Clendaniel: 00:58:04
It needs to be a blend. At the end of the day, you’re always trying to help people meet their needs. Data scientists use data and algorithms and techniques to be able to achieve that goal, but the goal isn’t different. The more you understand people communication, storytelling, visualization, the so called soft skills, that is going to be what greases the wheels to be able to get people to where they need to be, and to solve their problems in the best possible way.
Scott Clendaniel: 00:58:35
You can’t surrender on that and lead a data science team and be terribly effective.
Kirill Eremenko: 00:58:42
Love it, so not just data science leadership, but data science itself is about people. If you one day want to become a data science leader, then start now. As a data scientist, start honing in on your people skills.
Scott Clendaniel: 00:58:59
Absolutely.
Kirill Eremenko: 00:59:00
What’s the recommendation? How does somebody go about… There’s not many online courses on people skills. A lot on technical skills. Where do you learn the people skills?
Scott Clendaniel: 00:59:10
It depends on where you look. I’m actually going to push back on that one a little bit. We in the data science community like to read our data science blogs. If you focus there, that’s where you’re going to find those results. There are all kinds of resources. I’ll tell you my particular favorite is the work that was done by Stephen Covey, and also my book recommendation, I’m going to sneak in here while you’re not looking, which is the Seven Habits of Highly Effective People.
Scott Clendaniel: 00:59:36
It talks a lot about people skills, talks a lot about problem solving. That can be applied to data science or accounting or physics or sociology or anything else, and focusing on those types of skills. Also, data visualization classes, not just to show how to visualize data, but you choose a visualization based on your audience and what that audience’s needs are. Focus on that piece. What do we need to communicate?
Scott Clendaniel: 01:00:02
What do they need to be able to solve their problem, and how do I give them back? As opposed to, “Here’s my way manky cool analysis that I did, 700 pages long that no one’s ever going to read.” That’s not the solution to the problem.
Kirill Eremenko: 01:00:17
Gotcha. Absolutely. Let’s move on to future questions, so about the future. Snehal asks, “What will be the next after advanced AI?”
Scott Clendaniel: 01:00:34
If we’re not careful, we’re going to run into another AI winter. Let me tell you what I mean by that. If you look at Gartner’s Hype Cycle, where they talk about the stages of a technology, you tend to get overly hyped expectations, and then you end up falling off the cliff into what they call the trough of disillusionment. We can bicker over whether those are good names or not. But if you set people’s expectations too high, and then you don’t meet them, they don’t tend to say, “Scott Clendaniel’s particular model didn’t do very well.”
Scott Clendaniel: 01:01:07
What they tend to say is, “I knew that AI stuff was a bunch of hooey, and we never should have invested in it. I don’t want to do a model again. I don’t want anyone coming in here talking about statistics. I don’t want to hear about machine learning. I don’t want to hear… It’s all garbage, because Scott Clendaniel’s first model didn’t do well.” You need to really be aware of that. 85%, according to Gartner, of all models do not reach production.
Scott Clendaniel: 01:01:29
Think about that for a second, 85%. Our industry currently has a 15% success rate. I don’t know of too many fields who can survive that, so my big concern about AI is unless we get back to actually solving the organizational issues and fixing the problems as opposed to, “Gee, look at my AUC. Doesn’t it look great?” The future of AI is going to go into a dark period for a while.
Kirill Eremenko: 01:01:57
What about on the flip side? Jacques asks, “There’s a fear that AI could replace humans in their jobs. What would you tell a concerned human being about that?”
Scott Clendaniel: 01:02:11
I would look at a lot of the research that’s come out of RPA, robotic process automation. To the largest extent possible and the conferences I’ve been to is people’s jobs don’t get replaced. In other words, people don’t get replaced. They get different jobs. Meaning that if they’re working in the accounting department, they stop working with copying stuff from Excel from spreadsheet A to spreadsheet B, and get to work on, Do we need the spreadsheet in the first place?”
Scott Clendaniel: 01:02:37
That’s not a bad thing. There was a lot of concern that AI was going to replace all kinds of people, and I haven’t seen a lot of it happen yet. I don’t think that radiology is the first career I would jump into right at the moment, because a lot of that is being automated, but you may end up with different types of jobs that a radiologist might apply like, how you apply the results from lots of different analysis, from all kinds of different x-rays in terms of diagnosing a disease.
Scott Clendaniel: 01:03:05
I think some jobs always get replaced by technology. There aren’t a lot of buggy whip manufacturing jobs left anymore, but I don’t see AI replacing all kinds of people. The head of Stanford’s AI lab had a great quote that said, “We’re a lot closer to discovering a smart washing machine than we are of terminators taking over the world.” I think that’s true. I think we need to be careful of it, but I think people are perhaps overly concerned at this point in time that AI is going to replace everyone’s job.
Kirill Eremenko: 01:03:42
Gotcha. There was a report by the World Economic Forum in 2018 that predicted that I think by 2025 or 2022, I’m not sure exactly of the year, but AI will displace 75 million jobs worldwide, whereas, it will create 133 million jobs. That’s a coefficient of 1.7.
Scott Clendaniel: 01:04:07
I think that’s a much better example than the one I just gave.
Kirill Eremenko: 01:04:13
I think they’re both absolutely valid. What are your thoughts on AI replacing data scientists themselves, specifically auto ML and products like data robot?
Scott Clendaniel: 01:04:25
I need to be careful here. I need to choose my words correctly in terms of that.
Kirill Eremenko: 01:04:29
We can skip that question. That’s okay.
Scott Clendaniel: 01:04:32
I think that a lot of AI functions can be helped by automation. I think ensembling, I think picking the correct algorithm, I think hyper tuning parameters, I think a lot of those will become automated, but there’s a lot of room for creativity on the feature engineering side. There’s a lot of room for creativity that’s hard to replace. Even something as simple as ratios, models are terrible at calculating ratios. They just are.
Scott Clendaniel: 01:05:02
For example, if you think of a credit score, if you think of everything I currently owe as one input, not a great predictor. If you think of how much total credit I have available, if you add up the credit lines from all my credit cards and stuff, also, not a very good predictor by itself. If you use a classic algorithm to go and say, “Okay, let’s throw them both out, because they don’t have high correlation to my result.” The trick is the percent of utilization, so out of that big pile of credit, what percentage am I using is hugely predictive in terms of your credit score.
Scott Clendaniel: 01:05:35
It’s those types of things as simple as ratios that I think are going to be hard to automate away. I think that many functions may be assisted by automation even in data science. I think if we focus on the right skills and the problem solving aspects and those type things, it is less likely to be automated away at least in the short term.
Kirill Eremenko: 01:06:00
Gotcha. Thank you, very, very cool answer. Adly asks, “Does every business need to adopt AI?”
Scott Clendaniel: 01:06:11
No is the short answer. [inaudible 01:06:16] not every business does. I think it’s silly for us to attend that every business in the world needs AI. I think every business could use to make better decisions than they do right now, and to the extent that AI helps with that, great. To the extent that AI doesn’t help with that, no. Also, there are some businesses that don’t have a lot of good data. If you don’t have good data, you can’t really build great models, so AI isn’t going to be a particular help.
Scott Clendaniel: 01:06:45
I think that every business needs to make better decisions, and businesses that have access to good AI should take advantage of it. Those that don’t, don’t worry about it.
Kirill Eremenko: 01:06:57
What are your thoughts on Andrew Ng’s comments that AI is the new electricity, and similar to how 100 years ago, only 50% of the U.S. was electrified? Now, everything uses electricity. AI will also similarly but faster be adopted by virtually all organizations. Otherwise, they’ll be lose in terms of competitive pressure. What are your thoughts on that, with the thought in mind that not every business needs AI?
Scott Clendaniel: 01:07:25
Well, let’s follow that example through. You used to have organizations in the 1920s who had a CEO, but it wasn’t a chief executive officer. It was a chief electricity officer, whose sole responsibility was how to figure out… I don’t know a whole lot of organizations that are still hiring chief electricity officers. I think that better decision making, again, is the key more than AI itself. I think it’s going to help more and more industries.
Scott Clendaniel: 01:07:55
I just don’t think that you’re going to have VIKI from iRobot making all the decisions for the planet. I think that is an overblown fear. I think it’s going to impact more and more organizations, but I think that we tend to go to this pendulum. “No AI. It doesn’t help anything,” to, “AI solves everything.” The answer tends to be somewhere in the middle, and just be aware of that.
Kirill Eremenko: 01:08:25
Gotcha. Understood. One final question, this one will be from me. There are so many types of ways to structure your data science division. This is like a data science management style question. One is to integrate individual data scientists across different functions like sales and then finance and operations and so on. Another one is to have a centralized data science team, who service all those functions. What is your preferred style and why?
Scott Clendaniel: 01:09:04
I’m going to steal this one from Harvard Business Review. That’s to use a hub and spoke model. You have a central core of folks who help the rest of the organization work on data science project. These are the folks who are going to be making sure that folks have the right tools that they need to help establish some processes, some standards, and so forth. That team is very small, and that most data scientists sit in the individual group that they need to serve.
Scott Clendaniel: 01:09:31
Your hub supports the people and the spokes in the different departments, and help them achieve their goals. I think that is the best way to do it. It is so easy to have folks be in a data science group who were out of touch with the needs of their clients that’s actually making them physically sit in that organizational structure helps solve a lot of those needs. That’s the way I would do it.
Kirill Eremenko: 01:09:58
Wow. Fantastic. Thank you. Scott, it’s been amazing. We’ve actually gone over time, but it was totally worth it. Loved these questions and your answers. Before I let you go, before we wrap up, I wanted to ask you what… Do you have a recommendation or just some piece of advice for specifically advanced data scientists out there who are listening to this podcast, so any parting thoughts?
Scott Clendaniel: 01:10:26
I do. That is that… I’ll tell it through another story. The first time I was ever invited to participate in an AI conference, I went running into my co-worker’s office. Her name was Beth. I said, “Beth, it’s fantastic. I’ve been invited to speak at an artificial intelligence conference. Isn’t that great?” She folds her arms across her chest. She leans back in her chair. She raises one eyebrow and says, “Do you really feel qualified to speak at such a conference?”
Scott Clendaniel: 01:10:57
I was like, “I did 10 seconds ago.” There is a lot of folks in our industry who look like Beth. That’s a bad idea. We need to be inclusive. Get down off your high horse. It’s a technology. It’s an area of expertise. We need to be inclusive, not exclusive. Try and be nice to people. Try and help them achieve their goals. Common manners and being polite and listening to people are really important in any field. But if you’re hoping for AI to have a big impact in your company, trying to prove how smart you are and how unsmart they are is a really bad idea.
Scott Clendaniel: 01:11:37
There are way too many of us who do that. Be inclusive. Incorporate as many people as you can. Be as helpful as you can, and stop doing this approach that, “I am some type of God of intellect because I know how to build a model.” You can build a model and tell folks. A lot of people could do that if only would someone take the time to show them how to do it. Be the person who’s helping bring more people into the fold, not explaining to everybody else why they’re wrong.
Kirill Eremenko: 01:12:07
That’s amazing advice. You actually walk the walk and talk the talk, right? That’s the saying. You live by that yourself.
Scott Clendaniel: 01:12:17
Thank you.
Kirill Eremenko: 01:12:19
I look at your comments on LinkedIn, and you’re always there supporting people, answering questions every time. Even in this thread, people asked you a question, you’d not just say, “Ask the question and thank you,” but you actually put a little image of a thank you, a written thank you. Every time, a different one. I could just imagine you have a whole library of these that you can use at any given time. It’s really cool. Why do you do it? Why do you help the community so much?
Scott Clendaniel: 01:12:48
I think because I was treated so poorly by the experts in our field when I tried to break into the field. I like to tease with people that for the first half of my career, people told me I couldn’t do this because I didn’t have a PhD in statistics. The second half of my career, everyone tells me I can’t do it because I don’t have a PhD in data science. I was like, “But some of my models seem to be working pretty well, I don’t know.” I think that it’s just a way of bringing more people in and being helpful because people need encouragement.
Scott Clendaniel: 01:13:22
We’ve got enough people out there in the world telling everyone else that they’re wrong. I think a little bit of kindness and support to other people goes a long way. I think we’re in better shape if we all just treated one another with a little more respect, a little more kindness, and a little less roughness and a little less intellectual aloofness.
Kirill Eremenko: 01:13:42
Awesome. Fantastic. Well, Scott, thank you.
Scott Clendaniel: 01:13:44
I don’t want anybody else to have a three-year-old on the phone saying, “Did you pack my toys yet?” with no prospects of finding a new job?
Kirill Eremenko: 01:13:54
Thank you very much for sharing that. I think it’s-
Scott Clendaniel: 01:13:57
Thank you. I really enjoyed this.
Kirill Eremenko: 01:14:00
Awesome. Me too. Tell us how can people get in touch, follow you, connect with you?
Scott Clendaniel: 01:14:07
The best way is to follow me on LinkedIn at T.Scott Clendaniel. I can’t accept all the invitation requests because I’m almost at the 30,000 limit. They won’t allow more people in, but please follow me. If you have questions, send me a message. I can’t answer everybody, but I do my best. I think I answered, gosh, over two dozen questions in the existing forum, and I will continue to try and do that.
Kirill Eremenko: 01:14:30
Fantastic. Thank you very much. You already gave your book recommendation. Could you just remind us? I think it was Seven Habits of Highly Effective People.
Scott Clendaniel: 01:14:38
Seven Habits of Highly Effective People by Stephen Covey, who is no longer with us-
Kirill Eremenko: 01:14:44
Awesome. That’s-
Scott Clendaniel: 01:14:44
… but his legacy lives on, great advice in there.
Kirill Eremenko: 01:14:48
Gotcha. Wonderful. On that note, once again, thank you so much. We’ll share all the links in the show notes, and please guys and everybody listening, connect with Scott. This has been a great opportunity to have you on the podcast. Thank you for coming.
Scott Clendaniel: 01:15:07
Thank you. Take care.
Kirill Eremenko: 01:15:14
There we have it, everybody. Hope you enjoyed this conversation as much as I did. I learned a ton from this discussion. There were so many cool advanced things that I didn’t know about before, and just blown away. Thank you so much, Scott, for coming on the show, and sharing all these insights with us. Perhaps my favorite part was when we spoke about over sampling the minority class. I could feel Scott’s confidence, and quite a tricky technique to just throw away a lot of your data in order to make sure that the positives and negatives are equally roughly the same.
Kirill Eremenko: 01:15:48
It’s a difficult decision to make, but the confidence which he spoke of was clear that he’s done this speaking many times. It’s obviously works for him. I really liked the discussion about data science leadership, and that if you want to be a data science leader one day, start now because soft skills are going to be… You need soft skills as a data scientist, not just as a lead data scientist. There we go.
Kirill Eremenko: 01:16:12
As usual, you can get the show notes at www.superdatascience.com/385. That’s www.superdatascience.com/385. There, you can get the transcript for this episode, any materials we mentioned, including a URL to Scott’s LinkedIn. Make sure to connect with him, or just look him up on LinkedIn. It’s T.Scott Clendaniel. He’s always, always very helpful. Just recently, he shared some amazing cheat sheets for machine learning. Even that is worth checking out.
Kirill Eremenko: 01:16:45
I had a look at them. This was some cheat sheets that was shared by Stanford University. He shared them on his LinkedIn, and there are some really cool cheat sheets there, including around cross validation. Check that out. As always, if you enjoyed this episode, share it with somebody, especially if you know an intermediate data scientist who’s looking to become advanced or an advanced data scientist who wants to further their skills in the space, a colleague maybe you know, a friend, a family member.
Kirill Eremenko: 01:17:17
Send them this episode, very easy to share. Send them the link www.superdatascience.com/385. On that note, my friends, thank you so much for being here today for sharing this hour or just more than an hour with us. I hope to see you back here next time, where we will be continuing to deliver on the promise of amazing episodes with very interesting, incredible guests. Until then, happy analyzing.
