SDS 259: Building Machine Autonomy With Neural Networks

Kirill Eremenko: Welcome to the SuperDataScience podcast. My name is Kirill Eremenko, Data Science Coach and Lifestyle Entrepreneur, and each week we bring you inspiring people and ideas to help you build your successful career in data science. Thanks for being here today, and now let’s make the complex simple.

Kirill Eremenko: This episode is brought to you by our very own data science conference, DataScienceGO 2019. There are plenty of data science conferences out there. DataScienceGO is not your ordinary data science event. This is a conference dedicated to career advancement. We have three days of immersive talks, panels and training sessions designed to teach, inspire, and guide you. There are three separate career tracks involved, so whether you’re a beginner, a practitioner or a manager you can find a career track for you and select the right talks to advance your career.

Kirill Eremenko: We’re expecting 40 speakers, that four, zero, 40 speakers to join us for DataScienceGO 2019. And just to give you a taste of what to expect, here are some of the speakers that we had in the previous years: Creator of Makeover Monday Andy Kriebel, AI Thought Leader Ben Taylor, Data Science Influencer Randy Lao, Data Science Mentor Kristen Kehrer, Founder of Visual Cinnamon Nadieh Bremer, Technology Futurist Pablos Holman, and many, many more.

Kirill Eremenko: This year we will have over 800 attendees from beginners to data scientists to managers and leaders. So there will be plenty of networking opportunities with our attendees and speakers, and you don’t want to miss out on that. That’s the best way to grow your data science network and grow your career. And as a bonus there will be a track for executives. So if you’re an executive listening to this, check this out. Last year at DataScienceGO X, which is our special track for executives, we had key business decision makers from Ellie Mae, Levi Strauss, Dell, Red Bull, and more.

Kirill Eremenko: So whether you’re a beginner, practitioner, manager or executive, DataScienceGO is for you. DataScienceGO is happening on the 27th, 28th, 29th of September 2019 in San Diego. Don’t miss out. You can get your tickets at www.datasciencego.com. I would personally love to see you there, network with you and help inspire your career or progress your business into the space of data science. Once again, the website is www.datasciencego.com, and I’ll see you there.

Kirill Eremenko: Welcome back to the SuperDataScience podcast, ladies and gentlemen. Super excited to have you on this episode because today we’ve got a very exciting, energetic, pumped up and informative episode prepared for you. On the show today I have Stephen Welch, who is a professor, a consultant, a YouTube celebrity in the space of machine learning, data science, computer vision and many other things. And the chat was incredible. We literally just got off the phone and I am so pumped for you to hear this. So this podcast has a few parts. At the start we talked about self-driving cars. You will learn pretty much everything you need to know about self-driving cars starting from the history of neural networks and how that was associated with self-driving cars from the ’60s, ’70s, ’80s and all the way until now. You’ll also learn about autonomous driving and the three components in the neural networks related to autonomous driving and what they are and how they work. You will find out about the five different levels of autonomous driving and where Tesla sits, where Audi sits, where Waymo sits, and where other companies are playing this field and what to expect in the next 10-20 years.

Kirill Eremenko: You’ll learn about the trolley problem and much, much more. This podcast is pumped with information about self-driving cars. Then we move on to Stephen’s day job, where he is a consultant in the space of computer vision and machine learning, in addition to being a professor at a local university. So as you can imagine, you’ll learn a ton. He’ll actually share a case study of how machine learning and neural networks can be applied to historically older industries which are a bit slow to pick up on these technologies, but that creates a lot of opportunities for data scientists. And finally, we’ll finish up with some very valuable career advice for those of you in the space of data science and AI, which I’m assuming is everybody.

Kirill Eremenko: So a very exciting podcast coming up. Before we dive straight into it, I wanted to give a shout out to our fan of the week. This one goes to Wilson Valle, who said, “Great resources, valuable information, weekly wonderful guests on the show.” Thank you very much, Wilson. And if you haven’t left a review yet, then head on over to iTunes or your podcast app, and you can leave us a review there. I would love to personally read it.

Kirill Eremenko: On that note, let’s dive straight into the world of self-driving cars, neural networks, machine learning and much, much more. And without further ado, I bring to you Stephen Welch.

Kirill Eremenko: Welcome back to the SuperDataScience podcast, ladies and gentlemen. Super excited to have you back here on the show, and with me today I have Stephen Welch calling in all the way from Charlotte, North Carolina. Stephen, welcome to the show. How are you today?

Stephen Welch: Great, thank you for having me.

Kirill Eremenko: It’s my pleasure, seriously, because as we were talking before the podcast, I magically realized that the way I encountered your work was when I was learning about AI myself and I needed to find some insights into neural networks, and I came across your YouTube videos, which are totally fantastic. Man, huge congratulations on the way you really simplify things and share that on YouTube. Thank you so much for that.

Stephen Welch: Oh, thank you.

Kirill Eremenko: Tell us a bit about yourself. How did you even get into this space of neural networks? And for those of us who haven’t encountered your channel, by the way, Stephen’s channel is extremely popular, hundreds of thousands of subscribers and millions of views, but in case somebody hasn’t heard about your work before, tell us a bit about yourself.

Stephen Welch: Yeah, I’d love to, yeah. So the channel is called Welch Labs, and I kind of got into it a little bit haphazardly. It’s a little bit difficult to describe how different the landscape of machine learning education was in 2013 and 2014. There are so many good resources now. It’s wild. You can learn what you need really, really quickly. So in 2012 I launched a startup with a good friend and we were using neural networks to try to build a better tool for musicians basically for guitar players. And it was all about neural networks. We thought this was a really important technology, and part of doing that, I had to train some neural networks from scratch on our own data sets. We started that work in 2012, and the year 2012 is an important year. That’s the year that the AlexNet paper came out, so I’m sure some of your listeners are aware of this paper. It’s the first really big landmark results in modern deep learning.

Stephen Welch: A lot of the technology had been there before, but the AlexNet paper just crushed this ImageNet benchmark basically. And that’s when modern deep learning as it is now, especially for computer vision, really came on the scene.

Kirill Eremenko: Was that by Yann LeCun?

Stephen Welch: So that is, the big PI on there, I believe it’s Yoshua Bengio I believe is the last author. The first author is Alex Krizhevsky. He’s at Google now. But yeah, the paper is definitely worth a read and not just for historical reasons. Really for a lot of the findings are still pretty relevant today.

Kirill Eremenko: Gotcha.

Stephen Welch: Yeah, yeah, absolutely. So I’ll get back to your question here. I’m getting a little sidetracked by some of the cool things that were going on then and how I got into this. So anyway, the point I was driving at here is as these new neural network and deep learning methodologies are coming around, the tools themselves and the academic literature and the textbooks, the tools you had to learn this stuff as an engineer were pretty limited. So I ended up, I needed to use neural networks. There was a couple libraries. I believe Caffe was around, so Caffe is a TensorFlow precursor, I would call it. And I think there is [inaudible 00:08:47]. So there are some old frameworks there. C++ for me, I don’t have a CS background, so really tough to learn. And as far as literature, there’s some stuff from Stanford. Andrew Wu was doing some work but either way, the bottom line here, it was really tough for me to learn neural networks.

Stephen Welch: It took me a whole summer. So all summer I’m sitting there trying to learn the differential equations, trying to code it up. I ended up making my own little simple library in Python, just because that was all I had that I could really make work for me. And when I finally got through I thought, “Man, you know, I should really just spend a little bit of time and try to make a resource for other people, so at least my efforts won’t only be siloed in my little area of working.”

Kirill Eremenko: And you also learn better when you explain things to others, right?

Stephen Welch: I think there’s tremendous value in, I would call it pedagogical thinking. I think Richard Feynman is probably my favorite example of this. I think the action of explaining something, especially for me for sure, the action of explaining something helps me understand it. So I think thinking like that is really, really valuable.

Kirill Eremenko: For sure. And if anybody’s interested more about Richard Feynman, a great book is You Must Be Kidding Mr. Feynman. I think it’s an autobiography. You read it?

Stephen Welch: Yeah, I have, yeah. And I’ve recommended it.

Kirill Eremenko: I loved it. It’s so funny. It’s about quantum physics and stuff, but it’s so funny at the same time. It’s ridiculous.

Stephen Welch: It is, yeah, yeah. It’s like a quasi autobiography, just him telling his stories basically. But he’s a character and just an inspirational person I think. I just love his attitude. Yeah, absolutely. So about that time I at least had a grasp on neural networks myself, and at the same time some of these early YouTube channels were really getting some traction. So I was in love at the time, I still am, with a channel called Minute Physics. The author is Henry Reich. It’s a really great channel. I encourage you to watch it. Just great topics, great content on physics. And I thought, “Okay, I am going to rip off Henry Reich. I’m going to try to steal as much of his production as I can. I’m going to copy his production as best I can, and I’m going to make it about neural networks.” So I was just, “I’m going to do it, see what happens, let’s try it, let’s release a video.”

Stephen Welch: So it took a few months to figure things out. Making videos, as I’m sure you know, it’s a long and slow and tedious process. But I finally finished. I released episode one. And the next morning literally, I woke up the next morning and it had a thousand views. And I was like, “Wow, that’s more than I thought.” And really in 2014 and 2013 it was an easier time on YouTube and Twitter. There was less good content, less competition. So that grew really quickly and that became a six- or seven-part series that I made on neural networks. And that series, I think episode one now has 600,000 views, which is crazy. That’s mind blowing to me, right?

Stephen Welch: That’s how I got into YouTube. That was some of my earlier education in machine learning as well. I did machine learning research in graduate school before that, but that was the first time I got really deep into the inner workings of an algorithm, I’d say.

Kirill Eremenko: Wow, wow. What a journey. And did you continue creating videos from there up to the six-part neural network series?

Stephen Welch: I did, yeah. It’s kind of like you have an early success and you’ll see this with bands. A lot of bands will have an early hit. Then they freak out, “Oh, what do I do next?” And the pressure’s on a little bit. So I floundered around a little bit. I did a series, something on physics. That was okay. And then I eventually landed on imaginary numbers. And this is not a machine learning data science topic, but it’s just a beautiful topic for mathematics and something I’d always been interested in. One of my first engineering loves was music and audio, and the really important piece of mathematics in music and audio but also in vision is the Fourier transform.

Stephen Welch: You look at the Fourier transform equation and the center is this imaginary number I. And I set out to make a series about the Fourier transform. I was very gung-ho. I was going to knock it out in a summer. No problem. And I realized I really didn’t understand what the I was doing in Fourier’s equation. This got me deeper and deeper into some research, and eventually it turned into a 13-part series on imaginary numbers. Which now, it’s my most popular series now. It has, I think the first episode has three million views or something wild.

Kirill Eremenko: Wow, congratulations.

Stephen Welch: Thank you. I’ve spoken to a lot of teachers that actually use it as part of high school education, which is really cool. I spent a whole summer working on this one animation where I pull this graph out of a page to show the concepts. Super time-consuming, but in the end definitely worthwhile. That’s where the channel went next. After that I did another machine learning series about decision trees, but it’s definitely been a journey. I’d say that it’s certainly been worthwhile. Really, really time-consuming and definitely, you don’t know what you expect when you get into it, but it’s definitely been rewarding, I would say.

Kirill Eremenko: Yeah. And our listeners out there, we’ll link to the imaginary numbers. So we’ll link to the neural network video part one and also to the imaginary numbers video part one. Highly recommend checking out that one. I’ve seen that video. It’s surreal how you move around. The graphics, it’s like augmented reality with your hand at the same time. And also part 13 is really cool. That’s a really advanced one.

Stephen Welch: Yeah, thank you.

Kirill Eremenko: People, check it out. Okay, cool. So then you moved onto some decision trees. Now you’re more into self-driving cars at the moment. Your latest series, right?

Stephen Welch: Yeah, great question, yeah. So I think that self-driving cars are one of the most interesting applied machine learning or AI problems really of our time. I think these are going to have a huge impact on our day to day life. And recently in the last couple years I had a chance to actually work on these as part of my day job. So as I was doing that in the back of my mind I was like, “Oh man, there’s so many cool stories I could tell.” So that eventually became this most recent series, and I really tried to go back into the history. I think the history is really, really fascinating. It really started back in the 1980s. Yeah, wild, right?

Stephen Welch: Actually if you go back to Carnegie Mellon, Carnegie Mellon in 1986. I was born in 1987, so I was negative one at this time. I think 1986, I think that’s when Richard Feynman died actually. That same year. Trying to tie the timeline together here. I was born the next year. So Carnegie Mellon, this is when Geoff Hinton, kind of the godfather of neural networks, he’s at Carnegie Mellon as an associate professor and he publishes a paper called, “Learning Representations by Data Propagating Errors,” something like that. It’s in Nature. It’s the data propagation paper.

Kirill Eremenko: I think I’ve read that one.

Stephen Welch: Oh, cool, yeah. It’s pretty readable. And the important result there is that they were able to train neural networks that were more than one layer deep. So before the dominant paradigm was the perceptron, and that’s a one-layer neural network. And those worked reasonably well, but there was this big book that came out in the ’70s from Marvin Minsky and Seymour Papert at MI basically bashing perceptrons, saying there’s all these problems that single-layer neural networks, also known as perceptrons, can’t solve.

Kirill Eremenko: Perceptrons have been around since the ’60s, right?

Stephen Welch: Yeah, yeah. It goes back to, I think Frank Rosenblatt was the big proponent of perceptrons. I give a lecture about this in my computer vision class. There’s some really great old historical stuff. I want to make a video out of it. But basically in the ’60s Frank Rosenblatt, he figured out what’s called the perceptron learning algorithm, which is how you can train a one-layer neural network. And he basically claimed that it can solve any problem. He went out to the press and he was like, “Oh, it can tell the difference between cats and dogs and do this and that,” you know. I was able to dig up some old quotes from the New York Times. They’re pretty ridiculous. So I’m hoping to use them in a future video. But yeah. That’s the one-layer neural network.

Kirill Eremenko: So basically it’s input layer, output layer, no hidden layers in between.

Stephen Welch: That’s right. Zero hidden layers, yeah, correct, yeah. So you just have one weight matrix you’re multiplying by in there. Set of biases. Just one layer, exactly. Input, output. So that was in the ’60s. In the ’70s neural networks went out of style. They came back in the ’80s thanks largely to this work at Carnegie Mellon by Geoff Hinton and David Rumelhart. And what I was getting to, we’ll come back to our autonomous driving, I promise. This paper came out in 1986 and it was how can we train multilayer neural networks, at the time called multilayer perceptrons. And really not that different than today’s deep learning models. Really not that different at all. The math is almost identical. So that was 1986. And at the same time there was a graduate student at Carnegie Mellon, this guy named Dean Pomerleau. I had a chance to speak with him. He’s the nicest guy. 

Stephen Welch: So at the same time he came to Carnegie Mellon and he started working on robots, self-driving robots in cars. And I guess he had enough contact with Hinton’s group that he thought, “Okay, maybe I could use a neural network to drive a car.” And that was the impetus. So really one of the first successful applications of multilayer neural networks was self-driving back in the ’80s. It was pretty wild, right? Because you would think they would use it to solve other problems, right? So I thought that was just fascinating. In the series, this most recent series, I really get into the history. There’s actually a little clip of me talking on the phone to Dean Pomerleau and he explains his thinking through this. It was cool to get to talk to some of these people who were really the pioneers. 

Stephen Welch: So we talk about that technique. That technique is called end to end deep learning. That’s where you use one neural network to learn the whole driving figure. Where it actually learns the angle to turn the steering wheel at. It’s incredible that it works, and actually Nvidia just recently in 2016, they released a paper where they did the same thing with a much deeper model. And their network does incredibly well. It drives across all different kinds of terrains and roads really, really well. The big catch, and I get into this in the series a little bit, is that these end to end systems where you use one neural network to drive the whole car, they’re really hard to test. In autonomous driving, the other reason it’s a really hard problem is because of safety and reliability concerns. So if you have this big monolithic neural network, it’s really hard to make it reliable. So I get into that in the series as well.

Stephen Welch: I think I got a little bit off topic, but that’s a little summary of the most recent [inaudible 00:19:28].

Kirill Eremenko: That’s really cool. It’s very different to models that are recently coming out. Have you had a chance to look at the full world model?

Stephen Welch: Yeah.

Kirill Eremenko: That one has variable autoencoders. For instance, for recreating different scenarios for self-training the model, it has MDM RNN for predicting the future, and then all of that is combined into one model and separated into parts so that training can be facilitated by controller and made, it can go through faster training, things like that. So completely different style. Now you have models with multiple neural networks working together to accomplish a common goal. Perhaps it’s the future, since they’re faster to train and more reliable, I guess.

Stephen Welch: Right. I think the other big thing that makes these, I’m just looking at the paper now, catching up a little bit here, but I think the other big thing is that with these techniques, a big challenge in autonomous driving, so generally you can divide autonomous driving into three big areas. So one is the perception, the computer vision perception algorithms. One is the mapping, so localization, knowing where you are in the world. And the third is planning and policy. So a lot of the leaders in the field think that this third, this planning and policy piece, which is kind of how do you actually interact with other drivers and things like that, they think it’s the hardest. And I agree. And one of the reasons it’s so hard is because it’s very difficult to solve with offline machine learning. So if you’re solving a perception problem, for example, you can record a bunch of data of driving and you can label all the cars and you can teach an object detector to detect your car. It’s no problem. The problem with planning and policy is that there’s this feedback loop.

Stephen Welch: So if I make a decision in my car about how to drive, then another driver’s going to respond to that, right? So if I just capture data offline, then I’ll only have one version of the world. So when you want to train these modules, these systems that have to deal with an interactive world, then the reinforcement learning becomes much, much more important. And I think that’s what this world model is getting at a little bit, I think. But I just looked at the paper very briefly. But those things are super relevant, I think.

Kirill Eremenko: So what role do you think, so when you talk about offline learning, for instance we put a camera in a car and we drive for 100 kilometers, collect all that data and then we train our model on that. Cool. But as you mentioned, it’s not good enough for when the algorithm actually makes decisions and other drivers react and so on. So what role do you think simulations play in this whole space? How reliable and realistic are simulations when we hear that Tesla’s cars did a billion miles in simulation or some other crazy number? Is that comparable to real life conditions?

Stephen Welch: Yeah, wonderful question, yeah. So simulation I think is a really important piece of the puzzle. I don’t think it solves everything by any means. I think you’d be foolish not to do it, because as you said the simulators are getting very good. There is one quote that always comes to mind when I talk about or think about simulation. And it’s from, he’s kind of a character, the guy’s name is George Hotz. Are you aware of this guy?

Kirill Eremenko: No, no.

Stephen Welch: Okay, cool. So he was the first person to actually hack the iPhone and the PlayStation. [inaudible 00:22:48].

Kirill Eremenko: I’ve heard of that happening, but I didn’t know who did it.

Stephen Welch: Yeah, yeah. It’s this guy named George Hotz. I’ve had a little interaction with him through the company I used to work for. Really just a character. Brilliant, brilliant guy, obviously. So after he retired from being a hacker, at least on paper retired, he founded an autonomous driving company in 2015, I think, called Comma.ai. So he raised some money from Andreeseen Horowitz, which is a great VC in the Valley, and worked on autonomous driving, made a product and all kinds of stuff. But anyway, he has a great quote about simulation, which he said, “All simulators are doomed to succeed.” I think what he means by that is that you can overput your models to your simulation. Your models are only going to be as good as your simulator. And if your simulator was in fact as real as the real world, then that’s actually, some people argue it’s a harder task to make a simulator as real as the real world than it is to just learn from the real world.

Stephen Welch: So I think that there are real limitations to simulators. But again, in autonomous driving especially, you’d be foolish not to use them. Anything that can make your product safer you should absolutely use. And what’s becoming common, I mentioned those three pillars of autonomous driving, the sensing and perception, the localization and the planning and policy. What you’re seeing is people are using simulation more for the planning and policy to really model the agent agent interaction. And then they are using real data more often for the perception, which really makes sense. For computer vision problems, for example, having real data out in the world is really, really valuable because I think the theory is the probability distribution that that data comes from is really, really complex and difficult to simulate.

Kirill Eremenko: Interesting. What are your comments on the differences between the types of self-driving vehicles, for instance, as I’ve heard Google Waymo uses lidars. So laser radars, whereas Tesla uses just computer vision, just cameras. Any thoughts on that? Benefits?

Stephen Welch: I could talk about those things for a whole hour.

Kirill Eremenko: In a nutshell.

Stephen Welch: Of course, I won’t, I promise. I won’t do the whole broadcast on this. But in a nutshell here, there is this big difference of opinion in the industry. It really boils down to if you’re going with what’s called level five or you’re going level two up. So the Society of Automotive Engineers has published a recommendation, which quantizes autonomous driving into levels. So level one would be adaptive cruise control. Level two is what Tesla Auto Pilot is now. So it does lane centering and adaptive cruise control together. So it keeps you in the center of your lane and it adjusts your speed so you don’t hit the car in front of you. Level five, on the other end of the spectrum, that is complete autonomy. No steering wheel, no human [inaudible 00:25:43].

Kirill Eremenko: I’ve seen the infographic for that. We’ll put it on the show notes as well. It’s a really cool illustration.

Stephen Welch: That would be great. It is, yeah. So that’s one way to chop up the world of autonomous driving. And if you chop it up that way, then there are some conclusions that are going to be a result of that. So as far as I know there’s a really great book that I think I read last year. It’s called Autonomy by Larry Burns. And that book gives the history of the Google self-driving project as part of it. And I think initially when Google did its development in 2013 they had Priuses and they were doing a self-driving car project under Google X I think. I think internally it wasn’t clear to them which path was right. Should we just go for a level five or should we go level two up? Should we incrementally build our autonomous system or should we go for the moon shot, right? And they did this infamous experiment now where they internally tested their self-driving cars with their own engineers. So they said, “Okay, you can use a self-driving car, but make sure you monitor it and that you pay attention. You are the fail safe.”

Stephen Welch: And they recorded, they had a cockpit camera and they recorded their engineers. And apparently their own engineers were terrible at this. They would send text messages, they would eat food. They really over trusted these systems. So I think, I’m sure that wasn’t the only evidence that factored into their reasoning, but at that point Google really said and now Waymo is really saying, “We’re going to go full level five. We don’t think that level two and three can be done safely because it involves this handoff and the human really is the fail safe.” And they say, “We don’t think this can be done safely.” And a bunch of companies now, Cruise, Aurora, a bunch of companies agree with Waymo and they’re doing only level five and level four.

Stephen Welch: And Tesla is on the opposite end of the market where they’re saying, “We’re going to incrementally add features” and they get better and better in time. So those are the two really philosophical differences in how [inaudible 00:27:37].

Kirill Eremenko: Interesting. Do you think it’s possible for, is Tesla going to need to introduce radars as well? Or it’s possible to do the whole self-driving level five with just video cameras?

Stephen Welch: Yeah, wonderful question. I have a little bit on this in part one of the series, and I think it’s just a fascinating question. So Elon Musk has said publicly at a TED event, he said this publicly, he thinks that level five is achievable with cameras. Actually no, I’m sorry, I shouldn’t misquote him. He says superhuman performance is achievable with just cameras. At points I think you could infer from their marketing that they’re claiming level four, level five with just cameras eventually. Right now they also use radar.

Kirill Eremenko: In Teslas?

Stephen Welch: In Teslas, correct. Just one radar. But the radar makes a big difference because those two sensors, the passive optical camera and the radar, have orthogonal strengths. So the radar can see much better at night and much better through fog and stuff like that. And it has a longer range. So those two sensors are very strong together. The big one that’s missing that everyone else at the top end of the market, the little five guys use, is lidar. So lidar where’s you have this spinning disk on top of your car and you bounce lasers off everything around you and you use the time of flight information to make it [inaudible 00:28:56].

Kirill Eremenko: What does lidar stand for?

Stephen Welch: It’s a messed up acronym. There’s different versions for it. There’s no one interpretation, but I think it’s Light, Distance and Ranging. It’s a playoff of radar, and radar is Radio, Distance and Ranging, I think. So it’s a much more expensive sensor, tens of thousands of dollars today, coming down in price. You can think of it being two to three orders of magnitude better than a camera right now. As far as the quality of data you’re getting. So Tesla and Elon Musk, who is much smarter than me, are betting you can do this with camera and radar.

Kirill Eremenko: So it’s just like humans, the way we drive cars?

Stephen Welch: Yeah. And humans don’t even have radar, right? So I think that’s probably an important part of his argument. Hey, if a human can do it then it must be feasible, right? Which, that’s something that we can debate as well, I think. But there’s a bunch of other smart folks like Waymo who think that that’s not true necessarily, especially right now. They think that lidar is a critical part of this solution. So it’s going to be just fascinating to see how this plays out in the next 5 to 10 years. I’ve gone back and forth on this personally at least two or three times. What’s the right philosophy and what’s the right sensor set? And yeah, it’s just a fascinating question.

Kirill Eremenko: Interesting, very interesting. And so you seem to know so much about self-driving cars. How did you dive, how do you dive so deep into these topics?

Stephen Welch: Yeah. Well, that one I kind of cheated a bit. I was lucky enough to be in meetings with people in the industry and generally no matter what I’m doing I try to read a lot of literature if I can. So I’ve tried to keep up with the autonomous driving literature as best I can. So there it was kind of serendipitous where the stuff on the YouTube channel was connected to what I was doing for my day job.

Kirill Eremenko: Was there anything else really interesting or cool that you can share with us about self-driving cars?

Stephen Welch: Yeah, it’s a great question. I think that philosophical debate about level one versus level five is probably one of the most interesting things.

Kirill Eremenko: What about the trolley problem?

Stephen Welch: Yes, yeah. So that’s the other big one, right? Again, it’s connected to this philosophical difference. So right now, as of now there are I think it’s three to four fatalities that have happened in Teslas, arguably when they are in auto pilot. And this isn’t exactly a trolley problem, but I think it will connect here. So I think maybe the first, because really when you say trolley problem I can get into that in more detail, but generally maybe these philosophical questions about how do we balance safety versus independence? What I mean by that, let’s say that tomorrow we had a self-driving car that was 10 times safer than a human driver. It was demonstrably proven. Proving that is hard, but let’s say you could. So last year in the US 30-40,000 people died in car accidents, which is heartbreaking. It’s terrible. It’s the biggest cause of death I think among people younger than 35. I think from the ages of 15-35 it’s one of the biggest causes of death.

Stephen Welch: So you got an algorithm that was 10 times safer. We could go from 35,000 deaths a year to 3,500 in the US. That’d be huge, right? But the big catch is that those remaining 3,500, those could be caused by machines, right? So as a society, is that a tradeoff that we’re willing to make? And I think especially in America, or at least I’m biased because that’s where I live, but a lot of people I’m sure would not be comfortable with someone in their family being killed by a machine like that. I’m sure that’s true globally. Even if it is for a greater societal good. So I think that’s another really interesting philosophical question that again, you’re going to see it play out in the next five to 10 years.

Kirill Eremenko: Yeah, man, that’s a very, very difficult part for society to decide on these types of things. And the reason for these questions is that you will never be able to make self-driving cars that are 100% safe. There’s always going to be accidents, no matter what.

Stephen Welch: I believe so, yeah. There’s some folks that I think would debate. But if you just think about being in a dense urban area, for example, right? So you have a sidewalk, right? It is in some cases one foot away from traffic that’s going let’s say 35 miles an hour, right? So conceivably a pedestrian could jump in front of your car at the very last second, right? In that situation the laws of physics themselves won’t let you stop fast enough. So there’s something that’s intrinsically dangerous about how cars work. So I think you can never guarantee 100% safety.

Kirill Eremenko: And the whole trolley problem boils down to do you for instance in situations where a child jumps into the road or something like that, does the car hit the child or does the car swerve off the cliff and kill the passengers inside the car? And so apart from it not ever being 100% safe, there’s also this dilemma who’s going to program the algorithms to make that decision. Does the car protect the pedestrian and decide that they’re under 18 years old, who gets priority? Is it the passengers in the car? Is it the pedestrian? Does it depend on their age, social status, other characteristics of the person? Somebody has to put that into the AI itself. And that’s I think where a lot of debate is happening right now.

Stephen Welch: Yeah, absolutely. There’s a really great paper in Nature where they take a really big poll across the whole world and they ask some questions about this, like should we value the more educated over the less educated or the wealthy over the less wealthy or the younger over the older? And the answers actually vary pretty heavily based on geography and culture, which is pretty interesting I think. So one of the suggestions from that article in Nature was the policies may need to be culturally specific.

Kirill Eremenko: Very different world we’re moving into. And how fast do you think we’re moving into this world? Because I’ve heard that, I haven’t been in one, but I’ve heard that in several states in the US there’s already self-driving automobiles.

Stephen Welch: Yeah. It’s a great question, yeah. So I’m going to be a little more skeptical on this stuff. And there’s people much smarter than me who think this stuff’s going to go faster. So that’s my disclaimer. So again, there’s really two ends of the market. There’s the level two and up end of the market, which is being led by Tesla, and there’s the level five and down, which Waymo is the leader. Waymo is the spinout of Google. So I think in the next five to 10 years the level two systems are going to keep getting better certainly. Audi, in their AA they have what they’re calling a level three system. But they have very much constrained what’s called the operational design domain of that system, which means that you can only use it on limited access highways below 30 miles an hour, I believe is the constraints. Which makes it an easier engineering problem to solve, which is smart. So they won’t let you use it outside of that domain, for example.

Stephen Welch: So those systems I believe will continue to get better. I think you will begin to see level five systems like Uber has worked on and Waymo. I think you’ll begin to see those deployed in geofenced areas. We’re already seeing that in Phoenix, so Waymo has a fleet that’s driving down the streets of Phoenix. Now to be fully level five you have to be able to drive anywhere. There can’t be any geofencing or constraints. I believe that that is still at least 10 years away, maybe 20. I’m a little long on that personally. I could be totally wrong. There’s people way smarter than me that are saying faster. I just think, because really what Waymo is doing right now is level four, because there’s still constraints on it. You can only be within this area of Phoenix, for example. So again, I could be totally wrong here, but my personal opinion is that 10-20 years for level five everywhere.

Stephen Welch: But you will continue to see these level two systems get better and better, and you’ll see, I think within cities you’ll start to see the robo taxis come to life over the next five to 10 years.

Kirill Eremenko: Fantastic. Well, Stephen, thank you so much for that overview of self-driving cars. We’re going to link to the videos, the YouTube videos, the first one in the series. If our listeners want to get more information on this, highly recommend checking them out. In the meantime on the podcast let’s switch gears a little bit and let’s talk about your, YouTube is your hobby. All this that we talked about is just the hobby part of our life. Tell us about your work. As we chatted about for the podcast, you’re a consultant, right? Machine learning and computer vision consultant. How’s that going?

Stephen Welch: Yeah, right, thanks for asking. In the beginning of this year or late last year I began working on a few consulting projects. And it’s really picked up some speed, which has been great. One of my part time jobs here is I’m a professor at the local university. So as part of doing that, as part of teaching computer vision I was able to get connected to the community here in Charlotte. And I started learning about really interesting computer vision problems out there. So for example, one of the early projects we’re working on is automatic defect detection for manufacturing. So in a lot of manufacturing you would be just amazed at how manual some of these processes still are. There are, there’s people right now that their job is to find defects in products. They work in 10-hour shifts, 12-hour shifts and their job is to find those defects. That is a very, very challenging job and they don’t always get it right, which I certainly wouldn’t.

Stephen Welch: And there are lots of really great ways to blend your humans and your technology in more intelligent ways. Not necessarily limiting those jobs, but making those jobs better and making those people more effective at their jobs. Exactly, right, yeah. That would be such a challenging job to have. So one of the first projects we’re doing is we’re doing automated defect detection using deep learning computer vision models.

Kirill Eremenko: Sorry, Stephen, when you say we’re you’re referring to yourself and your interns, is that right?

Stephen Welch: That’s right, yeah. It’s Welch Labs, the consulting arm of Welch Labs, yes.

Kirill Eremenko: Gotcha. Okay, please continue. Sorry for interrupting.

Stephen Welch: Oh yeah, no problem, yeah. So when I see we, me and we’re scaling slowly. These things grow slowly. But one other advantage of teaching at the local university is being able to make some contacts. So I identified a couple students that I was really, really impressed by and have made a couple part-time hires for now to help me out with algorithm development. But yes, that’s one of the first projects we’re working on is using deep learning to spot defects in manufacturing lines, which I think is going to be a great way to add a lot of value, and it’s really interesting too how depending on what industry you’re in deep learning has been either completely adopted and it’s over or it’s still new. So far in manufacturing the existing systems we’ve seen, we’ve had a chance to look at some existing systems and learn about different processes, and it’s going to take some time for these new computer vision machine learning innovations to make it all the way out to some of these older industries. Which makes for a lot of opportunity. I think it’s an exciting place to be right now.

Kirill Eremenko: That’s such a great way of putting it, that depending on industry deep learning is either completely adopted and it’s over, in quotation marks meaning it’s very hard to add new value with deep learning and innovate, or you just go and find an industry that is old and that could benefit from deep learning and you innovate there. There’s a lot of opportunities. This is such a great example, manufacturing defects. When I try to explain how can deep learning be applied to an old industry, one of my best, my go to examples, let’s say you have a conveyor belt and you’re sorting apples. You have five people standing there sorting which apples go left for the juice, which apples go straight to be placed in the market stalls and stuff like that.

Kirill Eremenko: They’re using their eyes to sort things out, to decide which ones are rotten apples, which ones need to be thrown away, which ones are good apples, which is the biggest. All you do is put a camera above the conveyor belt and you don’t even need to program anything into it. You put neural network and you get it to observe the actions of humans for two weeks, three weeks, five weeks, two months. And learn the ones and zeroes based on the actions or based on the sizes of the apples and how they look. That’s it. It will automatically come up with the criteria. And from there that’s so much help to the factory, the humans, and it can do that job so much faster. 

Stephen Welch: Yeah. It’s a huge value add, absolutely. And like you said, the machine learning really automates a lot of these processes for you. The one caveat there probably, especially in some of this early work we’ve done and kind of what I’ve seen in the industry, the learning is super automated. What’s slow and time-consuming and still kind of a grind sometimes is really getting good label data. So these unsupervised algorithms are really coming along quickly, which is exciting, but in my experience thus far the heavy lifting algorithms from today that are really being deployed are by and large supervised. So with the product I just mentioned, most of our time really goes to getting a really good label to training set. There’s a really great talk from Andrej Karpathy, he’s head of machine learning at Tesla now.

Kirill Eremenko: Oh really? He moved to Tesla?

Stephen Welch: Yeah, yeah. He did. Yeah.

Kirill Eremenko: Interesting.

Stephen Welch: Really great lecturer from Stanford.

Kirill Eremenko: I love his, also his blogs on Medium or wherever else they’re posted and [inaudible 00:42:55].

Stephen Welch: He wrote a great tool too called Archive Sanity Preserver. I’m not sure if you use it or not. But the archive is a great place to get papers [inaudible 00:43:08]. And he wrote a tool called Archive Sanity Preserver. And what it does is organizes and sorts archive for you, so you can see the most relevant papers. It’s actually my browser home page so I can see what’s going on quickly. But anyway, Andrej Karpathy, obviously brilliant engineer. He has a great talk recently from his role at Tesla, and really he spends almost the whole time talking about how they’re dealing with data, labeling data, dealing with ambiguous situations. So for them they obviously have really cutting edge deep learning models, which is great. But for them to really get to the long tail and really get good performance, it’s becoming more of a data problem. And I’ve observed that in my own work as well, where we spend more time on labeling and really having internally consistent policies for labeling. Whenever you have more than one human labeling it gets complicated, right?

Stephen Welch: So really some of the challenges are shifting there because the algorithms are getting so good that your labels are really the bottleneck.

Kirill Eremenko: Okay. Well, help me out then. In the example I gave with apples, could it go with an unsupervised algorithm? Or do you think labeling would be required there as well somehow?

Stephen Welch: Yeah, great question. So you posed it in kind of a cool way. I didn’t think of the way, when you first started the example I was like, “Oh yeah, makes sense.” But then I was like, “Oh, that’s kind of cool.” Let me tease apart what I was thinking versus what you said. So if I had to make an apple sorter I would probably start by sitting down with the team, and I’d probably get them to label their own data sets. So we’d take a bunch of images of apples and I would say, “Okay, you guys give me between 100 and 1000 examples of this kind of apple, that kind of apple, and we’ll go from there.” Starting with that manually labeling. I think what you are suggesting is that you use a camera to observe what’s happening now in the factory. And that’s a really cool idea. Because then what you could do, you’d still have to do some kind of, I think there would still be some kind of manual component, but what you’re really saying is that we want to use this to mimic the decisions the workers made in the first place.

Stephen Welch: Instead of having the workers themselves draw bounding boxes around bad apples in images, we could actually base this off their actions. And that’s a really cool idea. I think you’d have to do something a little bit explicit. You’d have to detect where the workers are and what decisions they’re making, or at least have some way of knowing which apples went where. I think what you’re proposing is a little more of an end to end system, which is a really cool idea. In practice, what I’ve seen is more of the former, where you have to sit down and really explicitly draw bounding boxes around your defects or your apples or whatever and then go from there. And I think what you’re describing, I’m sure there’s some folks doing it now, and that’s probably where the industry is going, I hope. Where you’re doing more of an end to end thing where you’re trying to really model what your workers are doing.

Kirill Eremenko: Awesome. Thanks for that. Really valuable insights. And at this point I’d like to talk a bit about tools. So what kind of tools do you use and oh, recently TensorFlow 2.0 was released. Any comments on that?

Stephen Welch: Yeah. That’s a great topic. Another one I could talk about for a whole podcast. Tools are really interesting and we’re seeing a lot of movement really, really quickly. So I think I spoke about this a little bit at the beginning. When I first got into machine learning, it’s difficult to describe how much harder the tools were to use. That was 2012. The world has really changed in six or seven years. So now I’d say Python is by far my most comfortable language. I kind of think in Python. If Python ever goes out of vogue I’m really in trouble. Going to be pretty obsolete. But within Python my most comfortable deep learning framework is certainly TensorFlow. I’ve been getting into some Keras recently. Keras makes some things a lot easier, which is great. And now you can really blend those two together. TensorFlow 2.0 is very, very exciting. I think they’re going in the right direction for sure. The couple experiments I’ve done with it so far, I haven’t, nothing seems, I wasn’t blown away. Not in a negative way. It just seemed fairly similar to what was going on before. Then I’ll have easier execution and things like that, which is cool.

Stephen Welch: I think they’re going in a good direction, spending a lot of time on it. I hadn’t spent much time with PyTorch until recently. And I really like it so far. I think as someone who really enjoys the Python syntax, I think PyTorch is a little bit easier to get into. And it’s also a very powerful framework. I think if you’re going to only bet on one, I tell my students this, I’d probably bet on TensorFlow and Keras to learn first. That’s going to be the most adaptable. I think there’s probably more open source work done in TensorFlow and Keras right now than in PyTorch. But they’re both really great frameworks and neither is too tough to learn. TensorFlow can be hairy. There’s some really gross error messages that TensorFlow will give you sometimes.

Stephen Welch: But I think TensorFlow 2.0 is interesting and just generally the amount of effort being put into these tools is really exciting. It’s so much easier to build one deep learning model and then compare 100 different models by doing hardware parameter tuning than there were five or six years ago.

Kirill Eremenko: Gotcha. And how difficult is it to get started for somebody listening to this podcast who’s never done deep learning before? How far we’ve advanced with these tools? How long would you think it would take somebody to get into it?

Stephen Welch: Yeah, that’s a really great question. So this is something that as a professor of computer vision, I think about this a lot or try to. And also as someone who’s interviewed a reasonable number of folks who are looking for jobs in this field, I quickly try to figure out if they have, because a lot of people will put TensorFlow on their resume, which is fine. But it’s really easy to walk through tutorials. There’s all these amazing notebooks right now, and you can solve really challenging problems that would have been really difficult to solve five years ago just by pressing control enter through your notebook sales, which is cool. But something I try to tease apart really quickly when I’m interviewing someone is hey, have they gone beyond the tutorial, right? Did they really get stuck on something? Did they do something novel?

Stephen Welch: So your question here is how long does it take. And I’d say that you can go run these tutorials in no time. I would say when you’re able to really create something new and create some value, that’s when you probably really start to notice something. And that I would think probably in a focused summer, three months or something like that, you can really start to solve some real problems. And what’s really important I think is, the field is super overwhelming right now. I struggle to keep up with the literature. There’s so many new papers, so many new ideas. And at some point you really just have to say, “Okay, I’m going to do this one thing. I’m going to try to build something myself. It’s going to break. That’s good.” So getting stuck and having stuff break, that’s the only way you’re really going to get deep enough.

Stephen Welch: And yeah, I think you have to just stop, focus on one thing, and if you have a toy problem or a problem that you’re interested in, that’s so much better. There’s this really great quote from Richard Feynman actually, [inaudible 00:49:58], it’s on my home page right now and it’s, “Study hard what interests you in the most undisciplined, irreverent and original manner possible.” For me that’s the way to learn. Get in there, look at the tutorials, start there but use it as your starting point. And then from there try to really build something. You’re going to get stuck and it’s going to be crappy for a while, but really that’s how you’re going to learn these things. And yeah, thanks to these tools being out there, I really think in three months you can really start to get deep into solving some real problems. Especially if you have some coding background.

Kirill Eremenko: Yeah. Wow. Couldn’t put it better. I totally agree. And that’s how for me personally it’s happened every time. Only when I have a problem that really excites me and I can just work day and night on it and spend a couple of weeks persevering and write thousands of lines of code. That’s when you actually make huge strides of progress as opposed to like yes, it’s important to follow tutorials, important to get the basics and understand how to use things. But then you need to practice. It’s like learning a language. You learn how to speak, say some words but if you keep it only just in the classroom you’re never going to be fluent. You need to go out there and force yourself to practice. You’re going to fail. You’re going to fail a lot.

Stephen Welch: Oh yeah, absolutely. I had, back in undergraduate in sophomore year, I was 19 years old and a professor gave us this exact spiel. I think it was a class called, it was thermodynamics, which is really a tough class. And I remember rolling my eyes. I was like, “Oh, that’s stupid.” He was so right, man. He was 100% right. And that was a really tough class, and hopefully I learned something. But yeah, it’s super true.

Kirill Eremenko: Awesome, awesome. Yeah, Stephen, so getting toward the end of this podcast. I wanted to ask you from, you’re a professor, you’re a consultant, you’re an educator on YouTube. You see a lot in the space of data science and artificial intelligence, neural networks. What is the one biggest tip or piece of advice you can give to our listeners about the future of this field for the next three to five years? What should they look into to be best prepared, best positioned for the future that’s coming?

Stephen Welch: Yeah, that’s a really great question. It’s exciting to be part of a field that’s so popular right now. So data science, machine learning, AI, all these things are really in a big upswing, which is great. It’s good because there are new opportunities. There’s a lot of momentum in the field. At the same time, that means there’s a lot of hype. A lot of things are going to get exaggerated. So I think as a practitioner, especially if you’re new, it’s going to be really important to focus. I’ve done a lot of interviews with students or people early in their career where they just know the very first, they know the Twitter version of everything. They’re like, “Oh, this team did this at this place with NLP,” which is cool. It’s great to know about the industry. That’s fantastic. But you got to have some depth, right? And to have depth you’re going to have to sacrifice some breadth. Unless you’re just amazingly brilliant. But if you’re like me, you’re smart but you’re not a genius, then you’re going to have to focus.

Stephen Welch: So I think I’d really encourage people to pick a domain, something you’re interested in, something where you think you can provide value maybe that has connection to work you’ve done in the past. Maybe it has connection to where you want to go. But pick one of those areas and really get deep. Build something yourself. Share some open source code. Write some code. I think that’s really critical. Because if you just try to keep up to date on Twitter or keep up to date with archive, you’ll just drown. It’s impossible.

Kirill Eremenko: Gotcha. Fantastic. So it’s kind of like, it’s great to know all the most cutting edge tools and know about them and what they’re called and even maybe how they’re used and coded, but you got to have the substance, right? You got to have the applications. Some tangible applications that you can see, not even just show others, but see for yourself that yes, I’ve actually made a difference using this certain tool. That’s a good tool to know, to have. All right, moving on to the next or what else can I actually create with and make things happen with?

Stephen Welch: Absolutely, yeah. I know when I’m hiring I’m going to choose depth over breadth almost every time. As long as the depth is close enough to what I’m trying to solve. Because if someone’s gone deep into a discipline and they can really solve some of the core problems, then I know that they can learn new things as well. So that depth is so critical, and I would say that if you’re trying to allocate your time, I would really recommend spending five to 10 times as much effort and time on depth than on knowing everything that’s happening in the industry. Because yeah, there’s so much, and honestly a lot of it is noise. A lot of the stuff happening right now is not going to matter in five years. I think it’s just how these things go. We’re in this big upswing. This is a really popular area and that’s okay. Just don’t let it take too much of your time.

Kirill Eremenko: True. And I just want to reiterate, you mentioned this but just to say it again, you can’t go deep into everything. You can go broad, but then you have to pick, where are you going to go deep? Well, the best way to pick where you’re going to go deep is not what the hype is about, is actually what you’re passionate about.

Stephen Welch: Totally.

Kirill Eremenko: Like when you started, you were passionate about music and neural networks, right? So pick a topic and go deep there. You never know where it’ll take you. You might change direction along the way. But as long as you pick what you’re passionate about and you go that direction, well, guess what? The thing that you get deep in is going to be your passion, and therefore when you do find the right fit for your skills, the company, the organization, the consulting firm that needs those skills, you’re going to end up working on what you’re passionate about anyway. And that’s a perfect setup.

Stephen Welch: Totally, totally.

Kirill Eremenko: Awesome. Well, Stephen, thank you so much for today, for all the insights. It’s been a crazy ride of a podcast from self-driving cars to consulting to education to future of the field. Before I let you go, what’s the best way for our listeners to get in touch, follow your career, get familiar with more of your work?

Stephen Welch: Yeah, great. So the Twitter handle is just @welchlabs. That’s W-E-L-C-H Labs, L-A-B-S. The YouTube channel goes under the same name. If you just Google Welch Labs you’ll find all these things.

Kirill Eremenko: Awesome. LinkedIn is okay to connect as well?

Stephen Welch: Of course.

Kirill Eremenko: Perfect. We’ll share all those in the show notes as well, and one final question for you today. What’s a book that you can recommend to our listeners to help them in their careers and life journeys?

Stephen Welch: Oh man, this is a tough one. I’m a connoisseur of engineering and math books. I think I own like 200 books or something silly at this point. But there are around six or seven that I find to be fantastic.

Kirill Eremenko: Just one, Stephen, just one.

Stephen Welch: For this audience I think the answer is The Deep Learning Book by Ian Goodfellow. So there’s five or 10 deep learning books right now. This one is in my mind by far the best.

Kirill Eremenko: Which you can I think get free on the web, right?

Stephen Welch: You can, yep. You can buy it on Amazon for 60 bucks or something, or it’s all online as well.

Kirill Eremenko: Okay, perfect. Well, I’ve looked at a few chapters of that book. Definitely great read. So Deep Learning Book by Ian Goodfellow. On that note, Stephen, thank you so much once again for coming on the show. I really enjoyed our conversation. I’m sure our listeners did too.

Stephen Welch: Great, thank you so much for your time.

Kirill Eremenko: So there you have it, ladies and gentlemen. What a podcast. Such a charged up conversation. I totally, totally enjoyed it, and I hope you did too. My personal favorite takeaway was perhaps what Stephen said about deep learning and how it can be applied to pretty much two types of areas or two types of industries. Industries where it’s been completely adopted and it’s “all over,” where basically it’s very hard to innovate with deep learning and neural networks because those technologies are quite well-adopted there already. And other industries where they’ve been a bit slow taking on these new technologies. The fact is that there’s plenty of industries out there. We just looked at one of the examples where on conveyor belts you might want to look at defects and use computer vision for that. There’s plenty of industries out there. And that’s actually a good thing. I think there’s a lot of opportunities and the world’s going to become a much, much better place over the coming years.

Kirill Eremenko: So there you go. That was my best takeaway, and of course there were plenty of valuable insights. So I wouldn’t be surprised if your best takeaway was a bit different. And of course, if you want to get in touch with Stephen, if you would like to invite him to do a consulting project for your business, you’d like to follow his career, you’d like to watch some of his YouTube videos, all of that information can be found in the show notes at www.www.superdatascience.com/259. That’s www.superdatascience.com/259. So make sure to hit Stephen up and follow him on LinkedIn and Twitter, and let him know what you thought of this podcast. I’m sure he would love to hear from you.

Kirill Eremenko: And on that note, if you know somebody who’s interested in self-driving cars, even if they’re not a data scientist, even if they’re not into artificial intelligence as a profession, send them this podcast. I’m sure they will appreciate all of the insights that Stephen shared on this episode. And with that said, thank you so much for being here today. I look forward to seeing you back here next time and until then, happy analyzing.