SDS 299: Becoming Seasoned At Failure

Kirill Eremenko: Welcome to the SuperDataScience Podcast. My name is Kirill Eremenko, Data Science Coach and Lifestyle Entrepreneur. Each week we bring you inspiring people and ideas to help you build your successful career in data science. Thanks for being here today, and now let’s make the complex simple.

Kirill Eremenko: So here we go. One final time for DataScienceGO 2019. This podcast is brought to you by DataScienceGO and it is happening, this event, this conference is happening this weekend. Literally, in two days from now, on Friday we’ve got the workshops kicking off, I’m running one of those workshops on visualization, presentation and we’ve got five other amazing workshops. Plus, we’ve got the evening networking event on Friday.

Kirill Eremenko: Then Saturday is the main event of DataScienceGO with amazing speakers such as Hadelin, Ayodele Odubela we’ve heard on the podcast literally on the previous episode, then we’ve got Ben Taylor and many, many more great speakers coming from companies ranging from Atlasian to Google, from Amazon to Salesforce, and many, many more. Then of course we’ve got the networking on Saturday evening and then on Sunday we’ve got another day of awesomeness with lots of amazing speakers again.

Kirill Eremenko: So if you haven’t gotten your ticket to DataScienceGO 2019 then this is your last chance. Head on over to www.datasciencego.com, pick up your ticket there today and join us in San Diego. Join us and hundreds, literally hundreds of passionate, inspired, driven data scientists who are going to gather together to bring this community together, to share their experiences, and grow together. You don’t want to miss out on this so get your tickets today at datasciencego.com and I’ll see you there.

Kirill Eremenko: Welcome back to the SuperDataScience Podcast, ladies and gentlemen, super excited to have you back here on the show. I hope you are ready for a very exciting ride today. Today we had Michelle Keim. I literally just got off the phone with her, who is the Head of Data Science and Machine Learning at Pluralsight. What we did on this podcast is we went through all of Michelle’s experience, all the way from her degrees, from her bachelor’s and post-grad degrees, and then through the career steps that she’s made in her career journey, which have been numerous. This episode, to me, felt like a very intriguing and exciting TV show. Every time after every career path, I was left sitting on the edge of my seat thinking, wondering what’s going to come next, what happened next to Michelle, where does she go from there. It was very exciting and also each one of them had tons and tons of learning.

Kirill Eremenko: In this podcast you will find out things like working remotely, how that feels and what it’s all about in data science, failure, and why everyone should lose their job at least once. Churn and segmentation, we’ll talk about what they meant 10 years ago and what they mean now in terms of companies and how companies see them differently and what that means for data science. Imposter syndrome and what to do when you feel like an imposter when you’re applying for a role. Moving from centralized data science teams to integrated experts within the business and leading people on the three key learnings that Michelle has taken away from her experience of leading people.

Kirill Eremenko: One more thing to mention is that Michelle is a speaker at our conference, DataScienceGO 2019, which is happening actually this weekend. At the time of this podcast going out is happening this weekend on the 27th, 28th, and 29th of September. And if you haven’t gotten your tickets yet, you can find them at www.datasciencego.com. So, make sure to pick them up there today and meet us at DataScienceGO where you will get to meet Michelle and hear the rest of her story of how she’s driving data science at Pluralsight, which is an online education company. Speaking of online education, which we’re all very passionate about. So, there we go. On that note, I’m super pumped for you to check out this episode, and without further ado, I bring to you, Michelle Keim, Head of Data Science and Machine Learning at Pluralsight. 

Kirill Eremenko: Welcome to the SuperDataScience Podcast, ladies and gentlemen, super excited to have you on the show. And today’s guest is Michelle Keim from Pluralsight. Michelle, how are you today?

Michelle Keim: I am doing great. How are you doing?

Kirill Eremenko: I’m doing fantastic as well, and it’s really cool just now to chat about how we’re both working remotely today because… In general, we work remotely. I rarely meet people who as a career have structured their work remotely. So, Pluralsight is based in Boston, is that right? And you’re in San Diego.

Michelle Keim: We’re actually headquartered out of the Salt Lake City area. But one of our… We have a major headquartered office out in Boston as well, and I am fortunate enough to be based out of San Diego.

Kirill Eremenko: That’s really nice. How did that happen originally? Were you always remote based or is it just in Pluralsight?

Michelle Keim: Just a Pluralsight, and it’s been a little bit of a journey. We started our data function here out of San Diego. We had a small group of us who basically started doing data for Pluralsight, stood up our teams around data analytics, data engineering, data warehousing, and from my side data science. And we’re working together in a small co-working space and really grew that function here out of San Diego to the point where we had opened up an office and had a number of product development teams here in addition to our data functions.

Michelle Keim: And so, have a fairly large presence still, and through the early days of Pluralsight we did a number of acquisitions and ended up with employees and offices all over the country, and realized that we really needed to consolidate geographically to align where we were at with the vision of where we wanted to go by 2020, which is hard to believe we’re almost there already, and sort of been doing that. Ended up settling on the geographies where we are now, but what that meant for us is it created a lot of opportunity for remote folks who had been with the company for a while.

Kirill Eremenko: Wow, that’s exciting. As you say, it’s so crazy to think that we’re almost 2020. I didn’t think of this, but I was at a company in 2015 I think, start of 2015. And we were working on the five year vision, the 2020 vision, and now we’re already here if you think about it.

Michelle Keim: Yeah, that’s exactly what you’re saying. We have a 2020 vision, which we’re now having to start checking against because it’s almost here. 

Kirill Eremenko: Do you find that it’s hard to keep up with this vision in the sense that technology has moved… Has technology moved in the direction that you thought it would move when you put the 2020 vision together or has it moved in a different direction? Like some things might not even be relevant anymore in that 2020 vision.

Michelle Keim: I think we’ve learned some things along the way. But I think largely, given the mission and vision we had for the company as a whole that we’ve remained true to that. Maybe some of the ways we’ve tactically accomplished it, what we might have envisioned, a couple are a few years back. But I think we’ve been pretty straight on to where we’re going.

Kirill Eremenko: Do you have an example of something in the 2020 vision that you can share with us that maybe was accomplished or is going to be accomplished or was a learning?

Michelle Keim: I think one of the terms of where we were a few years back, we were still pretty immature in the space of really being a true online learning platform. We certainly have years of experience of delivering online courses, and really being able to scale up technology professionals. But we saw a greater need and opportunity to do more and really seeing the need not just with individual technologists, but with technology leaders and the massive transformations they were going through, and what we could provide for them to support that. In addition to looking at it at an individual basis. And so, that’s been an area where we’ve seen tremendous growth and opportunity, and continue to expand, and really see a lot of opportunity to help companies of all sizes be able to bring their teams and the skill sets up to where the tech stocks are going as well. And really becoming proficient in cloud where for a lot of them that’s really the transformation that they’re trying to make.

Kirill Eremenko: Okay. So, basically moving from just helping individual technologists to helping companies, and executives, growing the whole suite. This is actually a good segue into, could you give us a quick rundown of what is Pluralsight? What is the company today? What services do you provide and to whom?

Michelle Keim: Yeah. So, we really are all about skills development. Our mission is to democratize skills, learning across the world. And so, what we do is we provide a platform to help folks to skill up, whether that’s on an individual basis, or an entire organization. So, we’re very mission driven to that. Given where we’re at in this day and age, and everything being technology based, and the things that it enables, and the transformation that can bring to the world as a whole, is really what we’re after. And figuring out how we get that skills not only to the folks who are already in those roles, and trying to further skill up, but create equal access and opportunity around the world as well.

Kirill Eremenko: Okay, and is there specific topics that you cover in terms of skills? Or is it like Khan Academy, for example, they teach absolutely everything?

Michelle Keim: We do not teach absolutely everything. Actually, we focus on our core set of expertise. So, we grew out of the software developer profession, but as technology has changed, our core business is around the technologies in software development, IT professionals, security, data, data professionals, cloud, and so forth. So, we’re very much in that tech tech space and applying it in real world problems.

Kirill Eremenko: Okay. So you do teach things like data science, AI, machine learning?

Michelle Keim: We do. It’s been a growing area for us.

Kirill Eremenko: Okay. It must be very interesting for you, because you’re a data science leader yourself, and you’ve seen the company go from when I’m assuming you weren’t teaching data science, and now you are teaching data science while you’re doing data science in the company.

Michelle Keim: Yes. It’s very exciting. Gives us the opportunity and makes it more compelling for us to be able to use and try out our own product, which helps basically really reinforce that cycle of what works well, and where we see opportunity to improve what we’re delivering to our customers.

Kirill Eremenko: Yeah, for sure. That’s very… You must feel… Even your team must feel really proud that not only are we doing this stuff, we’re also teaching it. Okay. Well, Michelle, let’s move a bit to your journey. So, before we discuss your current work at Pluralsight, and which is very exciting, and I’m really looking forward to it. But first, could you tell us a bit about how you got there? What kind of background? What did you finish as a degree, and then through your roles where did your career take you that led to Pluralsight?

Michelle Keim: Yeah. I always have to chuckle because I didn’t start out as a child knowing that I would grow up and do this. I had not a clue if I will… The role didn’t even exist back when I was trying to figure out what I want had to do with my life. But what I did know, which is probably common to a lot of data scientists is I just loved solving problems. When I was a kid, and I don’t know if you’re familiar with this, but there was… I actually picked one up recently. But there’s a magazine called Games Magazine, just like this magazine you could go get on the magazine rack in the bookstore, and the airport that has all these logic problems, and weird types of crossword puzzles. I loved doing those when I was a kid. And so, I think there’s this sort of inherent piece of me that just likes that sort of logic and problem solving.

Michelle Keim: And so, I fell into the mathematical space, went through my undergrad and just continued to love math. So that’s what I chose to study with not a clue what I was going to do with it. When I was done, and got to the end of that program still didn’t really know, and I had explored accounting and teaching, and none of it really quite resonated. So, I stalled. I went off to graduate school. I had taken a couple of statistics courses through my undergraduate program, and I really just liked the tangibleness of it. The fact that there was data and things that I could picture or even touch and look at. And so, that fascinated me, and that’s how I ended up at the University of Washington doing a PhD in statistics, which in hindsight was just the most amazing accident because it led into a career in Applied Statistics, which really just transformed over the years into data mining and advanced analytics. Then into what we now call data science.

Kirill Eremenko: Okay. Wow. It’s very interesting because I kind of had a similar experience. I went to do physics, just because I liked it, I had no idea what I’m going to do with it afterwards. While studying it I realized, I still love it, but I’m not going to build a career in physics because it goes too narrow. You have to do a PhD… For me anyway, I want the more breadth. But nevertheless, that background allowed me to then augment it with finance and things like that and move into data science. Yeah, but very interesting that you chose mathematics and then led you to statistics. It’s a very interesting comment as well that it has this tangible aspect to it. It’s not just like theoretical math, and I’m assuming that’s what you’re comparing to. Theoretical math, which is great for advancing science, and/or theory of mathematics, but you can’t really apply it in business, right?

Michelle Keim: Yeah. You got data, you can plot it, look at it, explore it, it’s its own thing.

Kirill Eremenko: Yeah. I remember when I had a course, in my bachelor, I think in year four I had a course on statistics. This one specific one, I really didn’t like it. It was these ANOVAs and things like that. I was just like, “I don’t like this course, and so on.” I still did it, and now that is data science. I was like, “Oh, okay. Now I see the value.” I guess when it’s individual, just statistics on its own for me. Like you say, at that time, I didn’t see the tangible aspect of it. But once it becomes data science, it’s much easier to see how it’s applied in the real world. I was like, it’s an axiom that businesses need data science. Where did that take you? Now you have statistics, as a background, what was your first job?

Michelle Keim: My first job was actually, again, I feel very grateful, I was lucky. I ended up taking a role at the Boeing Company in Seattle. It was a role that was half research and half really applied statistics. And so, it was a really great transition out of graduate school. I had been in a very research centric department. Able to work on very applied problems, but that had been what I’ve been doing for the previous years, and to be able to take some of that research, continue it. And then even more so be able to think about how we applied it to some of the problems that the company was trying to solve just provided a really nice transition away from the academia world that I’d been in to really starting to understand business and how we could be leveraging some of these things to actually impact a company.

Kirill Eremenko: Okay. Got you. Then I see from LinkedIn you were in Boeing for three years. Why did you change?

Michelle Keim: Oh, why did I change? Well, the timing was such that there was the allure of the .com boom that was happening around the time. I’d hit three years in combination with I had colleagues who told me, if I stayed at Boeing for five years, I would be there for the rest of my life. It was sort of you either get out before five, or you become a lifer, which I don’t know if that’s true anymore. But it was funny because there were some really fantastic individuals that I worked with that had been with the company for 30, 40 years, which you don’t see anymore. I didn’t necessarily see anything wrong with that. But there was so much exciting stuff going on out in the world, companies starting, and I got lured to go try out this little startup, which did nothing, but we had a great time. We blew through the money, and six months later I was collecting unemployment.

Kirill Eremenko: Oh, wow. What a change. Did you feel any regret that you left Boeing at that point? Did you feel like, I shouldn’t have done that. I should have stayed at Boeing.

Michelle Keim: I didn’t actually because it was such a great experience. It was so different, and it really introduced me to just a more rapid pace of doing things and more of a sense of urgency, very different than being obviously in such a large company as Boeing. It was a really, really great experience, and I think it’s probably what has kept me excited about being in roles that are more agile, and/or I have more of an ability to have an influence, and really see what I’m doing, and how it’s impacting the business that I’m working for.

Kirill Eremenko: A lot of the time when I catch up with our students in real life, whether it’s at events or meetups and things like that, I face this common pattern of fear. People are fearful of changing jobs, jumping, changing careers, and so on. And that’s why this story of how you went from Boeing such a stable big company where you could have had 10, 20, 30 years successful career in future you went to startup, burnt through the money. And that’s it. You were collecting unemployment. It’s really exciting to hear that you were not feeling regret. I think that’s going to be very inspiring for people because even though you ended up in a situation, which you didn’t foresee, and is probably very uncomfortable, and has very little certainty to it. There, you saw the bright side in it, and the lessons you learned allowed you to, I guess, take your career even further. So what happened next?

Michelle Keim: Yeah, you’re right. I think there’s something to be said for getting… Everyone should probably lose their job once in their career. I think it’s really good for you to experience that, and realize that you can just get back at it. There’s so much opportunity out there. So, I did it again. I went to work for another small company.

Kirill Eremenko: You got addicted.

Michelle Keim: I got addicted. This one lasted longer. I think I was at the next company for two or three years. We were very early on in the space of trying to automate the process of developing predictive models. The founders had developed software that would optimize for predictive performance across both variable selection, and model type. We were trying to sell into companies, sell software. At the time, it was a really hard sell because small companies didn’t want to buy software, and big companies were already very well embedded with institutions using SaaS to try out models. And so, there was sort of, we weren’t really finding the niche where the software was desired. And so, we pivoted to more of a consultancy. So, leveraging our own product to deliver models to more mid tier companies.

Michelle Keim: We were doing a lot of work with multi channel retailers. Again, right at the verge when people were starting to optimize email campaigns, but there was a lot of money in mail, in hard postal mail through catalogs and other mailers and making sure that you’re targeting the right customer set that’s most likely to continue to respond and bring value to the company through those channels.

Kirill Eremenko: Okay. And so, what happened to that company? Did they burn down as well, or did you just get tired of working there?

Michelle Keim: [inaudible 00:23:01] it burned down as well, but the day we got pulled into the meeting room, I knew exactly what were going to [crosstalk 00:23:07]. Like, okay, I’ve been through this. Pretty sure I’m going home today. I know what to do next.

Kirill Eremenko: Oh, wow. You probably were the only one in the room who was well prepared. Everybody else was like la la la la la, I wonder what this meeting is about?

Michelle Keim: I think there was probably a sense, but for me it was less of an… I think I had seen it coming, and I knew that… At that point knew that I knew how to go look for a job, and that it would all work out.

Kirill Eremenko: Wow. That is so cool. After that you were definitely, become seasoned at failure, which is good, and which I want our listeners… I’d love our listeners to, if they haven’t had this experienced themselves, learn from your experience or take away as much as they can from this because being good at failure is the number one skill for success paradoxically. 

Michelle Keim: Failure is the biggest thing that you learn from. In large, and sort of in the meta sense, but also even in the day to day.

Kirill Eremenko: Yeah. For sure. How often do we fail every day? You try cooking a new dish, are you going to be good at it the first time? Never. Everything, a new sport you pick up, anything, totally agree. All right. This is like a really cool TV show, tell us what happened next? Where did you go from there?

Michelle Keim: I guess this is the beauty of having a good chunk of my career under my belt, right?

Kirill Eremenko: Yeah, yeah.

Michelle Keim: Stories to tell. I actually stuck… The next one stuck for a while. I went and spent about five years at T-Mobile. Perhaps one of my… It was one of my favorite parts of my career. I really, really enjoyed it. It was a great company to work for, very good culture. I didn’t know what I was doing when I started. And because I had been out of work I wasn’t being particularly particular about what I did next. I just saw an opportunity and seemed interesting. But I was hired, and I wasn’t quite sure what I was being hired for. But there was this sense of we know we need someone like you to come and help us with this data stuff. We know we should be doing customer segmentation. We think there’s some areas where we should be doing this.

Michelle Keim: And so, it turns out, it was a fantastic opportunity because I had an opportunity to shape that and see that. And there was a ton of things that weren’t being done yet that I had an opportunity to go tackle. You here, again, not maybe our younger listeners, but back in that time frame a lot of the classic examples of modeling were around things like churn models. I all of a sudden found myself in a telecommunications company at a time when we were starting to get to the end of the growth phase where it was all about customer acquisition. Then we were having to pivot and really start thinking about customer retention. And so, I had an opportunity to build out and work with various folks throughout the business to build out that first churn model. And then see that into action, and into business use. So, it was really fun.

Kirill Eremenko: In grace, you were at the right place at the right time.

Michelle Keim: Yep.

Kirill Eremenko: Tell me this. How do you go from working in two failed startups to landing such a one would say dream job at a large company like T-Mobile? How did you apply for the jobs? How did they pick your resume, if you know, or what did you say at the interviews when they asked you, “So, it looks like there’s a common denominator in the failed startups you were in. What’s happening here?” With that kind of experience, what do you have to say for yourself that they gave you the job? Where did you get this job?

Michelle Keim: Well, I don’t remember what I said for myself at the time. Apparently, I said the right thing. I think that was probably the only job in my career where I just applied for something and got a phone call off of my resume. Everything else I’ve gone after has been through, after that first role at Boeing had really been more about me going out and seeing what was out there and figuring out… seeing what companies were doing things that looked interesting, and then kind of finding a way to get an introductory conversation.

Michelle Keim: I remember T-Mobile was literally I had just seen a job posting that sounded interesting. And somehow my background must have aligned with where they were at the time. Because you have sort of… I know, we all at some point in our careers get that imposter syndrome of, I’m not really qualified to do this job. I definitely had it at the time because I was stepping into a leadership role too, which I had never done. I was brought on board and had two employees out of the gate, and I’d never lead people. That was definitely a kind of thrown to the fire learning experience on that. But yeah, again, just right place, right time, I guess. Right set of skills for the role.

Kirill Eremenko: And good on T-Mobile. I guess they made a fantastic choice putting you into a leading role. Look where you are now, and obviously somehow they maybe had sensed that you have a knack for being a leader. And so, you started with two people there, did the team grow while you were there? Five years, you were at T-Mobile for five years.

Michelle Keim: Yeah, it was five years. We grew quite a bit. I don’t think we grew… I think we grew to maybe a half a dozen folks by the time I left. I can’t remember actually exactly the size. But we’d gotten to the point where we were still in that place of being a centralized team supporting the organization, doing more modeling work, doing more kind of just that sort of decision support type of analytics as well. We had gotten deeper with the customer segmentation type of modeling as well. Looking at lifetime value, those sorts of things were hot at the time.

Kirill Eremenko: They were hot at the time, are they still hot now? Or have customer churn models or segmentation models or approaches to these problems, have they changed since then? Well, they most likely have. It’s been over 10 years. How has it changed?

Michelle Keim: It was funny because the next couple of roles I had it wasn’t the same businesses or the same… It wasn’t telecommunications anymore. But when you get into these, essentially, service oriented businesses, there’s always a component of retention. Even in my current role today, it still holds true that it’s easier to retain a customer than to go acquire a new one. And so, I had a couple of roles later, where the same types of different data, a little bit different application, but very, very similar type of question underneath of it.

Kirill Eremenko: So, you would say that these problems you were solving at T-Mobile back then they’re still relevant, and companies would still use the same approaches to solve them?

Michelle Keim: Questions are still relevant. I think that we have a lot more predictive modeling techniques at hand now that we can apply to them, and better ability to computationally optimized for what we’re trying to do. I think we know more now, and a couple of I think things that have changed is we have more methodology now that enables us to help make those models explainable. Whereas, back when we started, I remember, even if I go back to T-Mobile, there was a very high level of importance being placed on ensuring that we were getting the right factors and drivers into the model so that the business would have confidence in the model itself before they would just go out and apply.

Michelle Keim: The black box idea was unacceptable. And I think it continues to be something that we have to pay attention to. But there’s more… So much has developed over the years in terms of our ability to do that, and not have to rely on the trade off between perhaps some lost and predict performance of a decision tree. But still, having explainability. It’s not treating those as two separate parts of the problem has been an advancement there. I think there was another. I lost my thought on the second one, but that’s definitely been one where I’ve seen some change.

Kirill Eremenko: Okay. I guess also how companies approach customer relationships has changed as well. That before it was more kind of get a lot of customers, and put your product ahead first, that’s the most important thing, and then find people like… Now it’s more customers come first. Customers are like, all right, how do we serve people best? I guess it has… that change in mentality that rather than we have a great product, go out and sell it, to we have great customers, let’s find best ways we can serve them. That shift in psychology of business has also affected the approaches we take to actually turn modeling or segmentation and things like that. It’s just this overarching theme. Would you agree or do you think it’s something different in this space?

Michelle Keim: No, there’s a huge component to it. I’ll contrast a little bit. Back when we were modeling, doing churn modeling at T-Mobile it was new to us. What we were trying to do was automate who we should contact, and try to save. Kind of this little bit of a side story. But the funny thing was, that we learned was, and this will be of no surprise to anyone is you don’t want to contact people because we were looking at people who were… This was back in the days of contracts who were coming up on end of contracts, and who was most likely to turn off at the end of their contract. What we learned was that if we had an outreach to them, that essentially was the reminder call that their contract was up, and we were actually increasing churn. And so, what we found was we almost had to… We basically flipped how we were approaching the problem. It was more about who should we not contact? Who should we make sure we don’t dial up? There’s still this, it’s not just the modeling, it’s how do you apply it.

Michelle Keim: I think to your point around the customer aspect of it, I went through, and talk about failure. I went through a failure. I felt like in my time here at Pluralsight, where again, I started with this company at a fairly early stage. When I started, no one was quite sure exactly what they wanted me to do, which was fine. But one of the things that out of the gate was suggested that I might look at was building a retention model and looking at our business customers, and how could we help our sales team understand which ones were potentially at risk for not renewing.

Michelle Keim: I spent time with the business folks. I thought I understood that the underlying data and how it related to the business and what it did, and went through the modeling exercise. But the failure was really understanding what the use case for that model was. So, we had a model output, and no one was interested in a model at the time. To your point, there was a very hands on touch. It was, I’m going to need to talk to everyone. I have a relationship with these people. More of what I need to understand is what’s going on with them? If they are at risk, like what is the underlying factor? How do I get to the heart of what I need to converse with them about, as opposed to just tell me who’s at risk?

Kirill Eremenko: Okay. The model was outputting who they need to talk to, but they actually needed to talk to anyone anyway, right?

Michelle Keim: Or they had a sense as to… They’d had that human factor sense already. They had some of these relationships, and they had some of the intuition, so they needed data to support a different aspect of the problem. Maybe they would use it to prioritize their time, but really they wanted to know why. If I surface this as a high risk customer, why?

Kirill Eremenko: They explainability part basically, that’s what they [crosstalk 00:36:08]. Were you able to supply that?

Michelle Keim: We did eventually get to that, but it took kind of a reset on to really getting in with the individuals who were in that part of the business and understanding the work that they were doing on a day to day basis, and where data could come help. It kind of was almost like hitting the reset button and rethinking what the problem really was there.

Kirill Eremenko: Yeah, it’s a great example of failure. That’s such a common thing that in the data science process, that first initial step of identifying the problem, and actually either quantifying it or describing it in extreme detail and putting that down in writing with the stakeholders. That’s a step that a lot of people miss, and they just jump into, okay, let’s clean the data. Let’s analyze the data, model things or visualize and present. But if you haven’t done the first part, you don’t know exactly what problem you’re solving that the rest of that process might be in vain, and you’ll end up with something that people don’t actually use, which can be heart wrenching as well. If you spend a month working on something.

Michelle Keim: That’s the last thing a data scientist wants us to, at least for most of us is that’s a lot of why we’re in these roles is we want to see the work having an impact and being useful to someone.

Kirill Eremenko: Yeah. For sure. Okay, so that was T-Mobile five years, the story goes on. We’ve got three more seasons.

Michelle Keim: It does go on, it does.

Kirill Eremenko: What happened then?

Michelle Keim: Well, I moved to California. As I said, I’m down here in San Diego.

Kirill Eremenko: Where were you before?

Michelle Keim: I was in Seattle. So, I ended up in Seattle for graduate school, and then all of these other roles were Seattle based companies. And then I had kids, and I really wanted to be closer to my parents and my siblings, and so relocated down here to San Diego, and had to start from scratch looking for work in a geography I wasn’t particularly familiar with. The professional environment, and didn’t have context down here really yet. So, kind of hit the reset button, and ended up finding a small company against through that process of just looking at who’s doing things with data, and where might there be a need. San Diego is an interesting market because we’re not one of these cities where you get to see a lot of large companies with headquarters, and you know where the major employers are. It tends to be a little trickier than that.

Michelle Keim: And so, I ended up finding this company who was looking at analyzing energy data. So data coming out of meters from utility companies, gas and electric, and that the core of the business was really about revenue assurance, and working with utilities and helping them identify potentially lost revenue, either through fraudulent behavior, meter tampering, or just through meter failure, and being able to identify those. They had developed a number of… they called them models, but they were kind of algorithms to find anomalies in patterns of the data coming out of gas and electric meters, and being able to then prioritize for the utility companies where there was a strong case that something was up on-site, and then they would be able to send someone out to go and evaluate that, see what might be going on at the meter.

Michelle Keim: That was at the time where we were moving more into the smart meter technology as well. And so, we were starting to look at the data that was coming out of residential smart meters with measurements coming at frequent, I think 15 minute intervals at the time. And how could we have identify potentially what appliances were in the home. One of the big applications in San Diego, because we get a lot of heat waves, and then you have to worry about blackouts and so forth. How do you manage that energy demand? So, we were trained… and there’s a lot of pools, and so people who have pools in a hot climate have to run pool pumps. That’s a high, high load. And so, being able to identify where those existed at, and for utility companies to be able to incent, or request that. Homeowners run those at off peak hours was something they wanted to be able to target at the time. So, it was kind of an interesting problem to dig into to that data. It was very different than anything I had worked on, on priors. It was very interesting.

Kirill Eremenko: Okay. What were the results? Were you able to create that model? 

Michelle Keim: We did. By the time I left, we were very early phase had some success with pool pumps. We were looking for some other high loads as well, less common here, but electric dryers are a high load as well. And so, we were exploring some of these things, and trying to assess the business opportunity at the same time. It was a new space for the company in terms of is this something that was a sellable product, even if we could do that? [crosstalk 00:41:21]. So yeah, it was very interesting.

Kirill Eremenko: So, you were able to detect areas that have pool pumps based solely on the load that you’re seeing in the grid?

Michelle Keim: Yes. You would see certain patterns, you would start to detect in that profile of the electric usage.

Kirill Eremenko: That’s so cool. That’s like data science investigating. That’s just like, wow. That’s really cool. Finding physical objects or elements about people’s lifestyles through a byproduct like the electric output or electric utilization. All right. Very interesting. And so, when you joined that, was this a small company because T-Mobile was a very large company. Did you go back to your small company [crosstalk 00:42:18]?

Michelle Keim: I did. I went back to a small company. I think it was around maybe 40, 50 people at the time, which ended up being why I left. The company was small and struggling a bit. And so, it was financially becoming challenging for them to move forward in some of these more innovative areas. So, I ended up moving on, and that was the start of where I found my space more in the education world, and ended up working for a company called Bridgepoint Education. It was parent company for a couple of online, higher ed for profit universities. [crosstalk 00:42:55]. Kind of got into that area.

Kirill Eremenko: Was this a leadership role again?

Michelle Keim: This one was back in a leadership role. And it was, again, it fit the model of, they’d hired a couple of people to start to do data science, and not even sure if we called it data science at the time. Do advanced analytics, and were looking for a leader. And so, I stepped in very early there as well to build and lead and grow a team in that organization.

Kirill Eremenko: Okay. All right. How did that go? What was your biggest learning? On LinkedIn, I can see that you changed a few roles inside, you went from quantitative analytics manager to director of advanced analytics to AVP? What does AVP stand for?

Michelle Keim: What was it? I think it’s an associate… I don’t remember, it was whatever they [crosstalk 00:43:47]-

Kirill Eremenko: Associate vice president?

Michelle Keim: Yeah, something like that. However, their internal titling worked. I think that just represented the growth in that we had in the team, and the span of the work that we were doing at the time.

Kirill Eremenko: Okay. So, what was the biggest… Let’s probably start with the biggest failure. You seemed to be very good at them. What was the biggest failure, and what was the biggest learning that you had there?

Michelle Keim: Oh, that’s a good one. Probably the biggest learning, and I don’t know if it was a learning so much as a frustration or just an experience that I had. But there was, because we had built, we had been doing a lot of really great projects, and really demonstrating what you could do with data, we started to get more attention from the CEO and the executive suite. And he saw that he could start asking questions that maybe we could answer with data. And so, we ended up getting asked to do a lot of special projects. These special projects tended to revolve around helping to figure out why something bad had happened. I’m oversimplifying that, but there were things that had been going on in the business, and helping to truly understand what might have caused… I can’t remember… I suspect there was, part of it again, was around retention because of the way the model worked, and we weren’t retaining students at the same level in certain sectors, and why was that?

Michelle Keim: The challenge with that is all of these types of analysis are asking you to look at data in retrospect, and trying to figure out how the heck do you control for all of the different things that had been going on and really be able to pinpoint any single thing while dealing with an audience that wants an answer, but yet from a data and statistical perspective you just really can’t provide one. And so, you’re trying to look at it from every single angle, and teach about what’s possible with data? And also, how you can answer these questions in the future and starting to think about testing different things, and how do you eliminate all of those confounding factors to be able to answer your questions. I guess, for me the learning of that was just the challenge of being able to communicate and be realistic about what data can and can’t do.

Kirill Eremenko: Okay. How did you communicate that to the CEO that this is not a question that data can answer?

Michelle Keim: Directly and with providing as much information as you can, and being able to get at what you can say, but being very clear at what you can’t. Because when you get in those situations, sometimes some of the things that we do, particularly as it relates to analytics and trying` to answer questions, you may or may not have major decisions being made off of them. But in some of these things scenarios where you’ve got an executive who really wants to lean on data to be able to answer a question, you want to feel really confident in, not only what you’re telling them, but in being sure that the message that they’re taking away from that actually aligns with the message you’re trying to communicate.

Kirill Eremenko: Okay. A lot of times people fall into the trap, especially when communicating with CEOs that they don’t want to deliver bad news, right? So, for instance this problem cannot be addressed through data. There’s something external. Something else is happening. How do you go about preparing somebody for that you’re going to tell them not what they want to hear?

Michelle Keim: Be prepared, it is stressful. It was definitely a stressful time because not only were we being asked these questions, we were being asked to answer them at very tight timelines as well. And so, I think one of the things that’s really important is in how you present the results, and it was really a exercise in ensuring that the data presentation, the format of the charts, the wording of what you were saying that you could conclude from the chart that required a lot of very thoughtful care. And so, I think the advice there is you can’t produce pages and pages of analysis. You really have to figure out how do you boil this down to the key thing? What is the key thing I learned, and how do I take that key message? And now figure out how I tell that story with the data that I’ve seen that supports it?

Kirill Eremenko: Okay, totally agree. Yeah. Especially with executives, you want to make it very short. Bad or good news has got to be short, and I like what you said that you not only need to communicate the right things, but also you need to make sure the things you’re taking away. At least 20% of information is lost in the process of communication, maybe more. So, you got to make sure, are they getting the key insight?

Kirill Eremenko: All right. So, we’re up to the final season of this epic saga. From Bridgepoint Education, you move to Pluralsight, how did that happen?

Michelle Keim: That happened through a colleague actually. I had a colleague who I’d been working with at Bridgepoint, and he was responsible for the data analytics function there and had also been teaching courses for Pluralsight as an author. And so, he had that connection. He’s got his own stories. I won’t try and retell it, but at the end of the day, he ended up connecting with the CEO, and they found a recognition that there was… mutually found this need in the company to start doing more with data. There was just so much opportunity being lost in terms of the analytics that they didn’t have visibility to.

Michelle Keim: So, he created for himself a role of standing up our data function. And in doing that, reached out to me and a couple of other folks, that we’d had the opportunity to work with in the past, and started with a small group of a half a dozen of us from scratch figuring out how do we go build a data warehouse? How do we stand up our analytics? That’s getting Tableau into the organization. What do we want to do with data science? What are the questions we can start tackling on that side front first, and grew from there.

Kirill Eremenko: Okay. So, when you joined Pluralsight, how many people were in your team?

Michelle Keim: There was no team, it was team of me.

Kirill Eremenko: Okay. So, back to the old school days. You went from leader to IC, that’s the word, Individual Contributor. Then what happened? How big is your team now?

Michelle Keim: My team is actually small again because we’ve actually grown so much that we’ve reorganized how we’re structured. And so, I actually have a small team of four principals in the data space. They serve as leaders and mentors, and expert coworkers to a larger set of machine learning engineers and data scientists that are then embedded in our organization. I actually don’t know total count how many we are now. Probably in the company, we’re probably in the 20 to 30 range folks doing some form of data science, and machine learning. It’s been a massive shift from where we were five years ago.

Kirill Eremenko: Okay, so you move… The way it’s structured is you have four principals reporting to you, and then you have these pockets of data scientists around the organization. So, rather than having a centralized data science unit or function, you have data science across the board, is that right?

Michelle Keim: Yes and no. We actually have an interesting structure where we have actually two areas where we’re doing data science. We have a data science team that is centralized in our finance organization doing strategic decision support. As we grew, I actually moved over into our experienced organization. So, I’m fully focused on everything now about data science and machine learning within our product, experience, and everything that leads into that. I think before we got on the line we talked a lot about why five years has been no problem to stick around. It’s been so much change in the role, and about maybe a year and a half, two years ago, we were really doing almost nothing from a data science, machine learning perspective in our product just because of where we were at from a maturity standpoint. We saw and had a vision for you know, where we wanted to go with that. So I ended up pivoting and creating focus and starting to essentially kind of almost back to scratch again, basically standing up and growing the data science, machine learning function explicitly within our experience organization.

Michelle Keim: It’s within that organization, which is I think we’re over 500 employees, out of the company now just in that organization. And so, we about six months ago moved from being centralized in the experience organization to this new model with these principals. We’re only about six months into that and still learning and figuring out what’s working well with that, and what adjustments we need to make as well.

Kirill Eremenko: Okay. Six months ago, what did it look like six months ago?

Michelle Keim: Six months ago we were about a dozen data science, machine learning engineers in experience organizations centralized in the organization. So folks were basically consulting out to different product experience teams, and dipping in and out. I’m sure not news to your listeners, but there’s goods and bads with that model. There’s definitely a lot of cohesion, and flexibility that comes with that centralization. But you also end up with this sort of not… You never quite get that proximity to the problems you’re trying to solve in that way, or this maybe full sense of ownership for them. And so, flipping this model we’ve been trying to keep both the good aspects of the centralization by having this team of principals looking at the organization more holistically and supporting them while still having individual contributors in the work with teams in specific product areas.

Kirill Eremenko: Okay. You just took those people that are in your team, 12 of them, and eight of them you moved around the business, and said, “All right. Now you’re in this team.” Team A, you’re in team B, you’re attached to this specific team. And so, now you’re going to be working… We’ll be helping you, but you’re going to be mostly sitting with them. Is that how it went?

Michelle Keim: Yeah, yeah, pretty much.

Kirill Eremenko: That’s so cool. I’ve heard that there’s two different parts, ways of doing a centralized data science seem more separated, integrate data scientists in separate teams. But I’ve never heard of a company of this scale move from, okay, we have a centralized team. Now let’s get these people integrated into the business. What are some of the learnings that you’re seeing or benefits that you’re seeing in these six months? Is it… obviously, there’s some challenges, but was the right decision?

Michelle Keim: I actually don’t know if it was the right decision. It was part of a larger organizational design. So we’ve designed this way across all of our disciplines including engineering, and product design, and product management. And so, I think a lot of things have worked well. It’s definitely given more autonomy, and more ability to be agile to the teams in that they have data scientists and machine learning engineers as part of the team. They don’t have to ask for them or negotiate with someone else to get them. And they do have these individuals who are fully embedded and understanding of their problem space and talking on a day to day basis with individuals in the team that are coming from other disciplines. There’s definitely a closer relationship there.

Michelle Keim: Some of the things we’ve been learning have just been more around… Particularly for the principals it’s been a lot about time management, and how do I figure out which of these teams needs my help the most right now? And how do I really make sure that I’m where I need to be and spending my time on the right things.

Kirill Eremenko: Okay. All right. Sounds like you’ll still see in the coming months how this plays out. Okay. What I’d like also to know is you being a leader in multiple of the roles that you’ve been in, what is it like to lead a team of data scientists? Creating this data science function in a business, maintaining it, and leading people. What are some of the key insights that you can share with our listeners who are maybe considering becoming data science leaders, are now practitioners or ICs, individual contributors, and are considering becoming leaders further down in their career. What are some of the insights you can share with us?

Michelle Keim: Oh, boy. I’ll start with the challenges. I think one of the… Maybe speaking specifically to the transition. I think depending on where you’re at, and the size of the team you’re working with there can be opportunity to play a dual role where you’re a player/coach. You actually have the ability to be in the work and doing work in addition to leading people. I’ve sat in that space for a period of time. But if you’re in a role that’s growing, your ability to do that won’t be sustainable over time. And at some point, you have to have that heart to heart with yourself of do I want to be a people leader, and a strategic leader in the organization, or do I really like doing the work? Because it’s hard to do both. 

Michelle Keim: For me, I’ve been through that a couple of times, and pivoted back. It’s only been perhaps in this role where I finally realized, I like doing this. There’s actually aspects to it that really resonate well with me, but that’s me personally. It’s not everyone’s cup of tea. I guess, that’d be the first thing is just really getting true with yourself and figuring out like we talked about before the show, what do you want to be when you grow up? What else? I guess the funny thing is that I see in it is people… Leading and working with people is almost in itself an analytical problem.

Michelle Keim: I actually really enjoy getting to know people, understanding them as human beings, but also them as a data scientists, machine learning engineers, and what motivates them, what makes them tick. What things are they really great at, and how do we help make sure that they’re really directing their skills where they can have the biggest impact? But simultaneously understanding both where they have opportunity to grow and where they want to grow, and being able to provide those opportunities in parallel. It’s almost a little bit of a problem solving exercise in and of itself. I enjoy that, and I enjoy the people part of it a lot.

Kirill Eremenko: Okay, got you. So, having that heart to heart with yourself, and understanding if you want to lead, and become more of a people person, or if you want to continue the technical aspect of your career. Also, number two was understanding people and understanding there was an analytic… is a challenge in itself. Understanding them as humans, as contributors, as data scientists. Is there a third one? Is there a third learning or share to run down?

Michelle Keim: There should always be three, right?

Kirill Eremenko: Yeah.

Michelle Keim: Probably the third learning, I guess, if I had to round out the trifecta is the aspect of just the part of the role that is really building the culture in the business that you’re working with. So, that the [inaudible 01:01:04] successful too, because it’s not just the managing the people and the work, it’s ensuring that you’re working on the right things. And that those things are going to have impact, and that the teams or parts of the organization that are going to be utilizing that understand it. So, there’s just this whole aspect around it of just education. It could be frustrating sometimes, particularly if you’re in a company that’s growing, and in some of the cases I’ve been, I was probably there sooner than they were ready for me. And so, really hoping to develop that in the organization at the same time.

Kirill Eremenko: Okay. Well, I’m really… It is a shame that we’re running out of time because I would really like to talk about, all these, how you went about making that change specific here at Pluralsight? And what were the challenges, and how you overcame those things. Maybe we can one day have a second podcast where we can dive specifically into that because it feels like there’s a lot of things there. But what I wanted to do is, first of all, thank you. A huge thank you for running down through your whole experience. It was really exciting to go through these steps of your career, and also understanding all the learnings. There’s been plenty of learnings and takeaways that you’ve shared today.

Kirill Eremenko: What I would like to do before we finish up though is shift gears a little bit and talk about DataScienceGO, and give our listeners who are coming to the conference a quick teaser or a preview. This episode is actually going to go out literally a few days before you’re going to be presenting at DataScienceGO. How do you feel about coming to DataScienceGO, and what are you going to be talking about there?

Michelle Keim: I’m super excited. Well, partly excited because it’s the first chance I’ve had to attend a conference in my hometown. So, I’m looking forward to folks getting to come to San Diego, and share in the city that I live in, and love. But I’m really excited just for the opportunity to meet and talk with people, both. I really enjoy the aspect of my role of mentoring folks who are maybe going through similar parts of their career that I’ve gotten under my belt. Even the ones around being out of work and trying to figure out what to do next. So, love just talking to people. I’m looking forward to that.

Michelle Keim: In terms of my talk, I think really kind of centering around these multiple experiences that we’ve talked about where I’ve had and really starting from scratch, and bringing data science into a company. I think as we discovered here, there’s hours upon hours to that. So really, focusing on the aspects of actually just building out that function, and growing, and leading teams is what we’ll dig into the limited amount of time that we’ll have in the session at the conference.

Kirill Eremenko: Oh, fantastic. So it’s basically going to be a great continuation of the podcast, basically.

Michelle Keim: Well, I hope so.

Kirill Eremenko: Listen to the podcast, come to the conference and continue from there. That’s awesome. Then ask any questions afterwards. Pester Michelle during lunch time, or at the drinks, and the dinner time. Okay, fantastic. All right. Well, Michelle, before I do let you go, can you please share with us how can our listeners contact you if they have any questions? You mentioned before the podcast that Pluralsight is always hiring. I don’t know maybe people will be interested to learn about that. What are some of the best ways to contact you, find out more about Pluralsight, follow you and your career?

Michelle Keim: Probably the easiest thing is just to come find me on LinkedIn. Most central place for all of that.

Kirill Eremenko: I agree, completely. Michelle Keim, and we’ll share your LinkedIn in the show notes as well. On that note, one final question for you today is what is a book that you can recommend to our listeners that can help them with their careers, or even just life journey, something that’s impacted you?

Michelle Keim: Oh, goodness, I have so many. I guess I’ll give the general response would be read anything you can about the domain space in which you’re working. I currently have found myself in product development. So, I’ve been doing a lot of reading about product development, and how that works. But I guess maybe to be a little closer to home, to data science. I’ll recommend it even though I’m only partway through. I’m currently reading the Weapons of Math Destruction by Cathy O’Neil, which gets into the ethics of modeling. So far, there’s some really interesting stuff in there, and I think it’s probably something that we should all be thinking about since this is a space we’re working in. It’s what is the impact of the things that we’re building and the work that we’re doing?

Kirill Eremenko: Okay, fantastic. That book has actually been recommended a few times. You’re enjoying it?

Michelle Keim: Yeah. I’m only a couple of chapters in, but it’s been really great to just see where it’s going so far.

Kirill Eremenko: Who recommended it to you?

Michelle Keim: Actually, it’s funny, we were doing a book club at work. And so, we’re doing a little bit of working through the book and coupling that with some different presenters coming in, and talking about different aspects of ethics as well. So, I’m getting to look at it with others at the same time.

Kirill Eremenko: Okay, got you. Ethics of what, ethics of books, or data science and machine learning?

Michelle Keim: More about the ethics of what we’re building, both from a machine learning standpoint, and how that has a potential to impact folks.

Kirill Eremenko: Okay, got you. All right. There we go, Weapons of Math Destruction by Cathy O’Neil. Once again, Michelle, thank you so much. Totally enjoyed this conversation. Flew by so quick, didn’t even notice. Thank you for coming on the show and sharing your insight. 

Michelle Keim: Yeah, thanks for having me. It was a pleasure.

Kirill Eremenko: Yeah, and I’ll see you at DataScienceGO. For us right now it’s a few weeks, but when this comes out, in a few days, so see you soon.

Michelle Keim: Great. Looking forward to it.

Kirill Eremenko: So, there we have it. That was Michelle Keim, Head of Data science and Machine Learning at Pluralsight. I hope you enjoyed this conversation as much as I did. I totally, totally loved talking to Michelle. What a journey. What a story of somebody who started out in data science and made her way through all these different career steps moving from small companies to big companies, from individual contributor roles to leader roles, back to IC, back to leader, and all the learnings. What amazing learnings Michelle shared with us. My probably biggest… The most interesting thing for me was this moving from a centralized data science team to a team of integrated experts throughout the business. I’ve never seen that before, as mentioned on the podcast, and I would love to learn more about how this is going to play out for Pluralsight. Very, very interesting move. And also demonstrates that they are indeed these two types of ways, two ways that you can organize a data science team. Either centralized or either integrated experts throughout different divisions of the business.

Kirill Eremenko: On that note, if you’d like to meet Michelle and hear the rest of her story, then come join us at DataScienceGO this weekend. We are happening in San Diego in the Hilton Bayfront 27, 28, 29th September. If you haven’t gotten your tickets yet, you can still get them at www.datasciencego.com, and we’d love to see you there. As always, the notes for this podcast are available at www.superdatascience.com/299. We’re almost at 300. How crazy is that? Once again, the episode number is www.superdatascience.com/299. There you can get the transcript for this episode, any materials that we mentioned on the podcast as well as the URL for Michelle’s LinkedIn where you can, and I highly encourage you to connect with Michelle.

Kirill Eremenko: There we go. That’s the end of today’s session. I hope you enjoyed it. Once again, if you haven’t gotten your tickets yet to DataScienceGO, you can find them at www.datasciencego.com, and I look forward to seeing you there. Until next time, happy analyzing.