SDS 349: Human-in-the-Loop Algorithms in Retail

Kirill Eremenko: Welcome to the SuperDataScience podcast. My name is Kirill Eremenko, Data Science Coach and Lifestyle Entrepreneur. And each week we bring you inspiring people and ideas to help you build your successful career in data science. Thanks for being here today and now let’s make the complex simple. 

Kirill Eremenko: This episode is brought to you by SuperDataScience, our online membership platform for learning data science at any level. We’ve got over 2,500 video tutorials, over 200 hours of content, and 30 plus courses with new courses being added on average once per month. So all of that and more, you get as part of your membership at SuperDataScience. So don’t hold off. Sign up today at www.www.superdatascience.com. Secure your membership and take your data science skills to the next level. 

Kirill Eremenko: Welcome back to the SuperDataScience podcast, everybody, super excited to have you back here on the show. And I’ve got a question for you. What do you think Netflix, Amazon, Spotify and Uber all have in common? Oh, that’s right. They’re all algorithms companies. And right now, according to Forbes, we’re living in the golden age of algorithms. This is a time when companies that are based on algorithms that use algorithms, not just data science and data analytics, but specifically algorithms, those companies are the ones that are going to thrive. And today’s guest, Brad Klingenberg, is the Chief Algorithms Officer at Stitch Fix. 

Kirill Eremenko: So let me tell you a bit about Stitch Fix. Stitch Fix is an online personal styling service. They were established in 2011, so just a mere nine years ago. But in that time, they’ve already accomplished so much. In 2017, they went public. As of February 2018, they were valued at $2 billion. $2 billion. Right now, they have over 3 million active users worldwide. And on their algorithms team, which is headed by Brad, wait for it, they have 125 people on that team. How cool is that? And so in this episode, Brad walked us through exactly what is happening at Stitch Fix, and how they’re using algorithms to their advantage. This episode can roughly be broken down into three core components. We’ll be talking about algorithms quite a lot, and you will hear about the sheer diversity of applications of algorithms at Stitch Fix. It’s mind-blowing. 

Kirill Eremenko: They use algorithms for everything from personalization, to NLP to process texts, to computer vision to process images, other models to solve the traveling salesman problem. It’s hard to imagine a single activity in the business that is not underpinned by an algorithm. So that’s a very exciting component of what we talked about. Another really cool thing that you’ll hear about is how Stitch Fix combines humans and algorithms together. This is a company that is completely destroying the myth that algorithms will replace humans. In this company, algorithms work side by side with humans, and you’ll find out exactly what that means, how they accomplished it, and how people are satisfied with that process.

 Kirill Eremenko: And finally, the third part is what will be interesting to you if you’re an executive or manager, you want to lead a company at some point, it’s about how to build an algorithms-driven company. What does it take to make sure that your business has algorithms in its DNA? What are the trade offs, what are the advantages, and what are the pitfalls to look out for? In a nutshell, this is going to be an extremely insightful episode about algorithms and algorithm-driven companies. So without further ado, let’s welcome Brad Klingenberg, Chief Algorithms Officer at Stitch Fix. 

Kirill Eremenko: Welcome back to the Super Data Science podcast ladies and gentlemen. Super excited to have you on the show, and today’s guest is Brad Klingenberg from Stitch Fix, calling in from San Francisco. Brad, how are you going today? 

Brad Klingenberg: I’m doing quite well Kirill, thank you. 

Kirill Eremenko: Very excited to have you on the show. It’s been a while. We originally planned this for several months ago and then my schedule didn’t work. Your schedule didn’t work. But finally we’re here. We’ve going to be talking about algorithms and Stitch Fix. Are you excited about this? 

Brad Klingenberg: Absolutely. It’s a pleasure, and I’m glad we could align the schedules and make it happen. 

Kirill Eremenko: Yeah. Same here. And what I really appreciate, already talking to you a little bit just before the podcast now, you’re so excited about the company you work for. You talk about Stitch Fix, and we can feel it in the tone of your voice. What is the one thing that you would say, that makes you most excited about waking up in the morning and going to work? 

Brad Klingenberg: Oh, it’s so hard to choose just one. So I might cheat and say two. So Stitch Fix, just the problems that we work on are just fascinating. And so I feel, after having been here for more than six years now, I still learn and get to think about new things every day, which is exciting. And secondly, the people are great. So I think that combination of working on interesting things, with colleagues that you enjoy spending time with, is pretty hard to beat, and keeps you engaged in the long haul. 

Kirill Eremenko: That’s fantastic, to continue learning after six years. Would you say your learning has shifted? Maybe at the start was more about algorithms, and now it’s about leadership? Or is it still a combination of both? 

Brad Klingenberg: I think it’s a healthy combination of both. So my journey at Stitch Fix, today I’m the Chief Algorithms Officer, or lead our algorithms team which, which is largely a data science and platform organization that is focused on using data to improve our business. And so I joined Stitch Fix about six and a half years ago. So an individual contributor. And so certainly, it’s taking on people management and larger and larger teams. The focus of my work has shifted. So there’s definitely a lot to learn there. But part of what’s still so exciting to be here, is we have these wonderful, difficult open-ended problems around making our clients happier and buying inventory. And there’s really a richness of this problem space where many of the fundamental things we work on are just not solved. And that keeps me engaged on a technical level as well as managerial level. 

Kirill Eremenko: That’s, that’s fantastic. And congratulations. This is a first time I’ve heard the title Chief Algorithms Officer. Did Stitch Fix invent this title, and what does the title entail? 

Brad Klingenberg: It’s a great question. I can’t make, with certainty, the claim that it’s unique. But it’s certainly unusual. I think what it reflects, as much as anything, is really the unique role of the algorithms team in the company. So data science is a really interesting field, and you have different companies organized quite differently. And very often you’ll find data science or machine learning and algorithm development is a sub-function of engineering or other verticals at a company. And Stitch Fix is somewhat unusual in actually having an algorithms organization that is a top-level department, alongside other functions of a business. And I think this has been important historically, and just generally represents the role of data science at the company. 

Kirill Eremenko: Yeah. It’s a very cool thing that’s going on now. I’ve recently, well maybe a year ago, read a blog somewhere, that before data-driven companies were crushing it, but now it’s the algorithm companies that are crushing it. Like you look at Amazon, they have their recommender system uplifts their revenue by 30%. You’ve got Google with their algorithm for displaying ads. Uber with their matching algorithms. And basically, all of the top companies are using algorithms. And it’s really cool to see a company in the retail space, in the fashion space also following along those lines. And would you say that this is something common that your competitors are also using or is it something a massive innovation that makes Stitch Fix stand out? 

Brad Klingenberg: I think certainly across technologies, the examples you cited at a Google or Amazon, there are many places where algorithms are used in core parts of the business. That’s fairly unusual, certainly in fashion and retail. I think a lot of that has to do with, as much, with the aim of the company, as what the data that the companies have. And so one of the advantages of the Stitch Fix model, which is, Stitch Fix’s personal styling service, a retailer, it helps people find what they love. But not in the traditional sense of shopping. So, we’re not a store where you browse or filter your way to inventory. It’s really a service where a personal stylist actually can make decisions, and we bet on those recommendations by sending them to clients. 

Brad Klingenberg: And as part of that, we close a feedback loop that’s unavailable to many retail models. So I’m sure if you think about the last time you were trying something on in a store, and it didn’t quite fit, or you didn’t quite the style and you left behind in the dressing room, chances are extremely good that nobody learned anything about you or anything about the inventory. In Stitch Fix, part of the magic of the model is our clients are generally very excited to share feedback with us. And so when we send things to clients, we get to learn about what they and what they don’t like. That helps us better serve them, but also better understand how to use inventory to make other clients happy. 

Brad Klingenberg: And so the algorithms we use at Stitch Fix, since the top, this feedback loop that really opens a lot of opportunities to use data, not just in a sense of building a dashboard that is providing the insight into something, but in a very active sense, where we can actually use data to drive to recommendation algorithms at the heart of the business, to make decisions about managing inventory or marketing. And so, I do think it’s a bit of a part of a vanguard data being used in a much more active way, in many different businesses that I’d expect to see continue to play out in the coming decades in other industries as well. 

Kirill Eremenko: Yeah, definitely great that you’re leading the way in that sense. You mentioned a bit about Stitch Fix, about how you’ve closed the feedback loop and that you’re in the fashion and design industry. Could you tell us a bit more about the company? So for somebody who’s never heard of Stitch Fix or never used the service, how does it all work? How does it all play out? Like for instance, if I want to go and buy something, some item of clothing, could you walk us through the customer experience? So you paint a bit of a better picture of how the service works before we dive into more, the algorithms and that side of things? 

Brad Klingenberg: Yeah, absolutely. So Stitch Fix is a personal styling service. The way it works from a client perspective is you come to us and tell us a bit about your preferences. So how much you typically to spend, how you like your clothes to fit, and a variety of things to help us get to better understand your sense of style. And then with that, we’ll actually pick some things to send to you. So, it’s unlike a traditional retailer or even most e-commerce models, we’re actually making choices on the behalf of clients. So, rather than just for yourself, Stitch Fix is actually making the choice of what to send to you, and being a personal stylist, in the truest sense. 

Brad Klingenberg: And so of course, it’s quite important to get that right. And that’s where the focus on using data to better understand the preferences of our clients, and the inventory that they’re going to love becomes so important. And so as a client, you’ll get a shipment that we call a fix, and have a few days to try things on at home. Try it on, with your mirror, with other things that you own. You keep what you love, and send back the rest, and you are able to continuously engage over time in that way. 

Brad Klingenberg: More recently, we’ve also opened up some new ways of engaging that are quite exciting. And so for example, being able to get recommendations for an outfit that goes with things that you’ve purchased with us in the past. And so whereas we’re really exploring more ways to make it an even richer experience, but I think the key distinction from traditional retail is really the company acting as a personal stylist, and us making a really literal bet on our recommendations. 

Kirill Eremenko: So what kind of payment model is… Like for instance, if I like an item, do I need to pay for it after you’ve shipped it to me? Or is it a membership fee that I pay every month? How is this monetized? What is the revenue model for the company? 

Brad Klingenberg: So the way it works is that you, as a client, would pay $20 for effects as a styling fee. And then, if you keep anything from your fix, that’s applied toward what you keep. And then if you keep five things, you actually get a discount beyond that. And so you can look at the styling fee essentially, is only something that you end up paying if you don’t keep anything. So if you keep anything that we send you, it’s applied toward that. And then that model includes both the shipping to you and any returns that you’d to make to the company.
Kirill Eremenko: Okay. Got you. And so that’s what you mean when you say, you close the feedback loop, that by seeing what the client returned, you’re able to adjust your algorithm. So even if we take for instance, the best retailer who takes care of all of their data possible, the only data they actually see is… They don’t see what the client is trying on, but they see what the client is purchasing. And then from there they can say, “All right, so the clients purchased these five pairs of jeans, or this specific pair of jeans. So we can learn from that.” But you can learn from that, and in addition, you get to learn from what they didn’t like. 

Brad Klingenberg: Indeed. So that’s exactly right. Learning why you didn’t like something is often just as useful as learning why you did like something. So for example, if something doesn’t quite fit you because it’s too big or too small, that’s a lot of information about how you to wear things. But beyond the implicit feedback of what people keep and what they choose to return, our clients can also share much richer feedback. So we ask structured questions about, “How do you like the size?” Or the style of things that are sent. And even have clients share freeform texts feedback with us. 

Brad Klingenberg: So part of the model is really this combination of algorithmic decision making, but also then, expert human curation. So every fix that we send is actually curated by a stylist who’s a Stitch Fix employee, and they engage with clients on a variety of unstructured ways. So when you request a fix, you can include a free form requests note. And when you’re providing feedback on things that you’ve received, likewise, you can include freeform text feedback on what you’ve liked, what you haven’t, to help the stylists and also our algorithms get to know you even better. 

Kirill Eremenko: So just to clarify, that means that as a client, my stylist is not just an algorithm, it’s an algorithm and a human being. 

Brad Klingenberg: Yes, absolutely. And so when we’re styling effects, which is our term for choosing things to send to a client, there’s really this combination of recommendation algorithms, and the tools that our stylists use. But ultimately in every fix, every single item that we select is picked by a stylist. And it’s really this human-in-the-loop process, and human-in-the-loop in a very strong sense. So not just having expert humans label things so that algorithms can learn, but actually having an active human decision making in our production loop. 

Kirill Eremenko: I like that term, active human-in-the-loop. Much better than just labeling stuff. That’s very cool. And so I think we’ll get to that in a bit, about how humans and machines work together at Stitch Fix. I think that’s a very important topic. But for now, I want to ask you just to put this into perspective for people who are listening who might even have used the service before, but don’t know the intimate details of the company. Two questions. How many people do you have on your team? And also, how many clients does Stitch Fix serve? 

Brad Klingenberg: So the Algorithms team at Stitch Fix, which is the team that I lead as Chief Algorithms Officer includes a data scientists and platform engineers. And we have, today, over 125 people on the team. And so, a very large organization, or company size. [inaudible 00:18:01] a couple things. I think, both the incredibly important role that the data science plays at the company, and the breadth of applications. And so, happy to chat about that more. And then, to the second part of your question, we have over three million active clients. So [crosstalk 00:18:22] a large population as well. 

Kirill Eremenko: That’s a lot of clients. You have a lot of data. I love this about algorithm-driven companies, that the better the service, the more clients you’ll have. The more clients you’ll have, the more data, meaning the better the service will become. It’s a self-fulfilling prophecy. 

Brad Klingenberg: Oh, absolutely. And I can certainly remember six years ago when I just joined the company, the volume of shipments we were making was much smaller. And that has a real impact on the amount of time it takes to run experiments and the amount of signal that you have to work with. And today, we’re able to run experiments, much more quickly than I’d ever dared hope, years ago. 

Kirill Eremenko: That’s crazy. When you joined the company, how big was the team then? 

Brad Klingenberg: So, the data science team was led by Eric Colson, my predecessor in the Chief Algorithms Officer role. And I joined as one of the first data scientists. And so I think Eric and maybe three other people, depending on how you count, something like that. 

Kirill Eremenko: Wow, that’s crazy. Okay. Very interesting. So that’s the scale of the company. Massive company. 3 million active users. Over 125 people in the Algorithms team. So let’s dive into a little bit. Let’s dissect this, the types of algorithms you use. I think this could be a very cool case study for somebody not necessarily even just in the retail space, but who wants to see how AI and algorithm can drive a company, and it can be at the foundation of a business. How it’s already possible, even at the start of the 2020s. Oh, well, it’s been possible even in the previous decade you’ve been doing for a while. 

Kirill Eremenko: So the first question I would have is, there’s two types of filtering that are quite well known. Collaborative filtering versus content-based filtering. In your algorithms tour, which by the way, I recommend for everybody listening to this, I will link to it in the show notes. It’s algorithms-tour.stitchfix.com amazing. Completely blew my mind. I loved reading through it. Such a volume of information and so diverse. It really explains how this whole thing runs. 

Kirill Eremenko: So I wanted to talk a little bit about some points in this algorithms tour. So collaborative versus content-based filtering. In the tour, it says that you use collaborative filtering plus mixed effects modeling. Could you go into a bit of detail on that? Why and how would you use collaborative filtering plus mixed effects modeling, and what are those things?

Brad Klingenberg: Yeah, absolutely. I think it’s helpful, just as general context, to think about the types of data that we have, when we’re thinking about what to send to people. And broadly speaking, I like to think of there being three categories. So there are things that clients tell us about themselves. And that could be through the initial onboarding survey where clients could tell us the sizes they to wear and their budget, preferences. It can also be things like, our clients are able to rate images in our iOS app, and on the web, that help us understand their style preferences. But generally things we know about our clients. 

Brad Klingenberg: On the other ends, Stitch Fix as a retailer, we buy and hold inventory. And so as part of creating or buying that inventory, we have a lot of information about it. So that could include structured measurements, or even subjective ideas about when you might want to wear this, or different occasions it might be suited for. And then finally, the third category is really feedback. So as we send things to clients, learning about what they like, in a way that helps us better serve them, and helps us get to know them over time. But then also helps us do a better job managing our inventory in a way that creates better client outcomes. 

Brad Klingenberg: And so, in general, in thinking about this recommendation problem, you have those three categories to draw from. And so, as you mentioned, a content-based recommendations is something that might be very focused on the things we know explicitly about our clients, and things are know explicitly about our inventory. And we know that there’s quite a lot to do there. And you mentioned the mixed effects model, highlighted in the algorithms tour, is one example of a type of algorithm, not unlike other applications where you use things matrix factorizations where you’re essentially trying to learn either through a transaction history or through feedback that people have provided, representations of clients and inventory that are useful for making future predictions. So not just something that’s an explicit attribute given by the client, or that we know about our inventory. 

Kirill Eremenko: So for instance, let’s compare it to a very well known company that has an algorithm in its base, Netflix, right? Netflix is the, probably… It is the most valued company in terms of movie production right now. And I think it’s actually been labeled the most valued membership platform in the world. Anyway, the point is, they use filtering, in terms of a content filtering. They look at content-based filtering. They look at, “All right, what movies have you watched, or what shows have you watched? What are the shows similar to those shows to recommend to you?” And then collaborative filtering or… I don’t know what they use. It’s proprietary information, but they could use content-based filtering, plus collaborative filtering. They could look at, “Okay, you’ve watched these shows, and then this other person is similar to you in these certain ways. So they’ve watched those similar shows. What else have they watched? Now we can recommend that too.” 

Kirill Eremenko: Whereas, what you can do is, you can do both those things, but in addition to that, you can take it to the next level. You actually have feedback. Your users tell you implicitly, feedback, saying that by returning items, you understand they don’t those things. Plus, in addition, they might fill in some surveys, or provide your textual information. Maybe get on a call with your stylist. Basically… Or explain attributes about themselves. Basically, give you additional contextual information that other companies such as Netflix wouldn’t have. And you’re able to add that in. And that’s what is called the mixed effects modeling. Am I on the right track here? 

Brad Klingenberg: So the mixed effects modeling refers to a very particular type of statistical model that it is useful, for example, in learning empirically, that a… For example, a particular blouse does better with clients of a certain style preference than another. And so it’s an example of an algorithm that can make those kinds of empirical inferences. One family of algorithm that we’ve used to some success here is something called the factorization machine, which is something closely related to matrix factorization, but it makes it easy to incorporate the types of explicit attributes that you’ve mentioned. 

Brad Klingenberg: And maybe to add, I think one, actually interesting feature of the Stitch Fix recommendation problem that makes it different from recommending movies in Netflix, or music at Pandora or Spotify is, again, that we have this human stylist in the loop. And so in addition to learning through the choices our clients make, and what to keep and what to return, and how they rate things, we get to learn through the way that stylists interact with our recommendations as well. So being able to learn from what a stylist chose to send, but what they didn’t choose to send, that was recommended by an algorithm. And so there’s actually this interesting dual feedback character to the problem. 

Kirill Eremenko: So basically, you’re using pins or images that the clients liked, in order to get additional insights, which they cannot explicitly tell you, and add that into the model. Tell us a bit about that. How does that work? 

Brad Klingenberg: Yeah, absolutely. So one of the best examples of this might be our style shuffle feature in our mobile app and on the web, which lots of people rate different images. And through those ratings, what we’re eliciting from clients is just their reaction to something, and whether they like the style. And through looking at all the many things that they share with us, then we’re able to derive some representations of their preferences that are useful on many different parts of the company. 

Kirill Eremenko: Got you. And let’s go into a little bit of technical details here. I found a very cool example of how you deal with this visual information. Because there’s lots of ways you could transfer visual information into machine information. How does this whole system work, where you create a vector for a specific style, or a specific item even? Like you have vectors representing different items. And then based on the cosine similarity, basically the angle between the vectors, you can tell if two items are similar or not. Is this a common approach to encoding images for further comparison and filtering? Or are there some innovations that you can share with us, that you are able to create along the way? 

Brad Klingenberg: Yeah, it’s a great question. I think… So, certainly, recent years have brought a lot of incredible improvements in computer vision generally. One of the applications that we were just chatting about though, it combines not just trying to process an image, but really understand how a client reacted to it. And their response to it. And so, one interesting aspect of our business is because we have stylists who largely interact with our inventory through software tools that allow them to make accurate recommendations, is we have great image assets of essentially all the inventory that we carry. And by showing that to clients and getting the reactions to it on a tool Style Shuffle, we’re able to learn an embedding or a vector representation of the client preferences for that item, that are really useful. And even simple applications as you noted by looking at cosine similarities. And it’s really the application of those embeddings throughout the business, that are really a differentiated application of that data. 

Brad Klingenberg: So for example, by playing Style Shuffle, we get to learn about your visual style preferences, in a way that leads to the really material improvements in the recommendations that we make, and therefore the outcomes that the clients have, how happy they are with their fixes. And so it’s really a very useful tool that shows up in a lot of places, including helping to pick a stylist for you in the first place. So helping to choose from amongst the stylist, the good style you’re fixed, to find the ones that will be most successful. 

Kirill Eremenko: So that’s also another algorithm to not just pick your clothes but pick a stylist in the first place. Right? 

Brad Klingenberg: Yes, absolutely. 

Kirill Eremenko: Wow. And what are the criteria there?

Brad Klingenberg: Well, there’s a lot of things. I think part of the value proposition for many clients, it’s actually creating a longterm relationship with a stylist who gets to know you over time. And that free form interaction that I mentioned earlier, being able to request things to your stylist, and provide feedback, create [inaudible 00:30:06], in many cases, a real relationship. And so, for many clients, they’re happy to stick with a stylist who’s gotten to know them and really knows what they love, and brings that human touch. Of course, for clients who are getting their first fix, or looking for a new stylist, there’s a question of, “Well how do we find that the stylist who you’re going to like the most?” 

Brad Klingenberg: And one of the important signals that goes into this is really understanding the strengths and weaknesses different stylists have, with respect to different styles. So, we all have our own preferences, and you might be better or worse at styling different types of clients. And so we try, again using this idea of a vector representation, or embedding representation of styles, something we call [lasing 00:30:56] Style, truly use that to understand the types of clients that stylists are likely to be successful with. 

Kirill Eremenko: I thought for a second, you said a vector representations of stylists. That would be funny. 

Brad Klingenberg: Yeah. At least the preferences that they share when picking things for clients. 

Kirill Eremenko: Yeah. Got you. Okay. All right. So you have algorithms to find the right clothes. You have algorithms to match customers to stylists. You probably have NLP to process text, right? 

Brad Klingenberg: Yes. NLP is an interesting thing for us too, because again, the human in the loop nature of the problem, it’s quite difficult, for example, for even with cutting edge NLP, to beat the stylist capacity to see a note from a client that might be something like, “Hey, I’m going on a vacation. Send me something for the beach.” To really understand that intent, it’s hard to beat the human capacity. At the same time, we can do a lot of things with that text, algorithmically. And so we work on, I think, learning from texts, both, algorithmically, but then also through the role that our stylists play in the process. 

Kirill Eremenko: Yeah. And you can pick out some keywords, and help stylists to sift through the text faster, I guess. I read that you also have algorithms to even optimize how your pickers go through your warehouse to pick up the items that need to be sent. Is that true? 

Brad Klingenberg: It is true. So we have a network of warehouses across the country, that we use to fulfill our shipments to clients. And within the walls of the warehouse, there’s a lot of opportunities for optimization, and again, using the data we have, to help make things more efficient. So we have a team focused on what we call operations algorithms that among other things, thinks about how to optimize the operation of our warehouses. 

Kirill Eremenko: Wow. That is very impressive. So basically you use algorithms pretty much for everything. Is there anything in the business you don’t use algorithms for? 

Brad Klingenberg: There might be a couple of small pockets here and there, but I think actually one of the interesting stories of the algorithm that we make at Stitch Fix is just the breadth of engagement it has with the business. So there’s a lot of things we haven’t even mentioned yet. And actually, the fact that we have 125 people on the team, more than anything, reflects the number of different problems that we’re working on. So if you drill down into any particular problem, like our core recommendation algorithms, you’ll find there’s less than 10 people working on those problems. And so the size of the team, more than anything, reflects that it’s really an enormous breadth of applications across the business that the team is working on. 

Kirill Eremenko: Yeah, it’s incredible. How do you even keep your mind on top of all these things as Chief Algorithms Officer? To me, it sounds impossible to deal with all these different algorithms in different parts of the business. And then just keep up to date with what’s going on. 

Brad Klingenberg: I think there’s couple of things that help with that. So one is, I get to spend a lot of time with the team and to understand the problems aren’t working on. And it certainly is a lot, and I certainly can’t keep it all in my short term memory all the time. I think beyond that though, another special feature of the team is really our strong culture of bottom-up innovation. And so the development of better algorithms, or better ways to use data to run the business doesn’t rely on me having a great idea, and then propagating that down the organization to be implemented. But really, rather the opposite. So some people who are closest to working on the business problem really seeing opportunities, exploring new models, and bringing that to life. And so, my role of leader of the team is less somebody directing things and a fine grain from above, and more a like a gardener just trying to make sure conditions are good so that all sorts of different things can grow.

Kirill Eremenko: I love that analogy. Analogy of a leader as a gardener. That’s perfect. Can you give us an example of a recent bottom-up innovation that you think pop to mind? 

Brad Klingenberg: Yeah, absolutely. I think many of the most significant improvements we’ve made over the years have arisen in this way. And I think even the Style Shuffle rating experience that I mentioned, is generating such useful data, was actually born out of just as a side project of one of the data scientists on the team who decided to try something, and prototype something. And it really got traction, has become a real product here. But that’s something that is a great example of an innovation that was not asked for in any executive way. 

Kirill Eremenko: Fantastic. I think this is a good time to make a small plug for your hiring efforts with such a grand team, and growing so fast. To me, it sounds like an amazing place to work, especially when you can drive innovation bottom-up, do experiments, choose what area of business you want to work on, choose the type of work, whether you want to do NLP, you want to do traveling salesman problems, you want to do computer vision. Pretty much all AI is there. You mentioned at the start, before we started the podcast, you’re hiring. Can you tell us a bit about more about that? If somebody is interested, what kind of roles you’re hiring for, and where can they apply? 

Brad Klingenberg: Yeah, absolutely. So I would say in my time here, we have been in a mere permanent state of hiring, and that certainly continues to this day. The company and team continue to grow. We seek people from a variety of different backgrounds. And actually one of the really fun things of working at Stitch Fix is just the variety of backgrounds and skills that people bring. And this is true within Algorithms, but in the broader company as well. And so we have a team that has enormous variety of different backgrounds, from the computer science and statistics, to the sciences like physics, social sciences, psychology, epidemiology. We really run in the gamut. And so, we draw from a lot of different backgrounds and find a lot of value in having that diversity of training. 

Brad Klingenberg: And in general, have a fairly strong preference for people who are able to be generalists. And really, not be too hyper specialized into any particular subset of data science. And a lot of this has to do with the business problems that we solve, and asking a very, very open-ended question like, how should we manage our inventory? How could we do a better job buying things that our clients are going to like? Which are just very open-ended problems that are amenable to techniques from a variety of different fields. 

Kirill Eremenko: Got you. And I’m looking at your careers page. You’ve got algorithms platform engineer, data platform engineer, data scientist, data acquisition algorithm, data science and client recommendation, data science and merchandise. Very, very cool. Most of these roles are in San Francisco. Is your whole team based in San Francisco? 

Brad Klingenberg: It is, yes. So the algorithms team is in our headquarters here in downtown San Francisco. 

Kirill Eremenko: Okay, fantastic. All right. So if anyone’s looking for a cool job, this is your opportunity. Check out stitchfix.com/careers/jobs. Okay. So the other thing I wanted to ask you is, it’s very evident that, and I think I read this somewhere, that algorithms are a part of the business model at Stitch Fix. Just by the way you do business, this is a just like a bonus, not just like a cherry on top of the cake. This is the cake of Stitch Fix. What can other companies learn that? We’ve got about 10% of our audience, executives and leaders and business owners. What can other companies learn from that? What takeaways would you recommend, or you could share from having the algorithms as a part of the business model? 

Brad Klingenberg: So I think for Stitch Fix, for a very long time, and much credit to Katrina Lake, our founder and CEO, for foreseeing the opportunity. Data and algorithms have been a key part of our differentiation strategy. And really, to be able to really predict what people are going to like, and to be able to give it to them in a fun and convenient way. And going back to the framing of making a bet on the recommendations, that’s just really important that we’re good at winning that bet. As a retailer, our strategy is not to carry the largest selection of things, or to carry things at the very lowest price, or to deliver them faster than other retailers, but to find things that you love. And so it was really that personalization, and ability to understand and learn from you, is at the core of the business. And so, I think to your question, we really have data and data science at the heart of the business, and it’s part of the core of the strategy. It’s not a second order optimization. But really, a first order existential strategy for the business. 

Kirill Eremenko: Got you. And so, does that make it easier for the business to grow and to understand its users? What’s the main advantage of having that data science as the core? 

Brad Klingenberg: I think in businesses where you first have the data, so either the feedback loops exist, or there are ways to learn from your customers or other stakeholders, you are able to really take full advantage of everything that you know. And even beyond using the algorithms to make those decisions, having a culture that is very data-oriented and empirical helps to make good decisions about how to think about uncertainty, how to think about measurement. And it really, in general, I would say, just brings a more scientific approach to driving a business forward. And so where the business model supports it, I think, making investments in data problems that are really of first order importance to the businesses is the right way to go. 

Kirill Eremenko: Got you. So increases certainty and predictability of your business. What is the main challenge of having algorithms as part of the business model, or having data science at the core? We’ve seen the advantages. They’re very clear. It’s obvious how now, you’ve grown to this level, how much more streamlined things are, and how different things are to a competitor that wouldn’t have that in a business structure. But what would you say is the one or two biggest challenges you’ve had along the way? 

Brad Klingenberg: So we mentioned that we’re hiring. I think data science is just in a wonderful period of rapid growth. And you’re finding applications across many different industries, which means that as a data scientist, everybody has many different choices of what to work on, and where to work. And so, one thing we’re always working on is bringing very talented people into the company. And that’s certainly a big part of what I think about. I think another challenge of using data and algorithms in this more active sense that we discussed earlier, is trying to figure out what is the best way to combine human judgment in algorithm decision making? And when do you bet on an algorithm, and when do you bet on human judgment? And there’s no fixed optimal answer to that. 

Brad Klingenberg: And I think we’re on something of a journey, of really understanding what is that optimal boundary between your algorithm making or human decision making? And what is really, the right way for humans and machines to work together in a way that enables folks at such Stitch Fix to do things that they wouldn’t otherwise be able to do? 

Kirill Eremenko: Let’s talk a little more about that. This is a really cool topic, which I identified before I started the podcast. One of the interesting themes at Stitch Fix, the humans plus machines. And if we think about it, companies like… Again, taking the case of Netflix, I imagine that all of that, the whole recommendation process can be done entirely by a machine. And if we take Amazon, especially there, with the volume of data that’s coming in, and number of transactions all the time, again, that’s most likely being done fully by machines. So what is your view on the situation of Stitch Fix? Do you think that the service that you provide could ever be done entirely by machines, or are humans an integral part of this whole setup in this whole service that you have created? 

Brad Klingenberg: So this is a question that I’m inevitably asked, anytime I speak at public events, or give interviews. I think my view is that the service would not be nearly as good without the role of the human stylist in helping to pick things for our clients. Nor is the work that we do on building algorithms. It doesn’t have the aim of actually automating away the human component. I think the way that we think about things is really how to combine humans and machines in a way that that’s better than either alone. And so, the goal is really not just automate things away, but to equip stylists with capabilities they wouldn’t otherwise have. 

Brad Klingenberg: And I think, we talked about earlier, the possibilities for interacting with clients where you can exchange free form text, you can exchange image data, those are all things you can work on algorithmically. And there’s really exciting advances, especially in recent years. But it is quite difficult, still, to match the human part for many of these things. And I think really, the human in the loop is a really integral part of our strategy for solving these problems. And I would expect it to stay that way for a very long time. 

Kirill Eremenko: Fantastic. And yeah, active human loop, as we discussed at the start, very cool. How do your stylists… What do your stylists say about this experience for them? How do they find it, working with artificial intelligence hand-in-hand? 

Brad Klingenberg: It’s certainly a core part of their experience. And I think you can really, in some sense, think of the process of picking items to send to clients as being something of a dance between algorithms and stylists. And so, we learn a lot from stylists. And so as I mentioned, the feedback loops we use to improve our algorithms are not just how clients respond to things, and how they rate them, which of course is very important, but also, the decisions our stylists are making. And if we make a change to an algorithm, does that make stylists have to search more for the things that they want to send clients? Or does it make them make decisions more easily? And really, taking a view of styling as a combined system that involves both algorithmic decision making and human decision making. 

Brad Klingenberg: And so we pay a lot of attention to stylists, or interacting with algorithms. In both, quantitative ways, so the stylists can tell us about the recommendations that they’re seeing, also in qualitative ways. So just spending time with stylists and the leaders of the styling organization to really understand their experience with the tools, but also opportunities we have to always be making things better. 

Kirill Eremenko: Got you. And you mentioned that like three million people, massive market, a huge market presence. We spoke briefly before the podcast, that you’re expanding, or you’ve expanded to the UK. Tell us a bit about that. Is that your first international expansion? And what does it look like now, in terms of growing your global presence? 

Brad Klingenberg: It is. So, yeah. We were quite excited to expand into the UK, and set up a business center. So that’s our first international business. The past few years have been a story of expansion in the US market. So starting as a styling service just for women and then expanding to men’s and kids and even different size profiles, so were petite and plus size within women’s, for example. And so, international expansion is similar, and a natural opportunity that we have. And yeah, we’re excited about growing our UK business. 

Kirill Eremenko: Fantastic. And what does that mean for the algorithms team? Are you going to stay in San Francisco? Or are you going to start hiring in Europe as well? 

Brad Klingenberg: So for now, everybody’s still in San Francisco though we do spend quite a bit of time with our colleagues in London, both virtually and also by visits to London. I always love to go when I get a chance. And yeah, it’s an exciting problem to work on. It’s a different it’s a different market of clients that we have. Largely different inventory. And when we launched of course, there was a bit of a cold start problem. You have all new inventory, all new clients, and starting a new business. And of course, we have the benefit of everything we know about our US business and all the algorithmic approaches that work there. But it’s really, I think, as time goes on, the feedback starts to accumulate and you can… There’s a flywheel that that kicks in, that helps drive things algorithmically. 

Kirill Eremenko: That’s very cool. And if you do, at some point, expand your algorithms team to Europe as well, then you’ll have access to a whole new talent pool European professionals and data center. 

Brad Klingenberg: Absolutely. Yeah. I think that is a very compelling consideration. And then thinking about hiring folks, as they’ve very large pools of talent that exists. Of course, not just in US but Europe as well. 

Kirill Eremenko: Fantastic. And what does the future look for personalization, using artificial intelligence? Because that’s predominantly the core part of the algorithm, or output of the work with algorithms is personalization and making it very user-specific, all these recommendations. And that doesn’t necessarily have to be just applied in the space of fashion, design, tailored clothing. It can be applied in lots of different industries. From your experience, extrapolating just, in general, onto personalization, what do you think the future looks in terms of AI developments and data science playing a huge part in this space? 

Brad Klingenberg: So I think from a consumer perspective, in retail, but in many other consumer services or products, we’re in an era of just overwhelming choice. And you see this with retailers competing to carry the most broad inventories. And Stitch Fix, I think, is a compelling counter position to that. The service is really a personalization and making it easy to find things that you love. Not to show you 1,000 pairs of jeans that you could choose from, but actually just to send you the pair that fits just right, and that you love. And I think, as you know, the value proposition there, while quite well suited to fashion and apparel, need to be limited there. And so I think, really, any industry where there’s this overwhelming burden of choice and selection and having to do a lot of work to find things that you love, is an opportunity for personalization, which of course, there’s a very successful one driven by data to actually help people find things that they love and relieve them of the burden of doing the curation themselves. 

Kirill Eremenko: I love it. I love that getting rid of that whole choice paralysis problem. I think I’ve experienced that. You go in. There’s so many things to choose from. Before, and I’ll tell you, in my life, like even 20 or 15 years ago, you go into a shop and there’s only one thing that suits your measurements, or what you wanted. You buy it, then you’re happy. Okay, it almost fit what I wanted. But now you go into shop, there’s at least 10 things that would suit you, and they’re all different colors, shapes, sizes and so on. They’re all perfect for you. And then you buy one of them, and you walk out feeling miserable because you are thinking of the missed opportunity of buying all these other things, and how much better they would have been. So it’s like you’re bring us back to that feeling of satisfaction from your purchases, because you’re taking the choice paralysis out of the equation. 

Brad Klingenberg: Absolutely. 

Kirill Eremenko: Yeah. It sounds very exciting. But then I’ve got an interesting, more philosophical question then. Do you think that… Is your work on algorithms, or these kinds of personalization algorithms, will, or have the potential of driving the world to a dystopian future where we no longer can make choices for ourselves? All we do is, “I’ll wait for an algorithm to make a choice. What are we going to eat? What are we going to drink? What movie are we going to watch? Where are we going to go for holiday? What are we going to wear?” Basically, in that sense it’s like taking the human, not just out of the workforce, but rather taking the human out of their life. And it sounds a little bit scary, if you think about it that way. Maybe a bit farfetched, but what are your thoughts on that? 

Brad Klingenberg: Yeah, I wouldn’t worry too much about that quite yet. I think the general trends in so many settings is, again, for this ever increasing selection, and just broader and broader choices for people to make. And I think there’s certainly both risk and opportunity in narrowing what you’re exposed to, too dramatically. But I think the prospect of that becoming so prevalent that it has dystopian implications is probably pretty remote, in my opinion. 

Kirill Eremenko: Probably in the meantime, it’s really great, what you’re doing. And another consideration is that it helps reduce waste, right? When there’s so much on offer, things that don’t get purchased and they go out of fashion, they get thrown away. But when you can personalize to the extent where you know your customers really well, you can reduce the inventory size. And that means the less waste, less carbon emissions, less impact on the environment, things like that. So even from that standpoint, this is the way to go, I think. 

Brad Klingenberg: Yeah. It’s a wonderful aspect of this model, where we’re able to run a very efficient inventory, largely because of how much we know about our clients. In addition to this, the general structure of the model. And something we haven’t talked about too much today is there’s quite a lot of effort in the algorithms team, in thinking about how to just make better decisions about what to buy, and how to manage inventory. And you can think about it in some sense as this giant portfolio optimization problem. How do we build an assortment of inventory that’s going to delight our millions of clients, and do so efficiently? And the power of personalization here is really to enable us to do a really good job, buying things we know our clients are going to like, which, to your point, in addition to being more efficient from a business perspective, it’s also more efficient and less wasteful than other models. 

Kirill Eremenko: Totally, totally. Well, it’s been really cool dissecting all of this and diving into some of the technical aspects and also the business implications, ethical implications. Really, really cool. But we’re slowly coming to the end of the podcast, and Brad I wanted to ask you, do you have any final parting thoughts for data scientists that are listening to this, who want to progress their career, or want to maybe get into the space of data science, who are by now, I hope, are very excited about personalization and what AI can do in this space? Any thoughts or wishes for them?

Brad Klingenberg: I think it’s a tremendously exciting time to be in a career in data science. And one I… Stitch Fix, and our models are extremely interesting. But I think, even in a larger sense, I think we’re part of a transformation that will play out over a lot of different industries as we really bring data and algorithms and just scientific approaches in general, to running and building businesses. And so I think the opportunities will only increase, and keep an open eye out for hard and interesting problems to work on. 

Kirill Eremenko: Fantastic. Fantastic. Thank you so much Brad for coming to the show. Very exciting conversation. I really enjoyed chatting with you. And before I let you go, can you give us some places, online, where people can follow you, connect with you, maybe follow Stitch Fix? What are the best places to get more information and stay in this loop? 

Brad Klingenberg: Absolutely. I think more than anything, I’d recommend checking out our blog, multi-threaded, which is a technical blog of the company that includes the algorithms tour that you mentioned, but then also a pretty regular cadence of really interesting blog posts about the way that we use algorithms at Stitch Fix. 

Kirill Eremenko: Got you. So that’s multithreaded.stitchfix.com. Very exciting. And one more question I had for you before we finish up, what is a book you can recommend to our listeners for them to further their careers or just in general their lives? 

Brad Klingenberg: Yeah, great question. I think one really interesting area of machine learning research these days is causal inference. And this is something that there’s really active research there. And I think, for folks who are interested in learning a bit about that, but might not have encountered it in their training or experience, there’s a pretty general audience book called Mostly Harmless Econometrics, that I think is a fun introduction to the subject. And a little more broadly for life, I’ve really enjoyed recently reading a book called Origins by Lewis Dartnell, and which was about the role of geology in shaping human history. And I think it’s a fun story of how we got to where we are today. 

Kirill Eremenko: Very cool. Have you read Sapiens by Yuval Noah Harari? 

Brad Klingenberg: I have. And it’s in the same genre of large tellings of the human story over thousands of years. 

Kirill Eremenko: Yeah, it’s really cool. And there’s another one, Silk Roads, Peter Frankopan. That’s another really good one. 

Brad Klingenberg: Yup. One of my favorite genres. 

Kirill Eremenko: Yeah. Very cool. Okay, thank you. So the recommendations are Mostly Harmless Econometrics, and Origins. On that note, Brad, once again, thank you so much for coming on the show, spending some of your time with us. I’m sure this is going to be very inspiring to lots of data scientists out there. 

Brad Klingenberg: Thank you. It’s a pleasure. 

Kirill Eremenko: So there you have it, everybody. I appreciate you being on this show with us today, and hearing from Brad Klingenberg from Stitch Fix. I hope you enjoyed this episode as much as I did, and got value out of those amazing insights that Brad was sharing with us. My personal favorite part about our conversation today was the sheer volume and diversity of applications of algorithms at Stitch Fix. This is really cool to see how a company is using algorithms in absolutely everything they do, and at the same time, they are not edging out humans. They are finding ways for algorithms to coexist with humans and they’re getting the advantage of having both working hand-in-hand. 

Kirill Eremenko: I’m really impressed at how Stitch Fix is, in my view, leading the path of becoming an algorithms-driven company. They’re not just using it for one application to set themselves aside from the competition. They’re using it across the board. They actually have an algorithms team and the Chief Algorithms Officer. How cool is that? Totally. Totally love this conversation, and I’m sure you have your personal favorite takeaway from today as well. 

Kirill Eremenko: And as usual, you can find the show notes for this episode at www.superdatascience.com/349. That’s www.superdatascience.com/349. There, you’ll find the transcript for this episode, as well as any materials we mentioned, including the careers page for Stitch Fix. I highly recommend checking out this company, especially if you’re in San Francisco. This could be a fantastic place and sounds like a fantastic place to work at. They’re hiring. So check it out. It’s data science of all kinds in stores, whatever you like. I’m sure you’ll be able to find it at Stitch Fix.

Kirill Eremenko: Also in the show notes, you will find a link to the algorithms tour that I mentioned. Also very cool read, something to browse through. It will probably take you more than a day to get through all of it. It’s very well-written, highly animated, very insightful. Highly recommend checking it out. And finally, you’ll also find the blog of Stitch Fix that Brad mentioned. That could be a very useful read for you as well. So lots of materials there. The link is www.superdatascience.com/349. 

Kirill Eremenko: And one ask I have for you today, if you enjoyed this episode, share it with somebody. Share it with somebody who is interested in algorithms, who is maybe in the space of personalization, or online services, so they can get an idea of what’s happening in this space. If you know any data scientists, machine learning engineers who are passionate about algorithms, this could help them build their career structure. And also if you know any business owners who might be looking to build an algorithms-driven, or at least data-driven company, Stitch Fix is a great testament, a great example to strive for. And I’m sure they would find it useful. Very easy to share. Just send them the link, www.superdatascience.com/349. And on that note, we’re going to wrap up for today, and I look forward to seeing you back here next time. Until then, happy analyzing.