Kirill: This is episode number 39 with Director of Data Science at VSCO Ruben Kogel.
(background music plays)
Welcome to the SuperDataScience podcast. My name is Kirill Eremenko, data science coach and lifestyle entrepreneur. And each week we bring you inspiring people and ideas to help you build your successful career in data science. Thanks for being here today and now let’s make the complex simple.
(background music plays)
Hello everyone. Today we’ve got the legend, Ruben Kogel, returning for podcast session number 2. Ruben was my very first guest on the SuperDataScience show, back in August/September 2016, and since then it’s been about 6 months, and guess what. His podcacst, his session, has been downloaded and listened to 10,000 times. Over 10,000 people have listened to his session. How incredible is that? Very excited about all of the knowledge that people gained from that one episode that we recorded back in 2016.
So it’s going to be very exciting to catch up today again, and we had a fantastic chat. We talked about many different things, and specifically here is a couple of them. So Ruben joined the team at VSCO and he built up a data science capability from the ground up. And he mentions the three points that he focused on when building the capability, so this will be very valuable to those of you who are looking forward towards more senior roles, where you’ll be in charge of some elements of the data science inside the organisation.
Also, we talk in detail about what Ruben is looking for in people who he is hiring for VSCO. Ruben right now is hiring data scientists for a team in VSCO and he explains the actual skillsets and mentalities that he’s looking for in people he is hiring. And apart from all that, of course, there’s lots and lots of other value that we’ve discussed on this podcast, and I just want to reiterate again, Ruben is hiring, so at the very end of the podcast, we speak about how you can get an opportunity. And VSCO is actually in Oakland, San Francisco, and I know that a lot of you guys are in the San Francisco Bay Area, and a huge number of you are in California.
So if you are interested in a data science position at a company with an amazing culture with Ruben as your mentor, this is an opportunity to die for. So if you’re interested in that, if you’re open to new opportunities, even if you’re not, I would highly recommend checking out their job description. You can get it if you go to Ruben’s LinkedIn or the show notes. You’ll be able to see the link to it, but in short, it’s bit.ly/VSCOdatajob. So definitely check that out. I think that there’s going to be so much interest in this opportunity, it’s going to be a first come first served basis. Definitely make sure to jump on top of this.
And on that note, without further ado, I bring to you the legend, Ruben Kogel, Director of Data Science at VSCO.
(background music plays)
Welcome everybody to the SuperDataScience podcast, and today, by popular demand, I have our favorite guest Ruben Kogel back on the show. Hi Ruben, how are you today?
Ruben: I’m doing great. It’s great to be back. Thank you for having me again.
Kirill: Awesome. What an intro, right?
Ruben: Yeah!
Kirill: And your podcast, 10,000 views. It’s mind-blowing. 10,000 downloads all over the world, people listening, watching. How do you feel about that?
Ruben: I’m kind of shocked! I always think like it’s not real! Because it’s just like you and I recording this podcast just a few months ago, so it’s kind of weird thinking that it had that reach. But it’s really great and amazing. I think it is also a testament to how good a job you did at marketing the podcast and reaching out to your audience, so really great.
Kirill: Thanks man, thanks. But the thing that I think is — you didn’t tell me before the previous podcast, but you were actually a radio host yourself! You gave me a little hint about it afterwards, and I looked you up, and you — everybody listening, right away go and run and check out Ruben’s old podcast, which is called “All That Swing”, right?
Ruben: Yeah, “All That Swing”. Yeah, it was a short-lived podcast!
Kirill: But I listened to it, I couldn’t stop listening. It’s so fun, you mixing the music and so on. And no wonder that podcast started out well, because it was my first session and yet you had all this experience! So it’s a testament to you, you drove it home, man.
Ruben: (Laughs) Thanks. You want to hear a funny story? When I did this podcast, I had zero experience doing a radio show, a podcast, and I was really doing it out of a basement on some radio somewhere. I didn’t think anyone was listening to it. And one day someone calls in, and we had like this old phone ringing, so you had to wrangle and get the phone, and the person is like, “I really liked the piece of music you just played. I like the podcast, but can you stop saying ‘um’ in between every sentence?” It’s the kind of thing you don’t realize when you talk on the radio, on a podcast, but all those fillers, they are very, very visible and people can hear them on the other end.
Kirill: Yeah. So how did you get rid of them?
Ruben: I don’t think I did. (Laughs)
Kirill: (Laughs) Okay, gotcha. You took that feedback on board, totally took it on board.
Ruben: You know, I did a lot of editing. That’s what I did.
Kirill: Yeah. I know what you’re saying. Oh, the editing, I remember the days. When you’re sitting at editing, you’re thinking “It’s probably a good idea to actually improve my skills so I spend less time editing.”
Ruben: Exactly, yes. (Laughs) It’s definitely a good experience.
Kirill: Yeah, totally. And it’s good even for data science, in terms of presentation later on, when you’re talking to people and presenting data stuff. Like, any kind of public speaking, whether it’s podcast or anything else, I think it’s a good experience.
Ruben: Yeah, totally.
Kirill: Speaking of data science, after the last podcast—it was very interesting. You were at Udemy and we recorded the podcast. And before I could release it, you had already left Udemy, so I had to put like a little disclaimer on the page that “Actually, Ruben is no longer with Udemy.” And that’s a testament to how popular and valuable data scientists are. So tell us a little bit about where you went and what have you been up to for the past six months?
Ruben: Yeah. Shortly after recording our podcast, I accepted a position at VSCO. VSCO is a platform or mobile app for editing. It’s very popular amongst people who post photos on Instagram. They use it to edit photos because it has a really nice collection of presets and editing tools. But it’s more than that. It’s also a community of expression. We have a lot of professionals, so professional photographers posting photos on VSCO, discovering other people. It’s really a great, great community of photographers that we are building. I actually knew the tool and the app from doing some casual photography and I was really attracted to the challenge of starting a new data team and really building the data function at VSCO, not just pulling data and running analysis, but actually thinking around what kind of data do we need, how do we structure the data, what is the best way to look at it, how can I have the most impact on the product decision, on the strategy of the company. So it’s been a very interesting challenge.
Kirill: Oh, that’s really cool. Like, building it from the ground up, when you come in and there’s pretty much zero, or a little bit, but you have full flexibility and full control. Is that right?
Ruben: Yeah, there was already data and tools in place, but what was missing is a sense of direction and also we were at a crossroad where we were using too many vendors, too many tools, everything was competing, people were looking at different dashboards, different vendors to get their information. We just needed to rationalize a little bit how we were looking at things and also we needed to put in place a structure that would be scalable with the audience and the future of the app. So even though there were a lot of tools and great things that had been put in place, when I came in I had this mandate to actually take a look at everything, at all the vendors, all the tools, all the data that we had and decide which one we kept, which one we got rid of, how we would structure the data and what would be our 2 or 3-year strategy for collecting, storing and utilizing data.
Kirill: Okay. That’s really cool. So how did you go about that challenge? What was your thinking process?
Ruben: The way I went about it was I talked to a lot of people. I set about to reach out or ask people to connect me with other people who worked at similar companies that had faced similar challenges and started just talking to them. And in the process of talking with them, some themes started to emerge, and some tools, and I started to form some opinions about what was better than other solutions. So, to give you more concrete examples, I spoke with the Head of Engineering at Slack, Head of Data Analytics at Instacart, people at Pandora, people at Twitter. I also started reading a lot of blog posts and it helped me get my thoughts around “Do we need to go with like a full-on data lay, put everything in Hadoop and then use Hive or Pig to look at the data and build dashboards on top of it? Should we use a solution like Tableau, which seems to be popular with larger organizations? Should we maybe use Redshift? Should we go into Google BigQuery?”
So all those questions came and I had to sort of organize like, “Okay, why do we need to have these different solutions? What are the needs they are serving and what is the best architecture that will let us address all of the needs, that is both cost-effective but also scalable?”
Kirill: Wow, that sounds like a really cool thing. I’m looking forward to digging into that right now because I think after a lot of the podcast sessions that we’ve had on SuperDataScience, people have slowly started to build up a good overview, understanding of all the tools that exist out there, and all the types of data science that exist. I think this could be like a good summary of how to put it all together. So, you spoke to all those people, you found out what they do, what kind of tools they use in that and what their strategic directions are. And how did you then assess your situation and what is relevant to you and then take those pieces of advice and implement them in your organization?
Ruben: Yeah, that’s a good question. The other piece I only alluded to was that I had to look at the needs in my organization and the fact that I was a very small data team. I was basically a data team of one, so it was going to be impossible to provide the entire organization with on-going analysis. I had to find a solution that was more scalable.
And what I realize is that for companies like us, that have a mobile app, or have like a platform that is consumer facing, a lot of the analysis comes down to recording events, recording taps, clicks, people going from one screen to the next and counting how many people did a particular action in a particular week in the U.S. versus not the U.S., on iOS, Android, etc. And for this type of analysis, there were already standalone solutions that existed and that were doing a fairly good job of it and also enabled anyone in the company to look at the data without being an expert in SQL—in fact, without knowing SQL at all.
And so I realized that there were really different needs in the company. There was this need, especially for product people, designers, engineers, to have a quick look at the data, knowing over time how many people have published a photo, how many people have republished a photo, dice it maybe by country, operating system, and having that tool that let them explore the data very quickly and draw time series was extremely valuable. And if we were to rebuild that tool from the ground up, it would take enormous effort and resources and we wouldn’t even be able to match the kind of simplicity and latency that this tool offered.
Clearly, there was a need here that was at the moment filled by a tool called Mixpanel. We can talk more about Mixpanel and the alternatives, but there was this need that had been identified specifically for product engineer people. There was a second need around creating dashboards and reporting tools that were very clean and that sort of conveyed a sense of truth, because oftentimes you come into these organizations, and people have been looking at data from different ways and it’s not normalized, so the CEO might be seeing something, the COO will see something else, the analyst will see something else entirely. So having a place where people can agree this is the truth, this is what’s happening on our app with our audience was extremely important. So that talks to the dashboards reporting piece.
And the third part was around having a place that let analysts and data scientists do deeper analysis. That is the concept of a data warehouse. That’s the concept of you put all the data in one place, all the data is cleaned up, is verified, and that led people who manipulate SQL to do deeper analysis, do maybe some clustering or things that are deeper than just doing a time series or reporting. So, once I had identified the three needs – the product event analytics on one hand, dashboarding, and deeper data analysis – I think it was a matter of choosing the right technology and the right tools, making sure it was both cost-effective and scalable.
Kirill: Yeah. Wow! That’s a really cool overview. I like what you said that in an organization like this, you need to really think about how you’re collecting the data. You need to set up the right data points for event analytics or for other things that you want to know what’s been happening with your app, how are people interacting with it. So, you set up those data points, and then you also look at the different needs in your company. You spoke about three: you’ve got the product engineers, you’ve got the dashboards, and you’ve got the deeper analytics for—
Ruben: —the analysts.
Kirill: Yeah. So, is it like many different people, or does everybody have access to all of the data? Is it one team predominantly working on all of these things? For instance, product engineers, I understand they would have their own access. But for instance, dashboards and deeper analytics—is that one team, is that your team working on it? Or do you have separate teams for each one of these three points that you identified?
Ruben: In terms of building those tools and data access, it’s actually different processes. The product analytics suite—we use Mixpanel, and Mixpanel basically hooks up to your mobile app and you just have to set up a hook for them and they automatically download all of the event, clean it up, augment it with location, add in some anonymous, unique ID, and put it in a very high-performance database and put a user interface on top of it to let anyone ask questions without writing in SQL.
In that particular case, we are buying a service from this vendor, it’s a very high-performance service, and you’d be amazed. Like, running a query on billions of rows typically takes a second or two, so it’s a very, very optimized database, they actually run it directly in C. It’s not even SQL, it’s not Java, it’s like directly in C when you enter some command. So, it’s really optimized both for data reliability and data performance.
Kirill: Okay, fantastic. How many users do you have so that we know what numbers we’re talking about?
Ruben: On a monthly basis, we have an audience of about 40 million people worldwide, and basically we have a billion events flowing every day. And because we like to instrument everything, we will record any time someone goes onto one screen, tap on a button, maybe do a quick view on an image, republish the image, do an editing, what kind of editing they did, what were the parameters, the sliders, everything. We have like billions and billions of events flowing, so in order to manipulate that, you need a very high performance data pipeline and database system.
Mixpanel deals with all of that as a service. We don’t have to worry about any of it as long as the events are defined by our engineering team in the app, so they hardcode the events. As long as that happens, Mixpanel can collect all of it at the end of the app. They use an SDK and they organize, clean up everything, and make it accessible.
Kirill: Gotcha. That’s very interesting that you mentioned how you collect all the data. Sometimes I hear comments from data scientists that there’s too much data, there’s too much data in the world, in their organization, and they’re getting overwhelmed with it. To be fair, not all data is useful. You can’t extract information, you can’t get insights from every single piece of data that you have. Most of the time you have to be very selective and only some of the data is going to tell you some stories. What is your view? Is it efficient for you guys to collect all this data and then go through it and actually pick out only the important stuff? For somebody else, would you recommend to instead just collect the data that is actually necessary? Why did you guys take this approach of collecting absolutely everything?
Ruben: There are two reasons to collecting all of the events we do. And by the way, we don’t collect absolutely everything. There are things that we don’t collect but we do err on the side of collecting more than less. So, there are two reasons to that. One is, oftentimes you have product managers that are launching features and they are interested in understanding how users interact with a particular feature.
To give you an example, let’s say that we just launched a subscription product, so we added a little tile on our apps so that people can click on it, they get brought into the store, and then in the store they can browse different products, including the subscription. So we are interested in following exactly what people are doing. We want to know how many people saw the tile, clicked on it, got into the store, viewed the product that is a subscription, and then eventually purchased it.
Because once you have that entire view, you can start optimizing the flow, you can start finding maybe there’s a problem somewhere and that’s why your conversions are not good. So, it is important that you really have access to all of the data so you know exactly what’s going on and you know exactly how to optimize your app to maximize engagement and revenue. That being said, you may not need to record everything at all time. There are events that are only important to record for a certain period of time when you’re trying to optimize things and after that you can stop recording them.
But it’s easier to over-instrument than under-instrument because what’s costly is not the instrumentation itself, what’s costly is then the storage and parsing through the data. But the storage itself, you can make a decision later on of “We are keeping this data and we are not keeping this data.” And I think that’s where the data scientists come in because they have an intuition of what is important and what isn’t important and they can parse the data and sort of separate the two.
Kirill: Gotcha. Very, very cool. That’s some good advice, just record as much as you can and then remove some of it. Can you walk us through a recent example, out of what you can disclose, of how you went about in a project? I think in a situation like this, with all this data, it’s very important to ask the right questions, right? If you have so much data and then you don’t know exactly which question you are asking, you can get very lost. So can you walk us through a project and start off by how the person or whoever was requesting this project, how you guys went about asking the right questions and then solving it?
Ruben: Sure. A project that I completed with something was looking at people who converted for a subscription. There was a general question of “Who are the people who are buying subscriptions and are they behaving differently after buying a subscription than they were beforehand?” And the question came to me exactly this way, so very general, and it was on me to start engaging the conversation and start specifying like, “Okay, what exactly do we mean by ‘Who are those people?’ Why are we interested in this question?” Basically, it was a question that came from one of the investors that got transmitted through our CEO.
And basically I was trying to understand what exactly we’re trying to get out of it to help me orient the analysis. You know, it quickly became clear that the question was really a marketing question. We have all these millions of users that are potential buyers, but we know they are not all as likely to purchase, so we’d like to know who’s more likely to purchase because there are two things: one, that will tell us what is the size of the opportunity; and two, that will tell us who should we market this product to to be more effective in our messaging and be more effective with our marketing dollars. So once that was clarified—
Kirill: Sorry, if I could interrupt here — what were the other options? It’s a marketing question. What were the other options that it could have been, just so we’re up to speed? What were you pondering? What decisions were you making?
Ruben: I think it could have been—you know, we just want to know, generally speaking, what those people look like in terms of maybe their behaviours in the app, or it could have been a question of what do those people look like in and of itself, like descriptive questions. You know, “We have all these people buying. Tell us a little bit more about them.” But instead, I took it as comparative questions. “We have all these people buying. What makes them different from the other people and what makes them more likely to buy?”
Kirill: Okay, gotcha. So a comparative question, as in an absolute question. For instance, when you were saying that, it came to my mind that you could have been asked that question to identify what they look like, not because you want to sell to them, but because once they’re in there you want to better service them, you want to maybe identify which products they like once they’re in the subscription membership. But instead, as you say, this indeed is a marketing question, so two different types of questions.
Ruben: Yeah, exactly.
Kirill: Sorry I interrupted you there.
Ruben: So, once that was clarified, I think it became fairly straightforward because I know there are only a limited number of actions that can predict that someone is more engaged and is more likely to purchase the subscription, and all I did really was to collect all of those actions into one big table and then run some sort of analysis. It was not even a fancy statistical analysis, it was just a propensity analysis.
So what I did is I said to myself, “Well, the things that are predictive of whether someone will buy a subscription probably are whether they’ve purchased something in the past and how much of it they have. Have they been active in the last 30 days, and how active have they been. What country they are in, because that might affect their purchasing power. Whether they are iOS or Android. So the list is really not that long, and once you have all of that, it’s really a matter of doing the right comparison.
What I find was the most challenging piece of the analysis was not collecting the data or knowing which event to look at, but more like doing an apple to apple comparison because, for example, you might have a group of people who purchased the subscription in February, but out of that group, there are people who sign up to VSCO in February. So they have no history before February.
You also have people who were completely inactive in the 30 days prior and somehow came back to VSCO because they saw an ad on Facebook or Google, so they randomly came back. You had to be very careful because you can’t compare the people who just signed up in February and bought a subscription with people who were active in January and didn’t buy the subscription. So I think it’s important to establish the right comparison here, which in this case was, “Let’s take a segment of people who were active in the 30 days prior to February and let’s look at the people who did purchase and the people who did not purchase.” Because then we have like a fixed base of the people who did that same thing, they looked exactly the same in January, but somehow one portion of them did purchase and one portion of them did not purchase. So establishing that baseline is very important, and once you have that, it makes it a lot easier to compare the two groups.
Kirill: Yeah, very cool. Love it. And you mentioned you do some propensity modelling on that. Why did you choose propensity modelling over logistic regression or other types of classification techniques?
Ruben: Because I think I didn’t want to overwhelm our team with unnecessary complexity. I didn’t think that the goal here was to have one formula that would predict the probability of someone buying. I thought more important was to surface trends and surface propensity according to different dimensions. So to be specific, the kind of thing that we looked at is, “Does the fact that you are on iOS make you more likely to buy a subscription, and if yes, by how much?” It doesn’t matter that you’re on iOS and you live in Australia and maybe you’ve used like three paid presets in the last 30 days. Altogether increase gives you a likelihood of 30% or whatever it is. What matters is, on each individual dimension, how much more likely you are to convert because then we can start narrowing down our marketing effort to those people. And we can also separate out the effect of all these different variables. So, in a way, it’s a much more natural and clear analysis to communicate to the marketing manager rather than coming up with a long logistic regression forming at the end.
Kirill: Yeah, got it. And that’s been a very interesting kind of trend that I’ve seen in this podcast. A lot of the time, people like yourself who have the skills, who have the access to the tools and methodologies, you sometimes (but not always) choose to use a simpler approach simply because you have a further agenda after your analysis is complete. You’re not just delivering a consultant report. You’re part of a team. You’re building something. You want the company to succeed. You want the marketing manager to understand what’s going on rather than having this convoluted formula which might add an extra 10% or 2% accuracy, but overall it’s not going to add that much more value because they don’t understand how to use it properly.
Ruben: Exactly. I think here the key is that you want your recommendation to be actionable. You want people to understand what the analysis says and to be actionable, so telling someone that being in the U.S. gives you like a 50% lift over other people to convert or having purchased a paid preset in the last 30 days gives you like a 60% lift, that is much more clear and actionable than saying “The overall probability is blah-blah-blah.”
Kirill: Yeah, gotcha. Very interesting example. Thanks for that. The other thing I wanted to ask you and really pick your brain on, being in your position, as we discussed at the start, just before the podcast, you are a Director of Data Science, but a lot of the stuff that you’re doing is also strategic. It’s also which database systems are we going to implement, who’s going to have access to what and so on. You’re performing the role of Chief Technology Officer in combination with Director of Data Science, as I understand it.
A big part of that is spreading the culture of data across the organization. And that’s something I’ve personally faced in different roles I’ve been in, and it’s often tough to get that culture or appreciation across to people so that they become data advocates, that they help you out, that they look for these things in their own roles, how they can apply data and so on. So, how have you been dealing with that? What kind of challenges have you faced and the opposite – what wins have you had in the space of data culture in your organization?
Ruben: Yeah, it’s a good question because I think you touched on something that’s much more practical than running a machine learning algorithm or regression, it’s like how do you have an impact on your organization and how can you be effective? I would say that generally speaking, people are open-minded and everybody wants to be data-driven. The problem is that sometimes data is not accessible, or it’s too complicated, or people don’t have the time or patience to figure out exactly what’s going on.
And so my role as Head of Data at VSCO is to sit down with people and make sure that one, they have access to the data that they need, but also that when they are trying to make a decision, and they always want to make a data-driven decision, but when they are making that decision, that they are looking at the right data, they are looking at it the right way, and they understand the trade-off of the data and they understand the limitation of what the data can say. So, to give you a concrete example, we actually have a weekly meeting between my team and the product team where it’s kind of like a freewheeling meeting, but more often than not we talk about some of the analysis that I came up with, or we talk about some of the A/B tests that the PMs run.
It’s actually a forum where I get a chance to educate the team on how to interpret data, how to interpret the results of an A/B test and what the data says and what the data doesn’t say. It’s been really the source of great conversations. In particular, we’ve had great conversations around how to interpret the results of an A/B test, what is significant, what is not significant, whether one week of data is sufficient, should we look at just one metric, should we look at more metrics and retention, etc. I think the key here is really having an on-going conversation and trying to serve the needs of the people who are making decisions and are trying to use data to make decisions.
Kirill: Yeah, that’s great. I think that’s a very valuable thing as well, training up the staff and having that opportunity. It’s good that they’re very open to meeting up with you on a weekly basis and having this discussion because that’s a first step, for people to be open with that. And what’s your vision for that? What’s your vision for spreading the data culture in the organization? I know that you’re still in the early days, just six months there. But say in two or three years, how would you like to see every employee in the company? To what level would you like them to be able to interact with data?
Ruben: Yeah, I think there are already tools in place that let people access the data. We talked about Mixpanel. We also have a dashboarding tool that lets people see the key metrics and play with the key metrics a little bit. I think the key is that I want to be able to automatically answer 90% or 95% of people’s general questions. I think the role of the data team is not to do data pools and do like a SQL query to find out the number of users who did that or the number of people who’ve purchased in China or in South Korea in the last month. All of this should be answered automatically with self-service analytics, something that we talked about in our previous podcast. So there is that.
The second part is also that people who make decisions around data, particularly product managers, I want to make sure that we have a very solid framework for setting up and analysing A/B tests. Basically it means that we have a playbook of “Okay, these are the rules of how you design an A/B test, how many people you put in each bucket. Do not touch your A/B test for like two weeks so that you don’t mess up the bucket. And then, when you analyse the results, I want to you to go into this dashboard and to not just a particular funnel, but look at bunch of other metrics and check that they are not affected, or check that the retention is also there.”
It’s basically a combination of making sure that everybody has access to the tools and is comfortable with using the tools, so they go through training, but also setting up for more specific use cases, like funnel analysis, A/B test analysis, setting up like a playbook and making sure that the people who are going to own those features know how to run the analysis, because at the end of the day, I think even doing an A/B test analysis is something that’s pretty programmatic, you know, you can follow a set of rules. I don’t have to redo the analysis every time. I don’t have to have my team redo the analysis. I can just set up the dashboard, set up the guidelines, and have the owner of the product do the analysis on their own. So that’s kind of like my vision for how I would educate the team around the use of data and how I would empower the team to use data.
Kirill: Okay. That’s pretty cool. What it reminds me of is, there’s a company in San Diego called Digital Marketer. It’s a large company for marketers and how to do online ads and stuff like that. But I like how they’ve got it set up. They’ve got like a set of onboarding training which, actually, they have been so successful within the company that they have made them available. You can subscribe to them, and in SuperDataScience we use some of their trainings as well. You can subscribe to them and your staff can go through this training and some of them are like A/B tests and statistical analysis for marketers, of course. But at the same time, it’s pretty valuable and I like the structured approach. It kind of reminded me when you said that a lot of it has to do with education, so when somebody new starts, or maybe once in a while employees can go through these trainings and educate themselves on how to use these tools properly. That’s very important.
The other thing I wanted to ask you while we’re on this topic, you’ve touched on A/B tests several times, so maybe you can educate us a little bit. What is your rule of thumb for an A/B test? What should the sample size be for an A/B test to most likely be statistically significant? Do you have a rule of thumb?
Ruben: Obviously, it really depends on the size of your audience, the number of people who will see the particular thing that you’re changing, and the conversion rate. There are like pretty standard calculators online that will tell you, you know, “You need that many people.” Typically, in our case we need like 5,000-10,000 people in each bucket because we are interested in looking at meaningful changes.
One of the things that I warn people against is running a lot of A/B tests just trying to optimize the button or an image and trying to see small changes because I don’t think that’s the right way of running an A/B test because one is, you will always have false positives. Even when you run a test at 95% confidence, it still means like 1 in 20 you’ll have a false positive. So if you run five tests in parallel, all of a sudden you have a much higher chance of just hitting a false positive.
And second is because I think it’s important to, even when you run an A/B test, to have a hypothesis behind it. You’re not just testing random combinations of things. You are testing a hypothesis. You are saying, “I’m going to change the conversion flow on my store from A to B” or “I’m going to make the shopping cart much more prominent and therefore I think that people will see it and I think people will convert at a higher rate.” So it’s important to test the hypothesis and it’s also important to hold yourself accountable to a big change. You don’t want to optimize things and move like conversion from 10% to 10.1%. You want something that will move it meaningfully, that will move it from 10% to 11% or 12% or 15%. I think that’s the right way of running A/B tests. And when you do that, it also means that you don’t need as big of a sample because you are really looking for big changes. And if you don’t see a big change, then maybe it’s not that meaningful of a test.
Kirill: Yeah, gotcha. And last time you were on the podcast, you recommended a book by Nate Silver, “The Signal and the Noise.” I actually have it here. I’m reading it and I know I probably should have read it years ago, it’s so good, but it talks a bit about what you mentioned now, and I want to reiterate this point for our listeners. Your tests that you’re running, they have to be meaningful and they have to make sense, right? Because in the book, Nate Silver does a good comparison of Bayes, who was 18th century or something like that, and Fisher, who was 20th century or maybe he was 19th century; Bayesian probabilities and inference versus Fisherian, as they call it, probability and inference, and how Fisher didn’t really like Bayes for this concept. A lot of the stuff that we do, especially in this just pure statistical testing, is actually developed from the concept of the—what were we talking about—
Ruben: Frequentist statistics?
Kirill: Yeah, frequentist statistics, exactly. And therefore, we forget sometimes about the whole purpose, the whole idea behind the test. He has a good example in the book that if you use this approach, of just standard A/B testing, standard frequentist statistics, you can kind of prove that frogs can predict earthquakes just because there’s some correlation there, but correlation doesn’t always imply causation. So that’s a good example of you have to think about that what you’re testing makes sense. There’s no point even thinking about testing that frogs can predict earthquakes because it just doesn’t make sense. Even if you have a positive result, it’s most likely going to be a false positive, you know in advance. I think that kind of stands to the point which you raise.
Ruben: Yeah, I totally subscribe to that. I think it’s important to be thoughtful when you set up experiments and when you analyse A/B tests.
Kirill: Yeah, exactly. All right, moving on to my next area of discussion, your LinkedIn says you’re hiring. It seems like this is a credo that you carry through life. (Laughs) Wherever you go, you are hiring data scientists. I don’t think we talked much about this last time, but this will really benefit people listening. What are the skills—not just skills, what are the qualities that you want in your ideal candidate when you are hiring data scientists?
Ruben: That’s a great question because I think there’s a lot of misconception about what it takes to be a data scientist and how do you get hired. There are a couple of things that are important. The first thing is sort of like a general business intuition and communication skills. I think it’s easy for people to get mired into complicated analysis, trying to spin up Python and run this crazy regression, maybe boosted trees or whatever, but not being able to one, bring in the right ingredients in the analysis, and two, explain what comes out of the analysis.
Let me be a bit more specific. Let’s say that you were tasked with predicting churn of your customer base. I think probably one of the most important pieces of this work would be to come up with the right variables, the right features in your analysis that are most likely to predict churn. In order to do that, you really need some business sense and intuition. You need some data intuition so that you construct the right features in your analysis. And when you have the right feature, usually that’s like 80% of the job. It almost doesn’t matter what techniques you use to identify the most important features, having them in your dataset is the most important part and they will always come out at the end. So that’s very, very critical.
And the second thing is then being to explain to a lay audience what does the result of the analysis mean because, again, if you’re just writing a technical note that no one will read, then you have zero impact in your company. You want to be able to talk to the CEO, to the product manager, to the marketing manager, and tell them like, “I looked at the data and the data says these are the five things that predict churn and these are the rank order of the things that predict churn.”
Kirill: Yeah, totally. Break it down to them and make it digestible, accessible, the insights. And actionable, as you say.
Ruben: Exactly. So that basically speaks to both business intuition and communication skills. I think what’s important on top of that is also structured thinking. Like what you just said, the ability to take a problem that is very vague and complex and breaking it down into more manageable pieces and coming up with a data question that corresponds to that business problem. Earlier we talked about subscription and the business question was like, “What do our people who subscribe look like?” You have to be able to turn this into an actual data question that will yield a meaningful answer, and the data question is like, “Oh, let’s look at all the people that were active in the month of January. Let’s look at their activity in terms of editing and purchasing, etc., and let’s look at what makes them more likely to convert than not.”
So this ability to critical thinking, structured thinking, is also extremely important. And all of that has nothing to do with skills and knowledge. It’s really more about a way of thinking and grappling with problems. Now, obviously, there are important skills and knowledge. Number one, at least in my line of work, is SQL. You can’t really go very far without knowing SQL. Fortunately, SQL is a fairly simple language, so I would never hold it against a candidate if they are not experts in SQL because I think anyone can learn it. But having that baseline is important, and also having a very good understanding of statistics is crucial, I think. By that, I don’t mean to know every distribution and every statistical test on the Earth. I mean understanding a few statistical concepts and understanding them very, very well. Because people will come up with edge cases and this and that, that unless you’re very well steeped into your statistics you will not be able to answer.
Kirill: Gotcha. That’s very interesting. I want to get back to that in a second, but before, I want to comment on what you said about it’s not about the skills. I totally agree with that. For instance, even when we’re hiring at Super Data Science, we not as much look at what the person can do in terms of how experienced they are or what skills they have.
Personally, I look for passion. I look for drive. I look for purpose, meaning. I look for people who are eager, who can’t sit still. They want to know everything, and even before you’ve spoken to them, they already know everything about your company, they already looked everywhere and they know what they can improve, they come with value add propositions and so on. Those are the people that add value to the bottom line. I just wanted to comment on your point that it’s not only about the skills. In fact, it’s less about the skills than it’s about the type of person you are.
Ruben: Yeah, I totally agree. In fact, there are a lot of skills that can be learned. I spoke about SQL being an easy language. You know, if someone can demonstrate that they have the ability to learn, they are very curious and they want to learn, then I wouldn’t at all hold it against them that they are not experts in SQL. Because I know that they will be able to master it and there are things that are more important to me than their pure technical skills on a programming language.
Kirill: Yeah, I totally agree. And speaking of statistics, can you give us a few concepts, hypothesis testing, A/B testing, things that you think are important for a person to know, just examples, and then our listeners can pick and choose from there what they would want to focus on.
And I’m asking because I very often get this question. You know, people start understanding the skills and the techniques and methodologies and tools in data science, but then they are still left with this bit of uncertainty that there’s so much statistics out there. Which concepts should they focus on in order to, as you say, when they come in to an organization, be well-grounded and be able to stand their ground in terms of statistical discussions and presenting their findings and things like that?
Ruben: That’s a great question. I think you’ve mentioned a few. So, hypothesis testing is absolutely fundamental. And there are subtleties in hypothesis testing. It’s not something that’s very intuitive—in fact, it’s not intuitive at all, so really mastering the language and the logic behind hypothesis testing is very important, in part because that’s what people use to prove or disprove a hypothesis. That’s what people use to accept or reject an A/B test, so understanding that is very critical.
Also, understanding the key statistics that are used to accept or reject an A/B test, whether it’s like a chi-squared test or whether it’s just a t-test between two variables is important. One other thing that I think is also important is to understand the concept around the central limit theorem. The fact that when you have individual events or individual users and you group them together, you always end up having some sort of a normal distribution.
If you take—let’s say that we were looking at the average number of photos edited per cohort on VSCO, so we start with the January cohort, the February cohort, the March cohort, and we just take the average of number edited per cohort, it will obey a normal distribution, even though people individually are certainly not editing photos on a normal distribution. If you just took the distribution of the number of photos edited per person in a given month, that is an exponential distribution. You have more people editing one, then two, then three, then four. It’s monotonous and it’s decreasing. However, when you start taking averages, it always follows a normal distribution.
So being able to move in and out of this concept and understanding that is actually pretty important, because few people really understand that. Another thing is regression analysis. I think it’s still pretty key. It’s a simple tool, but it can be pretty powerful when you have a few high-powered features or predictors. So being able to correctly interpret regression analysis, and also knowing the trade-off and the problems with regression analysis is important.
Kirill: Awesome. Yeah, the limitations of linear and multiple linear regression.
Ruben: Yeah, exactly.
Kirill: Gotcha. That’s been a great overview. So, hypothesis testing, mastering the language and logic, A/B testing, key statistics used for this, so it’s either chi-squared test or a t-test, central limit theorem, being able to jump in and out of understanding how things become normalized or turn into normal distributions when you do certain statistical changes to them, and regression analysis. I want to add one more, my favourite one which I always hold dear to my heart – the law of large numbers. It’s very basic, but the more times you do something, the closer your average is going to be to the expected value. You’ve got to know that one. (Laughs)
Ruben: Yeah, totally. I totally agree with that.
Kirill: All right. Cool. Okay, time flies with you, Ruben. We’re getting close to the end. I’m like, “We need a third podcast. Where is this going?” But I wanted to do a big shout-out to you, to your hiring in VSCO. I think about 30%-40%—I might be lying here, so I won’t say the statistic, but a huge percentage of our listeners are in San Francisco, not just in California, but specifically in the San Francisco Bay Area. So, guys, VSCO is hiring, and whether you have a job or you don’t have a job already, get in touch with Ruben and first come, first served. Ruben, you’ll probably get bombarded. (Laughs)
Ruben: (Laughs) No, I love the promotion. Thanks for the shout-out.
Kirill: Yeah. So what’s the best way they can get in touch with you and see if this position is right for them?
Ruben: I think they can check it on my LinkedIn. I have a link to the position itself. I encourage people to check it out, see whether it vibes with their interest, with their qualifications. Feel free to either send me a message or apply directly on the website. I guarantee that we look at every single resume. Guarantee it! It’s not a black box, so either one of them works fine.
Kirill: Awesome. Gotcha. And I’m just looking at your LinkedIn. If somebody is listening to this in their car or something, it’s very easy to remember – bit.ly/VSCOdatajob. Check it out. If I was looking for a job, I would love to work in a team like that. I’m looking at some of your photos on the VSCO LinkedIn page. It looks like you guys have a really cool office, right? Is the corporate culture there pretty good?
Ruben: Yeah. It’s actually one of the most amazing companies I’ve worked for. It’s actually really incredible. It’s a bunch of very smart, but very humble people that are extremely collaborative. It’s never about who did what or who forgot to do what or who takes credit for a particular project. It’s always, “Let’s solve this as a team.” If anyone sees any bug or any problem, that person will immediately signal it to the rest of the team and everybody will work to solve it. When there’s a big achievement, the entire team is recognized, not just the one person who drove the project. It’s a very relaxed but also very accountable environment. People know the job that they have to do and they do it because they know that other people rely on them, not because their boss asked them to do it. I really love it for that. In fact, it’s a very flat organization. Some day you might be working with the COO on a project, and another day you might be put in a team with a couple of engineers and product managers working on it. It really doesn’t matter who’s in the room and who asked to do it, everybody will perform the same.
Kirill: Fantastic! Whoever gets this job or jobs, I’ve got an idea. Let me know that you applied from the podcast and you got the position and then next time I’m in San Francisco, you, Ruben and I will all go out for dinner and have a fun chat about how all this happened and came to be. (Laughs)
Ruben: Absolutely. Dinner is on me.
Kirill: Deal. Thanks a lot for coming on the podcast once again and sharing all of this overwhelm of knowledge. Yeah, incredible.
Ruben: Thanks for having me again. It was again a pleasure to talk to you and to your listeners.
Kirill: So there you have it. I hope you enjoyed today’s session. Lots and lots of valuable information. So diverse, right? We talked about so many different things.
We talked about the three points that Ruben focused on when he came into the organization. Ruben actually gave us a case study breakdown of how he went about analysing a project and answering questions and actually posing the right questions to answer in the first place. I think that was a very valuable example. Then we talked about the statistical concepts that Ruben is looking for in his ideal candidate and I think that was very valuable.
I’m kind of torn between the two, what is the most valuable for me, the example that he gave of the case study, where there’s these two types of ways that you could pose a question, or the statistical breakdown. They both have merits, but I’ll probably lean towards the statistical concepts that he outlined. That is very valuable, just having that summary of the five statistical points that you guys have been asking me for so many times, like “What do you learn in statistics?” Well, there you go. We’ve outlined them in the podcast and you can always reference these when you’re researching statistics for your career.
And speaking of careers, don’t forget that Ruben is hiring, so definitely check out his link. You can get all of the links and show notes at www.www.superdatascience.com/39 and there you’ll see the link to the job he’s posted. I don’t think it’ll be up for long, especially with this podcast coming out, you know, thousands of people listening to it per day, or especially in the first few days. So jump on it early, get in touch with Ruben, and I look forward to all of us catching up and having dinner together in San Francisco when I’m there next time. So make sure you get that job and I’ll see you there soon. And thank you so much for being on this show. I really appreciate you spending an hour with us today. And I can’t wait to see you next time. Until then, happy analyzing.
