SDS 211: Working on Apache Spark & R Package Development

Kirill Eremenko: Welcome to the SuperDataScience podcast. My name is Kirill Eremenko, Data Science Coach and lifestyle entrepreneur, and each week we bring you inspiring people and ideas to help you build your successful career in Data Science. Thanks for being here today and now let’s make the complex simple.

Kirill Eremenko: Welcome back to the SuperDataScience podcast. Ladies and gentlemen, I’m very excited to have you on the show today and today I’ve got a very interesting guest Javier Luraschi on the show with us. Javier is a Software Engineer at RStudio. If you use RStudio, you might know a couple of packages that he’s worked on or and/or co-authored packages such as sparklyr, mlflow, cloudml, and many more. And in fact, if you, even if you’re just a beginner in our, you probably have already incontinent the dark theme in RStudio and that’s something that’s Javier has also contributed towards. So we’ve got a very exciting podcasts coming up for you just now with Javier. We talked a lot about Big Data and Big Compute and specifically we talked a lot most of this podcast is dedicated to Apache Spark.

Kirill Eremenko: So in this podcast you will find out it’s the whole history of Big Data where Hadoop came about from why Apache Spark was created, how they compare, what a punch just park is used for, how it’s developed over time, how it’s developing now. You will also learn a lot about package development in RStudio and some of the exciting things that are happening in this space. And in addition to all events, you will feel a lot of passion. Javier has a ton of passion for this space, for RStudio, for a Apache Spark, for developing packages, for Big Data and Big Compute. So this podcast is full of that. I was personally sitting on the edge of my seat just enthralled by all these amazing stories that Javier is telling and this whole space of Big Data.

Kirill Eremenko: So all in all, an incredibly exciting, captivating podcast. Can’t wait for you to check it out. So without further ado, I bring to Javier Luraschi who is a Software Engineer at RStudio.

Kirill Eremenko: Welcome Ladies and gentlemen to this SuperDataScience podcast, today I have a very special guest calling in all the way from Seattle

Javier Luraschi from RStudio. Javier, how are you going today?

Javier Luraschi: I’m doing good Kirill. How about you?

Kirill Eremenko: I’m doing well as well and I’m in Australia right now in Gulf Coast. The weather is quite warm here, unlike Seattle. You said it’s getting a bit cold there?

Javier Luraschi: Yeah, I’ll probably have to go and visit you one of these days ’cause definitely we can use some of your sun.

Kirill Eremenko: Yeah, for sure. And you mentioned winters are quite harsh and saddle, like how cold does it get?

Javier Luraschi: It’s not that it gets that cold about, it gets cold minus five Celsius, which is not terrible, but we have very, very long winters. So definitely expect a visit from me somewhere around March or May. I’m assuming Australia has pretty nice weather most of the year.

Kirill Eremenko: Yeah. Most of the year. It’s quite good. The only thing is that it’s a bit like first time I got here I was not really expecting that during winter it’s warm during the day, but then at night the temperature drops to like maybe plus eight degrees Celsius, which is fine. But the thing is they don’t have central heating in the buildings. So the building is actually cold and you have to get blankets. So that was a bit unexpected. But other than that it’s really cool.

Javier Luraschi: I would say that at least in winter it’s a time where you can be very productive because you know, like it’s dark and everything and it’s cold. So it’s kind of like puts you in more to just get things done which is good. I kind of been on warm countries and sometimes getting work done in warm countries gets trickier.

Kirill Eremenko: I know. You work faster when it’s colder.

Javier Luraschi: I believe that’s for me, but I don’t know if that’s the same for everyone.

Kirill Eremenko: Same for me. All right. Well Javier RStudio. You’re on the podcast. It’s really cool to have you. As I mentioned, as I mentioned before the podcast already spoken to Nathan Stephens, Nisha Iyer. I’ve recently just been chatting on Linkedin to Jonathan Regenstein all from RStudio. You have such a fantastic team there. This is incredible. Every person I met from RStudio is like some sort of very interesting genius in their own field and I learned so much. I’m really looking forward to this podcast. I got really high expectations for today.

Javier Luraschi: Well we appreciate that, but like it’s definitely fun. I don’t know about being all of us genius but definitely it’s fun. Especially sine we’re a distributed company, I don’t even know honestly where Jonathan is these days, but whoever you talk with RStudio it’s likely that they won’t be on the same city ’cause we’re just all over the US also Europe and yeah.

Kirill Eremenko: It’s crazy. How is it to work in a distributed company like that?

Javier Luraschi: I really love it honestly. I do feel like it’s different. I feel like any other new job or any change in your life usually have a honeymoon period right? Where you … Even Data Science, right? I bet you start with Data Science and you’re like, oh my God, this is great. This is the best thing. For the most part. I think working in a remote team has been that way. I do think, and I would give us advice or unsolicited advice to your listeners, right? There’s definitely like a period where you get, you have to tweak your personal life to also maximize all of the the things that are not work related, if you know what I mean, right. So like when when you’re in the same office it’s very easy to have those personal interactions and keep up with people and what not and when you’re in a distributed team you kind of lose the personal face to face connection unless you practically say hey I want to catch up with these colleagues or friends that I haven’t seen right?

Javier Luraschi: But net net, I think it’s fantastic. I don’t even know what a community looks like and it’s nice be close to your family and what not so I definitely highly recommend positions and remote teams.

Kirill Eremenko: That’s so cool. That’s so cool. And I totally agree with the whole concept losing on the personal side of things you need something you do a take care of. Like for instance, at SuperDataScience, we have a remote team and we all catch up, we make an effort to at least once a year to all get into the same place and spend a week together. And the other thing is like every week we have a new body. So everybody in the team randomly gets assigned somebody else so it could be like a director. It could be an analyst together and for the week, no, not for the week, for the month you guys are buddies and so you plan your weeks together, you catch up once a week and that really puts people from different parts of their company closer to each other. So I think you’re right, the personal side of things.

Javier Luraschi: Yeah. So I’m actually curious like what does that mean? ‘Cause you’re still remote, right? Does that mean that you get to talk with this person in Slack or whatever you’re using or?

Kirill Eremenko: So Slack, you can talk to anybody anytime, but that means that at least once a week you need to catch up and talk to that other person in person. So we use Zoom, so not like in person as in-

Javier Luraschi: Yeah, we Zoom as well.

Kirill Eremenko: But like on video you need to catch up with that person at least once a week at the start of the week when you’re planning your week and you spend an hour together, and so you have already chatting about the weekend, what are you doing in your personal life? And feedback’s been really good. People love to find out more about that.

Javier Luraschi: Yeah connecting with other colleagues and what not. That makes a little sense.

Kirill Eremenko: That’s cool. So tell us, how did you get into RStudio in the first place? Before we continue, I just wanted to let everybody know. So Javier is a software engineer at RStudio. Maybe let’s start there. Before we go into the story of how you got into RStudio, tell us a bit about what exactly do you do at RStudio?

Javier Luraschi: Sure. Well, I mean, as you mentioned, I’m a software engineer, so I write software, right? That’s kind of like obvious. But I mostly have been focusing on R Packages. For those of you that might not be super familiar with what an R Package is, is basically our code, which is just R is programming language, right? And you package this code into modules that you can share with other people, right? So this is actually, it’s a concept that you can find in other programming languages, but I think what they R community, has going is pretty special. ‘Cause there’s a very nice relationship between R Package and an actual person that lives and breathes in real life, right? So a lot of the functionality that you use from R is going to come from packages and those packages have maintainers and authors. And at the end it’s just R code, but it’s packaged in such a way that it makes it very easy for you to reuse and for us maintainers to also keep up with new releases and new features and why not.

Javier Luraschi: So yeah I do R Packages and specifically I do packages that are mostly related with making R run faster or upscale or helping you share models with other people. So R Packages can basically do anything you want and the ones that I focus on are mostly about helping you work with Big Data and Big Compute and specifically with a package called sparklyr.

Kirill Eremenko: Interesting. That’s a very interesting description. I thought that R Packages mostly come from CRAN and that they are created by the community. I didn’t know that the R studio on purpose creates R packages. Can you tell us the difference between those two please?

Javier Luraschi: Well, they’re exactly the same. They’re just two different parts of the story, right. So basically an R Packages it’s literally like you could see it as a box with code inside it and CRAN is the store all the packages are free, right. But it’s where you go and get the actual packages. So there’s people that build these packages in the art community, they don’t need to work at RStudio. And honestly most of the packages that are on CRAN are not from RStudio at all. So anyone can package a useful piece of code into an R Package and then make it available on CRAN. And then anyone from the community can go to CRAN search for packages and then install them, right. So think about CRAN as the APP store, right? Whenever you go to the APP store, you look through apps and then you install them. CRAN is kind of like the APP store from the R world. And some of the packages very few are developed by RStudio. And many, many, many otters are developed by other people in their community.

Kirill Eremenko: Okay. Gotcha. And in that case, how does RStudio decides what the community is going to develop and what is going to be developed within RStudio, like for instance, within your team and just like within the company? Is there kind of some strategy behind that?

Javier Luraschi: Yeah. I think there’s some honestly I don’t think there’s an explicit strategy, like this is my own personal point of view. So I wouldn’t speak completely on behalf of RStudio. But the way that I see it is that if we see someone that is already working in a package and it’s a great package in the community, we just simply don’t work on it, right? We’re very pragmatic in the way we try to approach problems. So we try to look for the things that are the most painful and the most impactful that the community might need. And if we look at a problem, right? Like in my case, like, hey, Big Data, right? If someone in the community already providing that and if the answer is no and there’s enough big impact that we can help the community, we basically help.

Javier Luraschi: That’s kind of the very high level, like, hey, this is kind of how we see it. ‘Cause yeah, I think it could be possible that if they R community would take over the entire packages, there wouldn’t be a need for RStudio perhaps to develop packages, right. But something that perhaps, I don’t know if you already talked about this on the series is, but it is like the opportunity that exists in Data Science in general, and in my particular case, Big Data is so huge that, we need more people, right. So it’s not that we’re fighting for packages which is, I think it’s a great place to be at. It’s mostly like, oh my God, Data Science is growing so much and there’s so much to be done that, let’s just make sure that we don’t overlap because that’s just an efficient. But there’s so much opportunity everywhere that we haven’t had that problem of hey, having to do a lot of coordination or why not. It’s mostly pretty obvious when there’s a gap or an opportunity that someone can help with.

Kirill Eremenko: Gotcha. Okay. Very interesting role and lots of packages. So maybe let’s talk a little bit about some of the recent development. So you mentioned Big Data. Can you tell us a bit about Big Data, maybe even give a quick overview for new listeners on the podcast who are not familiar with the cost of Big Data. What is Big Data and Big Compute? What’s the story that’s happening there?

Javier Luraschi: No for sure. I honestly, I really love those two concepts, Big Data and Big Compute and mostly I really like them ’cause there’s people that have very strong feelings one way or the other one and you probably have seen these around. So kind of to me the way that I like to introduce them, it’s just from a historical point of view. Maybe that’s the most boring point of view, but I think it’s also pretty exciting, right? So if you think, like to me one of the most exciting things about in general Data Science and Big Data is if you ask a historian, right? Someone specializing in history, he’s like hey, how do you divide human development, right? The development of the human civilization. It’s going to be pretty likely that they will mention things like the stone age, pre industrial revolution, the industrial revolution and why not.

Javier Luraschi: And a lot of them will mention also the information age, right? I’m pretty sure that you might have already heard this term, right? The information age and this kind of concept basically means that there’s a lot of information that we’re creating that is digital, right? And if you look at one report from the World Bank, which basically creates reports about how the world is changing every year. One report that they had on 2016 kind of like try to analyze how much physical data we have that’s called analog data in some ways, like books and paper and anything that leaves a paper trail and why not. And also digital information, right. And you can see in the report it’s just growing an exponential growth rates. It’s just we don’t have to explain this too much, but if you look at how many cat videos we create per day, right? Or Instagram photos or Facebook photos or just data in general.

Javier Luraschi: We’re creating a lot of information. So back in the year 2003 around the time, there were companies like Google that were just starting, right? It’s crazy, but google is a pretty new company, right? It wasn’t here, what is that? 30 years ago or why not? And what Google was trying to do is say, hey, we have the worldwide web and there’s a lot of information that seems to usable. Can we make it searchable? And there were obviously companies before them like Yahoo and Excite and AltaVista for those of you that remember, but Google was one of those companies that said, how can we make search better?

Javier Luraschi: And the first program that you hit when you’re talking about the web, it’s like you can’t really put the whole web in one computer, right? It just too much data on if you try to put it in one computer, like it just doesn’t make any sense. So the way that they solved the problem back then was by using multiple computers, right? You have multiple computers, each with their hard disk across many computers you can load the entire web and make searches and that’s basically what gave birth to what’s called a HDFS and which I believe has been maybe mentioned already in your podcast. Which is basically a way of splitting data into multiple machines. Right. So that’s kind of like the beginning of Big Data is like, hey, if you have something that doesn’t fit in one machine, you have Big Data and it’s a pretty clear definition. There’s obvious problems that fall into this category, right? Like, hey I have my computer, I’m doing analyses, I can’t have all this data in my machine, I need multiple computers. Well now we’re talking about Big Data.

Kirill Eremenko: And HDFS e Hadoop Distributed File System?

Javier Luraschi: Yeah, it’s basically the Hadoop Distributed File System. Basically the way that this started is Google had a research paper where they explained to the world, hey, we have the Google file system, right? And we have a bunch of files, this is how we distribute them and then some engineers in India who they said, well, this sounds like a great idea, can we make it on open source project? And that’s where Hadoop came from. Is this open source implementation of the internal file system that Google made available to a research publication, right. And let me know how this-

Kirill Eremenko: Sounds great [crosstalk 00:18:44].

Javier Luraschi: Yes, it’s great okay. I want to make sure that I give enough information but not too much.

Kirill Eremenko: No. I love it. You’re totally right. It’s the historic context makes it even more exciting. Makes it like a story, like the journey. So yeah, no, I’m just listening. I’m really immersed in your explanation, please continue.

Javier Luraschi: Well, yeah for sure. So far so good, right? Google was doing their own thing and we have open source project like Hadoop that were doing their own thing and as I mentioned, they were mostly based on disk drives right? Disk drives even today can store the most information for the lowest cost, so it makes sense to put information there. But then a different project came out around, I want to make sure I get this right, but I believe it was around 2009 there was a project in Berkeley to kind of improve over Hadoop. Said, well, Hadoop is great we’re doing all these things based on a file system based on disk drives, but can we do it better? Can we do it faster? And sure enough, another open source project started in Berkeley called Apache Spark. And the premise at the point in time was, or one of the things that changed from their precursor was that, there was a trend in which memory RAM computer memory, was getting cheaper.

Javier Luraschi: Not as cheap as disk drives, but it was getting a bit cheaper, right. And it also happens to be the case that your computer memory makes your computer pretty fast, right? So whenever you go and buy a new computer, the amount of memory is one of the things that you want to go and check, oh, it has like four gigabytes or eight gigabytes or 32 or whatever. It makes a big difference on speed ’cause it means that a computer can handle more things in a faster and easier, way to access it, right. Anyways, so basically the Apache Spark project started by figuring out how to create something like Hadoop and improve over it based on memory.

Javier Luraschi: And what they found out is that sure enough you can get significant speedups by using memory instead of disks. And one of the things that we like to do as software engineers is sort data, sounds like such an easy task just to order data. If you had a list of names of people and you just want list them on alphabetical order, it happens to be a pretty compute intensive task, especially when you have a lot of data. Anyway, so what they found out back then is that it used to take to sort 100 terabytes, which is not a lot of data. It used to take Hadoop 72 minutes and 2100 computers. So, that was with Hadoop. And then with this new framework based on memory and with have a more rich vocabulary of operation that you could do, you could do the same in 23 minutes but only use in 200 computers.

Javier Luraschi: So that’s crazy, right? It’s an improvement of 10X performance, you need 10 times less computers and you actually make it … You can sort the information faster. So it was a really, really big deal back then. And in some ways it’s still part of the main trend of trying to figure out as a society and as humanity, how do we order, how do we make sense of all this information, right? How do we arrange it? How do we store it? How do we answer harder and more interesting questions over and over? So, that’s pretty much what Spark is. It’s an in memory engine that allows you to process any information. One of the things that is different perhaps with Hadoop is that when you talk about Hadoop, there was only support for one type of operation called map reduce. And maybe this is also something that people have heard around when they’re getting immersed into a Data Science and Big Data, right? It’s like map reduce
.

Javier Luraschi: Well, back when Hadoop was designed, there was another paper from Google where they proposed this model. Whenever you want to transform information, I guess I should say that so far we’ve talked about storing information on retrieving it. We haven’t talked enough on okay. Well information is across multiple computers. How do you actually make it do something meaningful if it’s distributed across multiple computers, right. So the first attempt to solve that, which also was at around the time a couple of years after Hadoop came to be, was another paper from Google who called MapReduce, which basically explained the world, say, hey, if you have a distributed system, that basically means you have a lot of computers, you can reduce all the operations to mapping meaning transforming some information in the same machine to some other information in the same machine, and then combining the information between machines.

Javier Luraschi: That’s kind of like at a high level what it is. And it was pretty good at that time. But it was also constraining in the sense that it didn’t provide a lot of verbs or kind of like grammatical constructs to make coding more easy, right? It was pretty bare bones at the time. And one of the other big improvements from Spark is that it enabled big vocabulary of operations to make Big Data more easy to analyze, right? So instead of just saying, hey, you need to tell me how to map data on each machine on then aggregate it, it introduced things like, hey, just tell me what you want filtered. What do you want to the average to be like. Kind of like more things that are now they feel closer to what data analyses. Like, hey, I just want to count how many earthquakes are available in these data sets. That’s a pretty simple question that used to be actually pretty tricky to ask back when we only had MapReduce.

Javier Luraschi: So with Spark you can say things like, hey, from all these data set, give me all the earthquakes that just happened in Australia on this period and give me the count. So that’s a much more reach way of expressing and analyzing data when it’s distributed across multiple machines that previous technologies didn’t really … It wasn’t as simple. You had to provide code for you to process each disk drive on each machine and then figure out how to upgrade and all that. So yeah it was a big deal and it’s still a big deal. I mean, it has been less than 10 years. So if you look at the Apache Spark project, I mean, we can talk about more of the things that have happened and why not. It’s still changing a lot. So it’s by no means what I would consider it done. And it’s growing everyday.

Kirill Eremenko: Gotcha. So if I understand correctly, Apache Spark not only, is faster than Hadoop in the sense that it works in memory, uses RAM instead of the disk data, but it also is simpler to use to operate. Is that correct?

Javier Luraschi: Yeah. That is completely correct. And the thing to say here, it sounds really easy and it is really easy compared to dealing with MapReduce. But you’re still talking about hundreds of machines or at least tens of machines. So it’s a pretty hard problem and it’s surprising the amount of progress that the open source community has made in the past 20 years. Where now really anyone, if we want to get into specifics of hey, how do I get into these? Really anyone can download the tools that they need in less than 15 minutes and get up and running and start doing data processing. That was really state of the art in companies like Google or Yahoo like 15 years ago. So that thing is just kind of like the technical explanation of why Data Science and Big Data, it’s such a big deal.

Javier Luraschi: It is ’cause these tools are so easy today than what they were before that we honestly haven’t figured out yet how to fully apply them and we can maybe talk a little bit about this. But just talking about potential. It’s like, well Google and Yahoo and other big companies were using these tools and they were great and now they just became so easy to use. What can we do with them? And I’m sure a lot of your listeners will have more particular ideas, but it’s definitely fascinating.

Kirill Eremenko: Yeah for sure. And do you have certain packages in R which to allow people to work with Apache Spark?

Javier Luraschi: Yeah, that’s a great question. And the answer is yes. And the package that I recommend using, it’s called sparklyr. It’s kind of like a corny name out of Spark. You have Spark this is like sparklyr or sparkly R. Yeah, so we try to name our package in fun ways ’cause it doesn’t make sense to name them with boring topics. So yeah, it’s called sparklyr and it’s basically the way that you use Spark from the R programming language, right. So R is obviously a pretty well known programming language or computing language I would want to say-

Kirill Eremenko: Very well known. My brother is studying at uni now and he’s actually using R. It’s really going well.

Javier Luraschi: There you go. So if you’re already have an age for learning R, it’s another natural progression to say well, what about things like Big Data? And we haven’t even talked about Big Compute so we need to get back to that. But yeah, definitely if you want to get involved into doing things in cluster computing with Big Data and why not, this R Package called sparklyr is a very nice way of getting started. And again in this particular case, it’s a package that RStudio developed, myself and other people in RStudio are authors of these package. But it’s actually an Apache licensed open source project. So it’s completely free. It’s available in CRAN, you can download it, it’s easy to install. And same for Spark. I didn’t mention that, but Spark also happens to be an open source project, it’s Apache licensed. So it’s pretty much there for anyone to use that has that need or interest.

Kirill Eremenko: That’s very cool. That’s very cool. How long does it take you guys to do all the packages? Just out of curiosity?

Javier Luraschi: Well in all honesty, I feel like we’re still working on it and we’re going to be working on it for a while and. So the original package we worked for a few months in it back in 2016. We actually announced it back in use R 2016. It was like the base basic functionality which included things like being able to do data analysis with dplyr which it’s one of the most used packages in the R community, which allows you to basically design data manipulation operations through our grammar. That makes it very easy to let you analyze even from very simple things like hey, give me the average of this column, we all used to do that in Excel or why not, or accounting records all the way to, well, I wanted to run, I want to join these two data sets and I want to run a specific computation on them and why not. So it definitely has that breadth of functionality for very basic data analyses to very complex data analysis.

Javier Luraschi: So that was one of the first features that we decided to include in sparklyr to allow the community to easily move their already existent data like basically analysis that they can run in their own machine. Like if you have a CSV file, just a text file or why not, or an Excel file … By the way you can use Excel with R if that’s what you like doing. So what dplyr allows you to do is to say hey I’m going to use Excel spreadsheet to do analysis so you can import it with read R and then you can do data analysis on the dplyr. But then now with the sparklyr and support for dplyr in the sparklyr you can say, well instead of running this same analysis in this Excel spreadsheet file, now I want to run it on like 10 terabytes of data or why not?

Javier Luraschi: It’s the same thing. Which is kind of crazy. You don’t need to regret it. You don’t need to worry a lot of those things. You do need to pay for the computers, right? We haven’t talked about this but like still computing is not free, right? So someone needs to pay for those computers and what not. But at least as a user and if you work in a company or you aspire to work on a organization that is going to do like data analysis in small scale and also in big scale, these tools allow you to really easily jump from, hey, I just want to do something quick and dirty while I’m on the bus, on my way to work, and it works, it’s great. And then like you can just literally copy paste the same code and run it against like a ginormous data set.

Javier Luraschi: Yeah, so that’s kind of like how it started back in 2016. But since then, what we’ve seen is that the Spark community keeps growing. They keep adding new features-

Kirill Eremenko: They have to keep up.

Javier Luraschi: It keeps getting more and more interesting. Like for instance, one of their conferences got renamed last year I believe from Sparks Summit to Spark Plus AI Summit. So like now spark is getting also intertwined with deep learning and it’s enabling a bunch of other really cool interesting things to do. So I honestly don’t think that we’re going to be able to call our work done at least not on the next few months, right. I don’t know if it’s one year, two years or another 10 years, who knows? But definitely it’s like a moving target. And I think that’s, that’s exciting, right?

Javier Luraschi: ‘Cause that means that you’re not jumping into something that started like 10 years ago and no one uses anymore and why not try? It’s like the opposite, right? There’s things that are very stable and you can be very reliant on and use them at scale and we are very confident that they’re going to give you the productivity results that you’re looking for. And they’re sort of other things that are just like very, very new. And they’re exciting and they’re probably going to get there, but they’re still like a moving target. So there’s definitely like interesting bits and pieces for the new comers and for the very experts to be excited, specifically about spark just ’cause we’re talking about Apache Spark, but also I think in general on Data Science.

Kirill Eremenko: Gotcha. And while we’re on top of a spark, I know we have so many other things we can discuss, but I’m really curious, Apache Spark 2.0 was released I think last year, at the beginning of last year. What are your thoughts on that jump from Spark to Spark 2.0, what new stuff was added?

Javier Luraschi: Yeah. For sure. Well first of all I would say that that was a big jump. Just to give some context, Spark 1.5 was like the first wave of people were starting to get familiar with it and I feel like it really hit mainstream. 1.6 is still one of the biggest released versions that is still in use today and the jump from 1.6 or 2.0 introduced many improvements and interesting new features. For instance, one of the ones that I was excited about was Spark structure at streaming. I know it’s a pretty long name but basically means real data processing. And I feel like that’s a really good segue to talk a little bit about Big Compute, right? ’cause we only talk about this Big Data kind of part of the story which is important and relevant. But then a lot of companies don’t have huge data sets, right?

Javier Luraschi: So what’s the point of Big Data? Why do I need Spark? Is this just hype and whatever? And you know like some of that is true, right? Not everyone needs to Big Data, but there’s this whole other side of the coin called Big Compute. And what Big Compute the way that I like to explain it is just mostly making things faster, right? So like when you have simple questions like, hey, count how many records I have. Well that’s usually pretty fast and you don’t have to worry about that, right? But when you start asking questions like, hey, could you please sort, arrange all these data set. Well, that’s a little bit harder, right? And then for those people that are in the track of doing Data Science, and again, I’m not an expert on Data Science, I’m a software engineer, but like they will get more and more familiar with a more complex models, right?

Javier Luraschi: So, some people might already be familiar with linear regressions, which are a type of model that is pretty common and pretty efficient and like a pretty good first step towards modeling. But then you can incrementally get harder and harder models. You want to really fit the data correctly and why not? So a lot of times what happens is like, well maybe I only have 100 megabyte data sets. Like well, that’s obviously not Big Data, right? But then you’re running these models and a lot of times what we see is we see data scientists just waiting like an hour, right? Or it’s like, well, I’m going to go for a coffee ’cause this thing is just running for the next two hours. And it’s like, well, you know, that might be a good case for Big Compute, right? Which means, it only really means saying, hey, let’s divide these tasks into multiple computers ’cause even though you don’t have Big Data, wouldn’t it be nice if instead of waiting two hours we’re going to give you the answer in like 20 seconds, right? And he’s like, well-

Kirill Eremenko: Then there is no time for coffee.

Javier Luraschi: Okay, don’t tell about your boss, but you can do it faster and still go for coffee, but anyways. So definitely there’s the other side of the coin where you say like, well, I don’t care about Big Data or I don’t have a need for Big Data, but I want to make things faster. And when you get to that point, it’s like, well, how fast is fast enough? I mean, you know, for you and me maybe we don’t have interest in data sets and we’re like, well, if you give me the answer in like 10 seconds that’s good enough. Who cares? That’s fine. But there’s a lot of industries out there, like I’m just thinking of a stock trading, right? I mean if I tell my boss, he’s like, hey, you know, I’m going to give you the answer in 10 seconds. He’s like what are you doing?

Javier Luraschi: That’s really not going to help me. So there’s a lot of use cases where you want to have instant feedback, right? And some of the ways that we describe this is with concepts like real time. We say oh, we want to process data in real time, right? Meaning I don’t want to wait for whatever is being processed to finish processing. I need the data right now. And there is definitely a niche there of who really needs real time versus who doesn’t. But Spark structured streaming enables those types of really fast execution models that are very useful in some cases, that make a lot of sense. And the way that they are tackling it, that we’re tackling with sparklyr and specifically streaming, it’s kind of like a very profound, interesting way. And the credit goes to the Spark project. But the way that they define Stream.

Javier Luraschi: So we’re talking about Stream and we haven’t even really defined like, hey, what is Stream? Right? Well, the way that we define what a Stream is in Spark is like a table, like Excel spreadsheet, but when you open Excel spreadsheet, you have a limited amount of rows, right? Like you open it and you have, hey, I have like 20 rows or whatever two million. But you have like a set number of rows. The difference with streams is that we consider them us data sets that have an infinite amount of rows, right? It’s not true that they have an infinite amount. But if you’re looking at the stock market and you were seeing like what’s the price of the Nasdaq every second. And if you try to see that as a table, well it’s a table, but I mean assuming that the Nasdaq doesn’t crash and disappears from planet earth, that looks like an infinite data set, right? Of data that just keeps coming.

Javier Luraschi: And it also have the quality that you want to process it really, really fast. You don’t want to wait of like, yeah, I’ll tell you what is a good prediction for the stock market tomorrow. Right? It’s like, well, I need it instantly. So kind of like when you start looking at those data sets, that data is coming really fast and it never stops coming, structured streaming is like the future that you kind of want to consider. So, that’s one of the newer features that I’m excited about. Doesn’t mean by any means that you need to get started with Data Science and Big Data with streaming, right? There’s a million other ways to get started, but it’s definitely one of those features that it’s pretty exciting. I could talk here for hours, so just telling me kind of like where do you want to kind of steer the conversation.

Kirill Eremenko: I’m glad we touched on Big Compute and that it’s a part of Spark 2.0. So Big Compute is kind of, as I understand it in this sense, is the … Or the predecessor for Big Compute is just paralyzed computing. That’s something I studied back at uni-

Javier Luraschi: Yeah it is-

Kirill Eremenko: About paralyzed processes.

Javier Luraschi: Yeah it is.

Kirill Eremenko: Okay. Gotcha, Gotcha. And Spark takes full advantage and that sense that it’s running in memory, right? Big Compute happens in memory and not on disk drives.

Javier Luraschi: And we were talking about these verbs that exist on Spark, that make it really easy to do operations, right? Like, hey, I want to filter, I wanted to get the average, I want to join these two data sets, right. Since you are already familiar with parallel computing, you probably also remember how hard it is, right? Or it used to be, right? It’s not trivial at all to say, well I have three computers with small data set, now I want to do calculation over these three things at the same time. And it’s like, well, it’s actually pretty tricky.

Javier Luraschi: I wouldn’t say necessarily that it’s fully sold on Spark ’cause there’s a lot of like, you know, if you’re doing genomic analyses are why not, right. You will probably have to do customs things. But just talking specifically about data analysis, that problem is well solved in Spark. You want to have to worry about, hey, how do I make these things run faster? You just explain it in terms like similar things like what we were talking about in our, we have the dplyr package. You can arrange a pretty Big Data set, like this hundred terabyte data set that we were talking about. You can still sort that data set in 23 minutes just by saying sort. Like sort parentheses, open parentheses, close parentheses, what a Pi before that and why not.

Javier Luraschi: So, kind of like those things get … Yeah and you’re right. Big Compute is not a new term. It’s just something that has been getting simpler and simpler. And hopefully with time it will get even simpler. I don’t even know how that looks like ’cause I already think it’s pretty simple. Honestly we could also talk about that ’cause I feel like some of the challenges today still on, on distribute computing is about troubleshooting when things are running at scale. Like in a lot of cases things will run smoothly and you can run your computation and be done with it. The reality today is that, the harder computations you’re doing, even though it’s easier to express what to do, there is still a lot of like, hey, I need to troubleshoot. Why is this machine failing? Does it have enough memory? My hope is that in the future those things will get even more and more automated?

Javier Luraschi: I’m just making stuff up at this point. This is almost science fiction. But like it would be cool that before you execute some data analysis and you say like, hey, I wanted to sort these data set. It would be cool if the tool would tell you in this case a Spark or sparklyr or why not would tell you, hey, you’re probably going to need, like if you want to run it in one computer, it’s going to take you three hours. If you get up 10 computers with eight gigabytes of RAM each or whatever, it’s going to take this much.

Javier Luraschi: That part today doesn’t exist yet. You need to figure out like how big the cluster needs to be or how small or get kind of like get some advice from your system administrator if you work in a big company or a big organization. But hopefully one cool thing that we could do is just make it easier to say hey, we’re going to help you optimize what you need and we’re going to tell you. And then you can execute or why not. But yeah, there’s definitely like a lot of really interesting work if other … If a lot of people listening to your podcast feel more on the software engineer track, I would also encourage to explore kind of like those areas.

Javier Luraschi: A lot of the questions that we get or I personally get today to answer is I’m a software engineer, how do I become a data scientist? And that’s totally fine for those people that want to become data scientists. But there’s also a lot of disciplines around data scientists where people can apply their skills-

Kirill Eremenko: Like you. You’re a software engineer working on Data Science stuff all the time.

Javier Luraschi: Right. And I love software engineering and I wouldn’t change it, but it’s surprising that I still find a lot of very meaningful problems and interesting challenges on Data Science without being per se a data scientists, right?

Kirill Eremenko: Yeah. Gotcha. That’s definitely a very interesting-

Javier Luraschi: Point of view I guess.

Kirill Eremenko: Yeah and career path that you’ve decided for yourself.

Javier Luraschi: I mean honestly I haven’t thought about other career paths, but I could see how someone that is doing marketing or why not like could also take a focus. I don’t know exactly how that would look like. But you could say, you know what, I like marketing but I want to compliment it with Data Science and how do I really apply marketing to Big Data and Data Science and why not. So for those people that are kind of like by curiously looking at Data Science or not I feel like there’s also strong career paths if you stay on whatever you’re doing, but put a strong focus on Data Science or Big Data or Big Compute, why not? But you stay where you’re doing and that’s probably also as valid.

Kirill Eremenko: Very, very true. Before we move away from Spark, I just wanted to ask you quickly from your experience and from what you’ve seen in this space, how difficult is it to learn Spark? You mentioned that it’s quite simple to use as in it paralyzes a lot of stuff for you. You don’t even have to think much about it, but in general, what would you say, how long is the learning curve from knowing some R programming and how to do Data Science on a basic level in R to actually knowing how to use Spark and querying Big Datasets with Spark and with R. How long do you think that would take?

Javier Luraschi: Well, so I would split this into two questions, right? So if you’re starting by your own and you don’t have like even a Spark cluster. If you’re literally like finished like if you’re on your own, I feel like that could be challenging ’cause it’s like, well where do you even get the computer? There’s a lot of questions to ask and how do you-

Kirill Eremenko: Amazon?

Javier Luraschi: Yeah, Amazon. So there’s a lot of services out there. There’s Amazon, Amazon has service called EMR. There’s a service called Azure HDInsight, there’s Cloudera, there’s Databricks. There’s like 10 different ones and I apologize for the ones that I didn’t mention. So there’s definitely ways of getting started if you’re on your own. But I think that’s usually not the case. Usually what happens is, well there’s … We can break it down again into two. One is like if you’re learning, if you’re like, hey, I want to learn about R and I want to learn about Spark. Well I have very good news for you ’cause it’s actually super, super easy. In fact that’s one of the goals that I have that brings me the most joy is to make it absolutely insanely easy for you to get started with Spark and R. So you can get started very easy. You download the sparklyr package by installing it as any other R Package and then you literally run, install_Spark, open parentheses, close parentheses, run dot. It will install it for you and then you run Spark connect.

Javier Luraschi: Don’t try to do these from the actual podcast. But if you want more information, go to spark.rstudio.com and we’ll help you get there. So it’s totally real easy. So, that’s one case. If you’re a student, definitely you can do it and the barrier or the cost to enter is super, super low. So give it a shot. The other way where you might end up working with Apache Spark, which is also really easy, is if you happen to end up working on an organization that already has an Apache Spark cluster, it is often the case that there’s a cluster administrator and there’s someone whose job it is to maintain that cluster or someone that is already paying the bill with Databricks or Amazon or Google or Azure why not, right?

Javier Luraschi: So if those clusters are already up and running and a lot of times the data is already there, it also happens to be very, very simple ’cause you don’t have the burden of setting that up, right? All you need to ask is, hey, where’s the cluster? What is the connection name? If you listen to … I’m almost sure that Nathan probably talk about connection strings and why not, right? All you need is a character string that tells you where is the cluster. And so you basically put data into Spark connect and that will get you up and running. And cluster administrators are pretty good at helping other employees or members of that community to get up and running. So I think in both cases it’s pretty easy to get started.

Javier Luraschi: So those are I think just great news. There’s almost no excuse to not try it out. I would say that it is a linear … Well, I would like to say that it’s a linear learning path. So it is not true that doing everything on Spark is as easy, if you know what I mean. As we were mentioning there’s things that are very easy and newer things like maybe Spark Structured Streaming or like other topics. I work practically every day to make them easier. But you kind of like want to start with small steps which will get you very, very far regardless. But then as you feel confident and feel proficient, you just need to keep walking, right? It’s just like a slope with our [inaudible 00:51:38] client. Kind of like, if you’re like mountain climbing or what not or kind of like that type of deal where you’re like well, you start going, it’s not that hard and you definitely at some point you’re going to be hitting harder problems.

Javier Luraschi: But the very nice thing is that to get started it takes very little. And I think that’s important ’cause if it take a really long time to even do simple things or why not, it’s like no one has time for that. But if it gets you up and running and you can do meaningful analysis very easily, which is where we are today, it’s very easy to get us started. It’s very easy to learn. And then as your problems get harder and harder and you are answering more interesting questions that are interesting to you or that bring value to whoever you’re working for or with, it just makes, makes it easier that you just need to keep working there and there’s, I didn’t mention it but there’s a great community in R. My guess is that some of the other R speakers here mentioned it, but there’s a very nice, warm community around R and specifically also about sparklyr and why not.

Javier Luraschi: So you can go to resources to community.rstudio.com and just ask for help. You can also go to our GitHub page and look at the issues and if something feels like that you really need help with, you’re going to open an issue. So you’re also not alone in this and there’s a lot of resources to get you up and running. So I definitely, at least will encourage everyone to, if they’re curious about Big Data and Big Compute and Cluster Computing, give it a shot, ’cause you’re going to be surprised. That is something that feels doable. Which would not have been the case few years ago.

Kirill Eremenko: Fantastic. Thanks so much Javier for that description. Hopefully that will encourage more people to jump into learning Spark, especially through R. It sounds like you guys really make it happen. Make it easy and those who are already using sparklyr like Javier he’s making things I like for you in the space. Well there’s so many more things that I wanted to ask you about Big Data, about your journey into this career and so on. Unfortunately we’re running out of time and probably we only have time for one more thing. And I thought out of all the things that I have written down, I thought the most important one would be your book.

Kirill Eremenko: So you mentioned before the podcast you’re working on a book and seeing how much value you’re giving on this podcast, how passionate are you about the space. I think it would be a shame for listeners to hear that you and some of your colleagues or some of your friends you’re working on a book and that is going to be published next year. So tell us a bit about that. What is this book going to be about so people are interested in this space can look forward to it.

Javier Luraschi: Right. So the name of the book is going to be somewhere around the lines of, the R in Spark. Kind of like funny name the R, what is the piece that we’re going to highlight from Spark or R, the R programming language. So we already have a website and we’re going to put more information when the book is published, but the website is, therinspark.com. So it’s pretty straightforward. But for now it’s a bit of a placeholder, but if you’re interested at least it’s good to keep in your bookmarks. But my goal with that book is to make it, well both with sparklyr and with the book is to make it the absolutely easiest way to get started with Apache Spark. So anything from, hey, what is this thing which we already … Your listeners are lucky enough to have you that they already know, kind of like what’s spark right and they got a very nice introduction.

Javier Luraschi: So the goal of this book is to make it very, very easy for anyone that opens the book to say, wait, [inaudible 00:55:37] is the Big Data? Oh, that’s what it is. What is spark? Oh, now I get it. Okay, now that I understand it, how do I get started? It’s going to be a very gentle introduction. But being gentle, it’s not going to remove the fact that if you go through the whole book we want to take you from being a very new user to being close to being an expert. And like any book, you need to do the exercise and practice and why not. But we’re definitely hoping that we can bring a lot of people to the Apache Spark community as well and just help them out. So yeah, definitely if you want to keep in touch, therinspark.com might be the place.

Kirill Eremenko: Awesome. I’m actually looking at it right now and there’s for our listeners, if you’re interested in what Javier was talking about today and you found it exciting to listen to R in Spark, if you go to the website, therinspark.com, there’s a way you can get early access to the book. So you just have to send an email to Javier and you’ll get early access. I think that would be pretty exciting to get early access to the content. So yeah, if you guys are interested, jump on top of that. Javier, we’re out of time. Thank you so much for coming on this show. Being an amazing journey just listening. Totally, totally captivated too. You have so much passion for this space. Before I let you go, I have to ask, what are some of the best ways for our listeners to get in touch with you to follow up? You mentioned the R in Spark, what other ways can our listeners get in touch with you?

Javier Luraschi: Yeah. I would say Twitter, but I’m honestly so bad at Twitter. I need to listen for 10 tips to be a Twitter user. So Twitter, definitely you can find me there Javier Luraschi and that’s just Twitter, first name, last name. I’ll do my best to answer there. But definitely on the GitHub page and we also have a Gitter channel. So it’s going to be pretty easy if you start looking at the sparklyr, if you search for sparklyr, one way or another one, you’re probably going to end up reaching myself and other colleagues in RStudio. So I would just say just don’t be afraid, whichever form you find of contacting us, there’s a Gitter channel where we can chat. There is Twitter if hopefully I get better at using it. There is GitHub, there is the book that will take us a few months to get it to you. But just in general, whichever way you can find us, just feel free to keep in touch and I’m looking forward to that.

Kirill Eremenko: Awesome. And Linkedin, is it alright for listeners to connect on Linkedin?

Javier Luraschi: Yeah. For sure. It’s first name, last name altogether. I don’t think they have like a nice search in there, but I my first name Javier, last name Luraschi. Twitter as well. You can find me there. I don’t know if I miss anything else, but I can give you my address if you want to. Just kidding.

Kirill Eremenko: And your social security number and where the money is.

Javier Luraschi: That’s for sure. Yeah, we’ll put it out there.

Kirill Eremenko: Okay. No, I think that’s all. Well, once again, thanks so much Javier. Good luck with the book. Looking forward to it coming out and I’m sure a lot of people are going to get from this podcast. Thanks so much.

Javier Luraschi: Well, Kirill thank you so much for having me and grade work with this podcast. I’m really happy I had the chance to be with you today, here.

Kirill Eremenko: So there you have it. That was Javier Luraschi from RStudio. Hope you enjoyed this session as much as I did personally. My biggest takeaway and fairest was the whole fact that we dove into Apache Spark so deeply and got to know this space so well from firsthand, from a person who actually works in developing a package to work with Apache Spark. So Javier is up to date with all of the changes in Apache Spark and he knows exactly everything that’s happening in this space. So it was really great to hear this information, these insights directly from him. And I’m sure you can also feel the immense passion than Javier has for the space and in fact it’s even contagious, so I’m sure if you have never heard of Apache Spark before, now you can feel that this is one of those really powerful tools that maybe one day you’ll add to your Data Science toolkit.

Kirill Eremenko: On that note, you can get the show notes for this episode at www.www.superdatascience.com/211. There you will also find all the things that we mentioned in the episode. All the materials we mentioned in the episode, a URL to connect with Javier and follow him and his career on all social media. You’ll also find a link to the upcoming book, to the website where you can register to get some contents of Javier’s upcoming book, which is going to be pretty awesome based on what we heard today. And of course, if you know anybody interested in the space of Big Data, in Apache Spark who wants to learn more, and who would be as excited about this episode as you hopefully and of course as I was on today’s show, then please forward them this link. Help spread the word, help other people learn these topics. Apache Spark is a really cool tool that is helping data scientist’s work of Big Data. So let’s help each other out. Send this episode to anybody who you think might benefit from it. Whether it’s a friend, colleague, family member or somebody that you just know that this will help them out.

Kirill Eremenko: On that note, thanks so much for being here today and sharing this hour with us. Can’t wait to see you back here next time, and until then, happy analyzing.