SDS 333: BERT and NLP in 2020 and Beyond

Kirill Eremenko: Welcome to the SuperDataScience Podcast. My name is Kirill Eremenko, Data Science Coach and Lifestyle Entrepreneur. Each week, we bring you inspiring people and ideas to help you build your successful career in data science. Thanks for being here today and now, let’s make the complex simple.

Kirill Eremenko: Hey, everybody. This episode is brought to you by Data Science Insider, our very own newsletter, which comes in your inbox every Friday. So, I’ll make this one brief. Basically, you go to www.superdatascience.com/dsi and sign up for an absolutely free amazing newsletter, which is curated by our team. We look at the top developments in artificial intelligence, machine learning, data science, and other exponential technologies which are relevant to us as data scientists. So, we look at the top five developments in the past week. We put them together, put some images, put a short description for each one, send them out in an email, put the link, which takes you straight directly to the source if you want to read further. That way, you can stay up to date with what exactly is happening in the world of data science and artificial intelligence.

Kirill Eremenko: So, once again, the link is superdatascience one word dot com slash DSI, which stands for Data Science Insider. So, head on over there and sign up today, and start receiving your updates on technology that is relevant to your career already this week.

Kirill Eremenko: By the way, in this podcast with Sinan, you will hear at the end how he is actually benefiting a lot from newsletter updates, and we talk about the ones that are our favorites. So, this one can be your favorite as well. On that note, let’s get straight into the episode.

Kirill Eremenko: Welcome back to the SuperDataScience Podcast, ladies and gentlemen. Super, super pumped and excited to have you on today’s episode. We got off the phone with Sinan a couple of hours ago, and what a blast. I am so excited to bring you today’s episode, and I’m really happy that it’s got the magic number of 333 because it totally deserves that number, totally deserves to be special and unique. The conversation was extremely insightful.

Kirill Eremenko: So, straight to the point, here’s what you will hear on today’s episode. You will hear how Sinan’s company, Kylie.ai, was acquired. If you’ve been with the podcast for long enough, you’ll remember that Sinan has appeared already twice on the podcast. Most recently, he told us about his new startup, Kylie.ai, and the magic they were doing in the space of natural language processing. Recently in 2019, they were actually acquired by Directly and with that acquisition, Sinan has now become the Director of Data Science at Directly, and he’s leading a team there. It’s so exciting what’s been happening in his world. You’re going to love it.

Kirill Eremenko: Then we’ll talk about explainable AI. We’ll talk about bias in artificial intelligence, and then Sinan will give us actual examples, case studies of how they’re applying NLP in an AI plus human synergy for companies such as Airbnb, Microsoft, and others. Well, he’ll actually walk us through two case studies, one for Airbnb and one for Microsoft.

Kirill Eremenko: Then we’ll talk about building company-specific AI models, and even product-specific AI models, and what that means for the world of AI. We’ll talk more about acquisition. We’ll talk about hiring. By the way, they’re hiring at Directly. They’re hiring data scientists and data engineers. You don’t have to be based in the San Francisco Bay Area. It is preferable, but if you’re not there, you can also apply and you’ll learn all the details about the jobs. That will be somewhere in the middle of the podcast.

Kirill Eremenko: So, make sure to listen to that if you’re looking for a job in data science or you’re interested in this company, which you will be after this podcast. It sounds like a very exciting place to work. Then you can hit Sinan up or apply for their jobs directly.

Kirill Eremenko: Then we talked about sharing deep learning models in the world. We touched briefly on things like Flask, Django, Docker containers, Kubernetes, and then we dove into the world of natural language processing. This whole second part of the podcast is about natural language processing.

Kirill Eremenko: You will learn about state-of-the-art NLP frameworks such as Google’s BERT, which has been the top of the talk for everybody in 2019. You’ll also learn about SQuAD, Word Masking bidirectional, [inaudible 00:04:58] bidirectional, why BERT is bidirectional, what that means. You’ll learn about the transfer theory, conversational design, and many, many more topics. 

Kirill Eremenko: In a nutshell, epic podcast. Can’t wait for you to check it out. You’re going to love it. So, without further ado, let’s dive straight into it, and I bring to you without further ado nobody else but the legendary Mr. Sinan Ozdemir.

Kirill Eremenko: Welcome back to SuperDataScience Podcast, ladies and gentlemen. I’m super excited as you can probably hear from my voice to welcome for the third time around the one and only, Sinan Ozdemir. Sinan, welcome, my friend. How are you doing?

Sinan Ozdemir: I’m doing great. Thanks so much for having me. I’m always happy to be here.

Kirill Eremenko: It is so cool. I don’t know. I just have this amazing feeling every time I talk to you. You just have some great energy about you.

Sinan Ozdemir: Well, thank you.

Kirill Eremenko: Yeah. Why? Why do you think that is?

Sinan Ozdemir: Well, that’s actually a really interesting point that you bring up because as listeners of the podcast may or may not know from my other times on here, I actually come from academia. I was a lecturer at Johns Hopkins, where I was teaching computer science and machine learning. I always knew that my favorite part about working at Johns Hopkins was actually the teaching, and more specifically, teaching people who had no idea what I was talking about, and then making sure that by the time they were done with the class, they actually understood and can hold a conversation about it.

Sinan Ozdemir: I think with data science, especially because it’s such a new field, there’s so few people who are majoring in it, getting to degrees in it, it’s really satisfying to talk about data science because it’s a topic that a lot of people don’t understand, and it’s a topic that a lot of people don’t really know what the right questions to ask are.

Sinan Ozdemir: So, every time someone like you or someone else, they ask me about, “What is data science and how does it all work?” I get really excited because I get to explain something that they don’t really understand and, hopefully, they’ll walk away understanding it.

Kirill Eremenko: Fantastic. Oh, that’s a great way to put it. Now, with the recent developments, you can explain to people business stuff as well. You just had your company acquired. Congratulations on that.

Sinan Ozdemir: Thank you. Thank you very much. Yeah, yeah. My company Kylie.ai was recently acquired by Directly, and we’re very, very excited about it.

Kirill Eremenko: It feels like yesterday, even though I realized it probably was one or two years ago when you were on the podcast the second time, when you were very passionate on telling us about Kylie.ai and what you’re building there. It’s still cool to see the success and acquisition is a great thing. It means a big company is recognizing what you’re doing, and you now have all these leverage through the other company to really impact even more people. Tell us a bit about the process. How did that all happen? You were building your business. Did you plan on going through an acquisition? How did they get in touch? What happened?

Sinan Ozdemir: Yeah. So, for us, specifically, and this is a mentality that I hold, as well as my co-founder of Kylie, we never really went into this hoping for an acquisition. The plan wasn’t to get acquired or something like that. The plan was to build a solid business. The idea was that if you build a solid business, and that means going through understanding your market, understanding your profit, your expenses, your revenue, understanding all of that, building a solid business eventually will lead you down the path that you want.

Sinan Ozdemir: So, our mentality was always, “Let’s pretend or let’s work towards that we’re going to IPO. What would it take to become a public company? Let’s make all the right choices along the way. If something happens, if we catch their attention and they look at us, and they are interested in acquisition, let’s have that conversation, but let’s not make every decision as if the whole point is to be acquired, because I think that’s when you start making decisions, where you start favoring things like growth of user base over growth of revenue, steady growth of revenue. So, we always make every business decision as if we were going to be doing this for 20, 30, 40 years.

Kirill Eremenko: Wonderful. Wow! So, how big did you manage to grow the business? How big was the team? How many clients did you have? In general, what triggered this interest from Directly?

Sinan Ozdemir: Yeah. So, the interest was triggered by … So, at the time we were hovering around 15-20 people, and we were servicing some very large like some telcos, and some retail brands. What really caught Directly’s eye was the fact that what we were offering was full conversational automation with robotic process automation or RPA.

Kirill Eremenko: Wow!

Sinan Ozdemir: Yeah. So, Directly is actually in the business of customer success automation as well with human in the loop AI. With the acquisition of Kylie, what that really brought to their business offering was full end-to-end conversation automation with that automation of backend processes as well. So, the idea was always with Kylie. Customer support isn’t just a conversation. It’s also about the actions that take place throughout that conversation.

Kirill Eremenko: Yeah. Sorry. It’s like someone putting in the details of the customer, making some notes along the way, putting some flags here and there, adding them to a segmentation, writing up some notes after the call, all that stuff, right?

Sinan Ozdemir: That, too, but also even during the conversation looking up information about the user to guide the conversation. A good example that I like to give is let’s say you’re calling in or chatting in to your internet provider, and you have a question about your past bill. Maybe your question is, “Why is the bill so high?” or “Why am I getting charged for this thing?”

Sinan Ozdemir: In the moment, the agent has to look up, first of all, who are you. Who is this person chatting in? What is their account status? To answer your question, I have to be able to see your bill in front of me. So, even during that conversation, the agent or bot on the other side has to be able in realtime look up that information and then use that information to answer the question.

Sinan Ozdemir: That’s actually that next generation of conversational AI that Kylie was offering to its clients. So, it’s not just whenever the poster or the user speaks, how does a chat bot respond. What we were really about was, “Well, how do we respond with the context of all the information that the client has to offer?”

Kirill Eremenko: Very interesting. Putting the conversation into context on the fly pretty much, right? As soon as they call, you already have the information up there.

Sinan Ozdemir: Exactly because if someone says, “I have my problem with my account,” and that could be for a number of reasons. Maybe they’re not currently a customer. Maybe their account was locked because their credit card was stolen. Maybe their account was shut down because someone shut it down a month ago. Having that context really disambiguates a person’s question, and it’s actually something that user may not even know themselves. All they know is, “I can’t log in,” but they don’t know why. The agent or bot on the other side can actually find out why and use that information in the conversation.

Kirill Eremenko: That’s really powerful. I can totally see how that adds business value to all your customers. Can you tell us a bit more, especially for our listeners who haven’t heard the previous podcast, by the way, if you haven’t, we’ll link to that in the show notes, what role does data science play in this product? Because we can really gauge how data science works in the product, but if you could describe in a bit more detail, please.

Sinan Ozdemir: Of course. So, data science, obviously, is a very big term, and there’s a lot of subsets of data science that go into a product as advanced as conversational AI with RPA. That really ranges all the way from the more analytic side, where understanding the client’s conversations and just knowing things as simple as volume and how volume ships throughout the day, and what kinds of questions are coming in all the way to using deep learning and transfer theory to really understand the natural language coming in and to generate a response back.

Sinan Ozdemir: So, we really run the spectrum between analytics all the way to deep learning and transfer theory to make sure that we are delivering state-of-the-art natural language processing, generation, and understanding, and making sure that we have the insights for our clients to understand what’s happening because a lot of the times when companies are deploying these really big behemoth deep learning models, they don’t often come with this insights platform as, “Well, how do I convey what our deep learning model is doing to our clients? How do we build that trust?”

Sinan Ozdemir: Because these days when someone says, “We’re using AI. We’re using deep learning. We’re using such and such,” sometimes there are people who will look at that and say, “Well, hold on a minute. What exactly are you doing with this data? What exactly are you doing with this model?”

Sinan Ozdemir: So, we’re really trying to build that trust with not only our clients, but our clients’ customers, making sure that everyone understands how the AI is working and what controls we do have over the AI systems.

Kirill Eremenko: That’s a very interesting question because it has been in 2019 and I think even more so will be in 2020 and beyond a central topic AI, explainable AI, and the implications whether ethical or operational of having non-explainable AI, and it’s really cool always to hear when a company manages to get one step closer to explainable AI. Do you mind sharing, of course, if you’re able to share how do you explain or how do you facilitate that explainability of your deep learning models?

Sinan Ozdemir: Of course. Yeah, it’s really not a secret. I don’t think there’s one single way to turn your deep learning models into an interpretable system. By the way, interpretability is one of my core tenets of data science because it’s really important to know how other people are using data around you.

Sinan Ozdemir: So, as far as deep learning goes, I think there is this gray area where data scientist can simply say, “Oh, it’s just a bunch of matrices being multiplied together. There’s really no way to know what’s happening.” I think that’s an excuse that data scientists can use to say, “Well, I built this great model. I don’t really have to explain how it works,” because that’s too difficult. I think that’s not really the right way to approach it.

Sinan Ozdemir: I think even for deep learning, you have an opportunity to say, “Well, here are the inputs. Here are the outputs. Here’s the training data. Here’s where we got the training data. Here’s what we did to the training data to make it more readable to the machine learning model.” There’s so many steps in between not having deep learning, and having deep learning that you can explain along the way.

Sinan Ozdemir: Something as simple as where does the data come from can answer sometimes a majority of our clients’ questions because even that is a mystery to some people is, “Well, where does all of this data come from? How do you learn all of this? Where does that information and insight derived from?”

Sinan Ozdemir: So, it’s not always just about how does the model work, but sometimes it’s as simple as, “Well, where does the data even come from, and then what do you do with it, and then how do you feed it back in to the system to update later on?” 

Kirill Eremenko: Yeah, and does the data have bias, which to your point, what source it comes from. Maybe you’re originally getting the data from a source that has bias inherent in it, and then, therefore, you’re training your models based on bias data.

Sinan Ozdemir: Yeah. Bias was actually a big part of the topic game at SuperDataScience last year is, how do you know where the data is coming from? How do you know that the source is valid? How do you confront those biases and resolve those biases?

Sinan Ozdemir: A big thing that Directly is actually doing is curating a network of subject matter experts around the globe to help understand and resolve those biases in our clients’ data. So, we are really working hard with humans and AI together to resolve a lot of those biases that are fed into the models.

Kirill Eremenko: Very interesting. So, I love that. What are these subject matter experts? Again, if you can share just probably some sense of information, but to the extent you can share, what do these subject matter experts do, and what kind of data they’re looking at?

Sinan Ozdemir: Of course. Again, this is not a secret. This is actually one of the core differentiators that Directly offers as our expert intelligence platform. What that really means is we are working with people around the world who actually have, they have knowledge and a deep subject matter expertise in our clients’ offering.

Sinan Ozdemir: So, a really simple example is one of our bigger clients is Airbnb. What we actually do is we work with Airbnb super hosts around the world, people who use Airbnb daily just on their own, and then we go to these people and say, “Hey, listen. Here is some data from Airbnb like an intent matching problem.”

Sinan Ozdemir: For example, if a user of Airbnb chats in and says something like, “I need to rebook because my host canceled on me,” or something like that, we work with hosts who say, “Well, I’ve been there. I understand this problem.” So, we’re really making sure that our intent matching, our data labeling, our conversational flows are being audited and looked at by real people who understand our clients’ offerings, not just Directly employees, but people who really understand how Airbnb works.

Kirill Eremenko: Wow! I am listening with all ears. This is so interesting. You do not only get to work on AI, but you also work with real people to tailor that. That’s probably living the definition of a synergy between artificial intelligence and human intelligence. That’s so cool.

Sinan Ozdemir: Absolutely. Yeah. Really, it’s one of those things where I’ve spent a majority of my data science professional career teaching about how do we as data scientists find data, curate data, work with data, process data, model data that sometimes what gets lost in that mix is, well, the data comes from somewhere, and usually that somewhere is humans.

Sinan Ozdemir: So, at Directly, it’s really important to us to create that synergy between humans and AI because if you have AI without the humans, you start to see that degradation. Interpretability becomes difficult. It starts to become unruly. So, working with our expert network is really what differentiates not just our business offerings, but our AI as well.

Kirill Eremenko: Fantastic. That was a very clear example with Airbnb that you gave. Are you able to share another example, maybe from a different industry?

Sinan Ozdemir: Yeah, of course. What’s really easy about that is this really works across several industries and domains. So, Airbnb is one, but another one that is pretty fresh in my mind from the work that I have been doing is Microsoft. Now, Microsoft, being another client of Directly-

Kirill Eremenko: That is so cool. Congrats. Such big companies. That is so exciting.

Sinan Ozdemir: Yeah. Of course. Thank you so much. With Microsoft, it’s actually in some ways a better example because Microsoft has so many offerings, right? They have LinkedIn. They have OneDrive. They have all of these different unique product offerings that all require a different touch to their customer support, and they all require a different touch from their AI as well.

Sinan Ozdemir: At Directly, just like at Kylie, we really focus on specialized company-specific AI models. So, each one of our clients and each one of their product offerings can have a very granular level of AI, and a model that’s curated especially for them. So, with Microsoft, for example, their OneDrive customer support and their LinkedIn customer support models can be very different because someone is saying, “I can’t log in to my LinkedIn account,” versus “I can’t log in to my OneDrive account,” may have very different answers depending on the type of product that they’re using.

Sinan Ozdemir: For LinkedIn, it maybe as simple as, “Here’s this website. Here’s how you figure out how to log in.” For OneDrive, it could be more complicated. It could say, “Well, you’re going to have to come back in at this time, and do this, and do that.” So, the answer may change even though it’s all under the big umbrella company Microsoft. So, it’s really important for us to understand not just at a company level, but at a product offering level how the AI is going to be different between them.

Kirill Eremenko: That is crazy. I was just thinking that you’re moving not just from company-specific AI models to product-specific AI models, you must have a billion people working in your data science team. Where do you get the time to build all these models?

Sinan Ozdemir: Well, it’s really a factor of understanding that there isn’t going to be some AI model that will work for every situation, every time of day, for every language. It’s really about understanding what are the best types of models for different situations. So, you don’t need a billion people to make this work. You need a few really, really smart individuals, us, like the people on my team, really, really smart individuals who understand it’s not just about, “How do we build this gigantic deep learning network that will understand anything at anytime, anywhere?” It’s really about, “Well, how do we understand our clients’ specific needs, and then how do we deliver AI that is right for our clients?”

Sinan Ozdemir: So, our Airbnb models, and our Microsoft models, and our Samsung models all might be very different from one another because they’re all trying to answer different questions.

Kirill Eremenko: Is there a company that you don’t work with?

Sinan Ozdemir: There’s probably a few out there. We’ll get them, though.

Kirill Eremenko: That’s awesome. Well, fantastic. Congrats on that. Sounds like a very exciting space to be in. Help me understand, though. So, you build a business where you are the founder, you’re co-founder. There’s two of you. You had 16 people in the team. Then along came Directly. You agreed to the acquisition, and you stayed with the business. So, obviously, I guess there’s usually a choice whether you leave, you just sell the business and you leave or you stay with the business. Why did you stay with the business and what is your new role in this company, in Directly?

Sinan Ozdemir: So, you’re right. There is usually a choice. There’s usually you say, “Well, I’m done. I’m going to walk away,” or you can stay on with the acquiring company. The reason I stayed on with the acquiring company, the reason I stayed with Directly was because their product offering, their roadmap, their vision for using AI in customer experiences aligns so much with Kylie that the acquisition really felt more like a merger, right?

Sinan Ozdemir: We had product offerings that they were hoping to build in 2020, and they had this network of experts that we knew were going to be so beneficial to our AI models. It just felt like a perfect match.

Sinan Ozdemir: So, really, for me and my co-founder, it really didn’t feel like we have to choose between working with them or leaving. We really wanted to work with Directly. It just felt so natural and like a perfect match for our teams, and for our AI, and for our product offerings. So, it was really, really nice to have that matchup together.

Kirill Eremenko: Nice. Did you get to meet the executive team for the acquisition, like feel for what kind of people they are?

Sinan Ozdemir: Yeah. For anyone out there who is unfamiliar with acquisitions in the startup space or maybe they’re thinking about doing one of their own, I highly recommend not just meeting the executive team before an acquisition, but really get a feel for what it’s like to work there.

Sinan Ozdemir: One of my favorite things that Directly did while they were looking at us was every quarter, Directly has a hackathon, where they invite every one of the company, not just engineers to work on a project or piece of code or something that they would not get to work on normally. While we were going through the acquisition, while we were still in talks, Directly had their hackathon, but they invited us to their hackathon. 

Kirill Eremenko: Awesome.

Sinan Ozdemir: At first I said, “Well, we don’t work there yet. We don’t want to be imposing.”

Sinan Ozdemir: They said, “Absolutely not imposing. We want to see what it looks like to work with you, guys.”

Sinan Ozdemir: So, we actually ended up joining hackathon teams and working with Directly employees to get a sense for what it would feel like to work together. So, that’s actually one of my favorite things that Directly did. It wasn’t just about, “Okay. How much more revenue will we get? Okay. How many patents are we going to receive?”

Sinan Ozdemir: For them, it was really more about, “Well, how do we work with these people? How does Sinan fit in to our team?”

Sinan Ozdemir: To answer your other question, I recently come on as their Director of Data Science. So, my role has really shifted from how do I build this product, how do I offer this AI model, this data science platform to the world, and that shifts to, well, I still do that, but now, I get to think about, “How do I bring data science to the rest of the company? How do I democratize machine learning and AI to a point where anyone at Directly feels comfortable talking about what our machine learning models do for our clients and for the market?”

Kirill Eremenko: Well, congrats, first of all, on the huge role. That’s massive at a company-

Sinan Ozdemir: Thank you.

Kirill Eremenko: … that large and that’s working with such great customers, that’s very responsible. Also, the description of the acquisition, amazing, amazing. I’m learning so much just by talking about this. One thing I wanted to understand, first of all, how big is your team in Directly as the Director of Data Science?

Sinan Ozdemir: Yeah. So, the way that Directly is set up is we have our data science resources spread out among several of our teams, our engineering teams. So, on my team, I have people who are data scientists, machine learning engineers, but I’m also working extremely closely with the analytics team. While they may not be directly a department under data science, they are still, in my mind, doing exactly the same things a data scientist would do.

Sinan Ozdemir: So, my team is very broad. I have about probably five to 10 people at the company, who are in some way performing data science tasks, and who are actually doing the analytics and the machine learning behind the scenes. We’re still looking to grow that team. We’re going to be hiring relatively soon for more data engineers, and more data scientists, and more machine learning engineers because we’re always trying to make sure that we’re staying at the top when it comes to delivering that state-of-the-art NLP.

Kirill Eremenko: That’s amazing. That’s really great. That’s already a decent-sized team, five to 10 people. It’s exciting to hear an example. This has come up on the podcast before. We have companies, which choose to have a centralized data science team. All the data scientists sit together, and there’s these companies that choose to have an integrated data science team, where the data scientists are spread out, but it sounds like yours is more on the integrated side of things where you have data science representatives within individual product areas of the business.

Sinan Ozdemir: It is much more-

Kirill Eremenko: I wanted to … Yeah?

Sinan Ozdemir: Sorry. It is much more in the integrated side, but at the same time, we also find a lot of value in working together. So, one example of that is every two weeks, we have what’s called a journal club. For those of you who are getting masters or PhDs know what a journal club is, is every two weeks we decide on an academic paper that came out in the last year. We all read it, we all digest it, and we all have to come with examples of how could we use this paper at Directly.

Sinan Ozdemir: So, we’re always reading what’s latest and greatest, and we’re always thinking about how do we apply this to the company. It’s really my personal conviction to say a data scientist is neither just on the business side or just in the research side or just in the software side. We have to be able to understand each other’s language.

Sinan Ozdemir: I want people who have PhDs in math thinking about how could we use this to really ramp up our devops for machine learning. So, I really want them really cross-functionally thinking about data science, not just sticking to “what they do best”.

Kirill Eremenko: Amazing. Totally amazing. I think this is a good time to do a recruiting plug because before the podcast, you mentioned you will be hiring.

Sinan Ozdemir: That’s right.

Kirill Eremenko: You’ve also mentioned it now. Tell us about it because we’ve got 10,000 data scientists listening to this. Let’s get them sending you their resume.

Sinan Ozdemir: Absolutely, yeah. So, Directly is hiring, and for more things than just data science, but we are hiring for data science, and for me, that’s super exciting because I always love growing my team, and I love getting different perspectives about data science. So, what we’re looking for are people who are ready to work with natural language processing people who are experienced and people who are ready to get their hands dirty with the latest and greatest in deep learning for natural language processing, generation, and understanding.

Sinan Ozdemir: So, anyone out there who is interested in working with a really awesome tech startup in San Francisco and is really excited about working on these really interesting problems that involve not just conversational AI, but also in robotic process automation, how do we automate these backend processes to empower these automated conversations, we want you. We want you to check us out.

Kirill Eremenko: Man, you’re making it sound so exciting. I want to work for you. This is so cool, especially the work, NLP, deep learning, RPA, the cutting edge discussions about cutting edge technology integrating to the business. The culture sounds fantastic. Do you guys hire people remotely or do you have to be based in San Francisco?

Sinan Ozdemir: Thanks for asking that. We don’t have a specific policy on whether you have to be remote or in San Francisco. We do have remote workers. We do tend to prefer that people are in the Bay Area, and that’s usually just so that we can start, to your point, to have that culture, especially at such a young or at a young stage of a startup. It’s really important for us to make sure that our culture is being build as well as we possibly can, but we do offer remote positions.

Sinan Ozdemir: So, if you’re thinking, “I don’t live in San Francisco. I can’t even apply,” don’t think that way. Please, please do apply even if you don’t live in the Bay Area.

Kirill Eremenko: Fantastic. Indeed. If you’re really good at NLP, you’re really passionate about NLP or there’s somebody really passionate about NLP that comes to you and says, “Hey, I live in Budapest,” or somewhere else, why wouldn’t you hire them, right?

Sinan Ozdemir: Yeah, absolutely.

Kirill Eremenko: You’re getting a super talented person from the other side of the world.

Sinan Ozdemir: Absolutely.

Kirill Eremenko: Fantastic. Well, I hope everybody is excited. How do they get in touch, Sinan, just so we put that onto risk?

Sinan Ozdemir: Of course. So, on the website, directly.com, we do have a page full of all of our positions that we’re hiring for. Again, like I said, it’s not just data science. We are also always looking for full stack backend, front end engineers. Even if you have a data science mind, that’s even better, right? I love working with full stack engineers who have built functioning websites around machine learning.

Sinan Ozdemir: I think in today’s age, that’s really important. As more and more people are switching to containerized applications and working with things like Amazon is the last container service or Kubernetes, it’s really important for a data scientist not just to understand the models, but understand, “How do I take these models that I am building and deliver them to the world with high availability and low latency?”

Sinan Ozdemir: So, it’s not just machine learning engineers, and statisticians that we’re looking for. I’m really looking for people who have data skills, and also run the models themselves.

Kirill Eremenko: Okay. So, people who have data skills, can run the models themselves, very, very exciting times. Yeah. What was I going to say? Natural language processing and RPA combined, what a really cool… This what I was going to say that people listening, I love this stuff, people listening to this podcast, just if you’re going to apply, just put in the application that “I heard Sinan on the SuperDataScience Podcast”, and right away, they know the Director of Data Science. How much ahead are they compared to other people who apply who haven’t heard you?

Sinan Ozdemir: Right.

Kirill Eremenko: Crazy. Hmm. Another way to get the job is come to DataScienceGO 2020. Sinan is going to be presenting there, doing a workshop for advanced practitioners, and maybe more. So, just meet Sinan at DataScienceGO, and give him your resume in person.

Sinan Ozdemir: That’s right. That’s right. No, I love that. People always say, “I hate being handed resumes. I hate it when people come up to me when I’m doing something else.” I love it. I mean, I used to be a teacher. I had open office hours. I love it when people come up to me and say, “Hey, I heard you on the podcast. I just wanted to give you my resume,” or “I just want to send you my business card.”

Sinan Ozdemir: To me, as someone who is both an entrepreneur, a data scientist, and a teacher, I love meeting people, and I love when people show that initiative. I just love talking to people about what they love to do. So, please, yes, come up to me and hand me a resume. I am fine with that.

Kirill Eremenko: Fantastic. Thanks, man. Speaking of DataScienceGO, first of all, thanks for accepting the invitation. Very excited.

Sinan Ozdemir: Of course.

Kirill Eremenko: What is it also, and what your workshops are going to be about or your talk? Any ideas?

Sinan Ozdemir: So, you know what? I have so many ideas, and I’m not just saying that because I have no ideas, but I really do. I’ll workshop a few of them right now with you. Maybe you can give me some of your feedback.

Kirill Eremenko: Sounds good.

Sinan Ozdemir: One thing that I really want to start talking about more is, like I said before, how do you take those models that you’ve built? These great models they have great metrics, they’re performing well, and they work in your Jupiter notebook, maybe, but how do I deliver that model to the world? How do I put it in a place where people can use it whenever they need to use it? How do I build APIs around my models? How do I build websites around my models?

Sinan Ozdemir: I think that might be something that I’ll explore in teaching people not just how to build those machine learning models, but how do you actually integrate them into your systems. How do you actually deliver them to the world?

Kirill Eremenko: Very cool. I like that.

Sinan Ozdemir: Yeah. I like that. That was really good.

Kirill Eremenko: I like that idea.

Sinan Ozdemir: Yeah.

Kirill Eremenko: Yeah. So, what tools would be involved in that?

Sinan Ozdemir: So, for that really depends on the way we want to take it, but we’d probably want to learn about some web frameworks, maybe like a Flask or Django. We’d also learn about Docker, and Docker containers, and containers in general. We’d also want to talk about how to deploy those containers using something like Kubernetes, and how to get to the cloud.

Sinan Ozdemir: There’s this whole pipeline of how do you serve up machine learning models. To get one step further, once it’s in API, how do you build a website on top of that? How do you build a Chrome extension that can actually call that API, so people can use it in realtime?

Sinan Ozdemir: So, I think that’s one of the routes I’m thinking of going. The other route was really just addressing this renaissance in natural language processing focusing on things like BERT, transformer architecture, GPT2 and really diving in not just into the inner workings of these deep learning models, but really, what is their best used case in the real world?

Sinan Ozdemir: So, I think that’s something that sometimes can get glossed over is, “Wow! There’s these great models out there. There’s BERT, there’s ELMo, there’s all of these different models coming out, but what do I do with them? How can I even begin to use such complicated models when I don’t have a PhD in whatever?” I think that would be something that I would want to really dive into is, how do you actually use the latest and greatest in NLP for what sometimes maybe considered even the simplest of problems.

Sinan Ozdemir: So, I think I’m between the devops, the dockerizing, the serving up machine learning architecture versus really diving deep into the latest and greatest in NLP, and using that to build the next generation of natural language models.

Kirill Eremenko: Wow! That’s such a tough choice. Both are amazing. Actually, both sounded like huge workshops. Are you sure you can cover all like Flask, Django, Docker, Kubernetes in two, three hours?

Sinan Ozdemir: I think we can do it. I think for the people who really want to learn it, I think we can do it. The reason I think that is because it all works together so well. I think that if we need more than three hours to go through at the very least the high levels of how it all works with an example and actually building one ourselves, I think it’s not being explained the right way.

Sinan Ozdemir: I think there really is this kind of natural flow between building out a model and serving it up in Kubernetes that I think really can be addressed in two to three hours.

Kirill Eremenko: Yeah. Amazing, amazing choices. It’s going to be really hard to pick, but I guess we’ll find out, and we’ll all see at DataScienceGO 2020. So, now, let’s switch gears a little bit, and talk about natural language processing. So, BERT, it’s been in the air for a while now. Everybody is talking about BERT. What is BERT? What is it used for in a nutshell?

Sinan Ozdemir: Yeah. Of course. So, BERT is one of those latest and greatest NLP models that I was talking about before, and the papers that originally started talking about BERT, which, by the way, stands for Bidirectional Encoder Representations from Transformers, but that’s not really that important. I think the first papers came out late 2018 about it.

Sinan Ozdemir: What it is, is a language modeling architecture. So, what that really means is it’s a way of pre-training a deep learning network to take in text, whether it’s English or multilingual text, taking in that language, and representing the context as a vector.

Sinan Ozdemir: So, if you think about it, and for those of you who are working with something like a scikit-learn or TF-IDF vectorization, what those do in a nutshell are not so different. They’re taking in text, and they’re outputting vectors that represent that text.

Sinan Ozdemir: BERT and other architectures like it, especially transformer architectures, are trying to do something very similar. They’re trying to take in texts and represent that text as a series of numbers, as a vector, and those vectors can then be used to train or fine tune a new model for a downstream task. When I say downstream task, I mean like a classification problem or a sentiment analysis or something like that.

Sinan Ozdemir: So, BERT is really a way to take in language, words, strings, sentences, phrases, and represent those pieces of texts as a vector because as we all know, machine learning models don’t work well with text. They work well with vectors. So, that’s that mapping from texts to vectors. It is all important in natural language processing.

Sinan Ozdemir: So, BERT is one of those latest and greatest that are able to take in text and output context, output vectors that have been used to achieve state-of-the-art results in a bunch of natural language processing tasks like SQuAD.

Kirill Eremenko: What is SQuAD?

Sinan Ozdemir: SQuAD? Oh, yes. Well, SQuAD is the Stanford Question and Answering Dataset. So, it’s basically a problem where you use a model to ask a question, and give it a paragraph that has the answer to that question in it, and it’s up to the deep learning model to say, “Where in this paragraph is the answer to the question?”

Sinan Ozdemir: It’s one of those datasets that are used pretty widely to illustrate how well their language modeling works is. BERT was able to achieve a metric or achieve a performance on the SQuAD dataset that outperformed many of the former state-of-the-art models.

Kirill Eremenko: Hmm. Okay. Wow! That’s very cool. So, help me understand, please. Is the main difference between BERT and other language, because the reason I’m asking is this idea of vectorizing words has been around for ages, probably decades, but is my understanding correct that the main difference between BERT and the prior existing models is that rather than vectorizing words, it actually vectorizes context?

Sinan Ozdemir: So, I think there’s a way to think about it where previous or more simple text vectorization problems are really looking at the tokens, the words themselves, and assuming that individual tokens are representative of the context. So, I think both types of models are trying to map context correctly.

Sinan Ozdemir: What BERT is doing or one of the things that it’s doing to achieve those state-of-the-art results is it’s working in a bidirectional format, meaning, it’s basically reading the text both left to right and right to left. That sounds, “What does that mean?” Right? Like, “Who cares?”

Kirill Eremenko: Yeah. Yeah. Why would you do that?

Sinan Ozdemir: Exactly, but what that really means is that the language model is trying to understand, “Well, I both want to know what the words mean, but I also want to know what the words mean when they’re stringed together one at a time.”

Sinan Ozdemir: So, if I were to say, “I like this dog,” if I read it from left to right, as a human, we obviously can understand that context. If we were to pick out the words and say like and dog, those words by themselves don’t really mean much to the sentence overall.

Sinan Ozdemir: What BERT is trying to do is say, “Well, let me take in this phrase. Let me read it left to right and right to left, and then basic try to understand not only the words that are being used, but the combinations of words and the sequence of words.

Kirill Eremenko: So, it tries to understand the sequence of words. So, reading backwards, it would be dog this like I. How does that add value to the algorithm reading it backwards? What additional insights does it get from that?

Sinan Ozdemir: Of course. So, the way that BERT is trained is really the key here. So, the way BERT is trained is what’s called Word Masking. Basically, what that means is our training set, our phrases input text, and we randomly take out words. The goal of the model is to say, “Given the rest of the sentence, what word should be here?”

Sinan Ozdemir: So, I may have said, “I like this blank because it is a very loyal companion.” The goal is to predict the word dog by using the words to the left and to the right. So, by reading it in a bidirectional fashion, you understand, well, if you read it left to right, “I like this blank,” that alone is not going to tell you the answer to the question.

Sinan Ozdemir: If you read it from right to left, you would say, “Because it is a loyal companion,” and you say, “Oh, okay. Loyal companion sounds like it’s probably a dog.” So, understanding what’s to the left and to the right of that missing word really helps put that one word into context.

Sinan Ozdemir: So, the professional nature of the architecture helps the model to understand anything around the words in the sentence, and the way that the model is trained with Word Masking really helps the model understand, “How do I simply pick up words and move them around without changing the meaning of the sentence itself. So, it’s really both about a bidirectional architecture and the Word Masking as a training.

Kirill Eremenko: Wow! Blowing my mind here. That is amazing. What do you do in cases like what you probably experience at Directly where you have live conversations, where the text, the whole sentence is not available, so you can’t really read it right to left because the sentence is not finished. Somebody is still talking or they’re still typing it up or the conversation isn’t over? Does it work there as well?

Sinan Ozdemir: So, I think the way to really think about it is BERT as a tool is used to model input text for any number of downstream tasks. So, you could basically take BERT and say, “I’m going to use a pre-trained BERT. So, at BERT that was trained on, let’s say Wikipedia or Twitter or some large corpus. I’m going to take that BERT and then I’m going to train a separate task, which may be what is the next word in the stream of thought.

Sinan Ozdemir: That’s what really transfer theory is all about. Transfer learning is about, “How do I take this model, which has already been trained on one dataset, and then use that model to train a task for a different dataset?” So, BERT can be used to do natural language generation even though BERT itself is not a natural language generation type model.

Kirill Eremenko: Okay. Wow! That is really insightful. So, that’s what the whole … What did you call it, though? Oh, the transfer theory. That’s what the transfer theory is all about. Very interesting. So, do you guys use this at Directly?

Sinan Ozdemir: Yeah. So, BERT is actually one of the many types of models that we deploy to our clients. I believe I said earlier, every single one of our clients’ product offerings have their own unique and different machine learning and AI model. BERT is just one of those many options.

Sinan Ozdemir: So, we use BERT and we’d also use some other types of multilingual transformer architecture. So, BERT is not the only thing that we are doing at Directly, but it is one of the state-of-the-art models that we are offering to our clients.

Kirill Eremenko: Got you. Well, Sinan, completely amazing conversation. Totally loving it. We’re running short on time, but I do want to ask you one more thing.

Sinan Ozdemir: Of course.

Kirill Eremenko: That is, we’re heading into 2020, what is your prediction for NLP, for the future of NLP in 2020?

Sinan Ozdemir: I think in 2020, what we’re going to start seeing, I mean, we’re already seeing it, but what we’re going to see a lot more of is integration of automated conversations not just in our Alexas, and in our phones, but we’re really going to start to interact with these automated conversation systems at work.

Sinan Ozdemir: We’ll start interacting with them in shopping malls. We’ll start interacting with them in places where we previously didn’t really think we wanted them. That’s going to be both good and bad. I think companies who are working diligently on curating and creating these conversational experiences with AI really have to think about, as I always come to back to you, the context.

Sinan Ozdemir: So, I think we’re going to start seeing AI in new places where concerns about privacy and bias are going to come up, and it’s really up to the data scientists and the data practitioners to alleviate consumers’ troubles and fears, and really make sure that everyone trusts and is comfortable with the AI that they’re interacting with.

Kirill Eremenko: Wow! That is so true. I was presenting at a conference recently for L&D managers and leaders about the future of artificial intelligence, and just generally what AI is. One of the top trends was natural language processing, and then later, a few weeks later, one of the people that I networked with there, an executive or an L&D manager or leader, they emailed me asking me if I knew any good conversational designers.

Kirill Eremenko: I was like, “Is that even a job?”

Sinan Ozdemir: Yeah, it is.

Kirill Eremenko: Now, yeah, I looked into it a bit more, indeed. Yeah. For all these chat bots, for all these, as you said, conversational experiences, now, we need design. This really ties in to the whole point like how people are, “Oh, AI is taking over jobs. We’re losing jobs.” Well, according to the World Economic Forum Research from 2018, for every job that AI displaces, there’ll be 1.7 new jobs that will be created.

Kirill Eremenko: This is a real life example. Conversational designers, seriously? Those words never came up in that sequence before three years ago or before that. Now, it’s apparently going to be a big profession.

Sinan Ozdemir: Well, it’s funny you mentioned conversation design because, actually, we were just talking about it at Directly. The idea of conversation design has been around for a while. Humans have been helping other humans have conversations more effectively, but to your point, now, we’re going to start to have humans who are helping AI have more fluent conversations.

Sinan Ozdemir: So, these jobs are shifting between helping humans versus AI, but the concept has always really been around for conversation design. Now, it’s going to become even more relevant as companies create more and more automated conversational experiences. We have to make sure that they are fluent, that people are comfortable with them, that people actually want to talk to them. If you build an AI that no one wants to talk to, it could be useless in some sense of the word.

Kirill Eremenko: Exactly. Some companies I call up, and it’s just a terrible experience sitting on the phone, waiting for a human to reply, this music playing in the background or even worse an advertisement playing. That’s ridiculous, right? How much better will everybody’s life be when we have AI doing these conversations, and then humans who are no longer have to service a million customers and, therefore, there’s a huge waiting line?

Kirill Eremenko: Now, those same humans can train the AI and tailor those conversations whether it’s to Airbnb, to Microsoft, to Samsung. Whatever company you’re calling about, you’re going to have amazing experience. I think it’s a win-win for everyone.

Sinan Ozdemir: Absolutely.

Kirill Eremenko: Fantastic. Well, Sinan, it’s been a huge pleasure. Thank you so much for coming to the show. One question I do have for you before we go. What’s a book that you can recommend to our listeners that’s impacted your life in the past year or so?

Sinan Ozdemir: A book that I have not written, I assume?

Kirill Eremenko: Yeah. We will make sure to link to all of Sinan’s books in the show notes. He’s definitely worth reading, but, yeah, a book that you have not written.

Sinan Ozdemir: I’ve been thinking about this, and one thing that I’ve been reading a lot of, and it’s not a book per se, but recently, I’ve started signing up for as many data science and AI newsletters as I can.

Sinan Ozdemir: Now, that sounds almost like I’m inviting people to spam my inbox, but what I really get from that is a lot of different perspectives around AI and data science that I definitely would not have gotten in my day-to-day life. Even just today in my Medium newsletter that I get, the first article was about how to appropriately do hypothesis testing for machine learning performance evaluation.

Sinan Ozdemir: I go, “Huh. I’ve done that before, but I’m curious to see this person’s take on how to do it.”

Sinan Ozdemir: Then on a separate newsletter today, I’m reading about different facial recognition bans in different states, in different countries, and their reasonings for doing so. So, I’m getting not only this statistical idea, but I’m also getting this policy and governmental idea about AI.

Sinan Ozdemir: So, getting these different perspectives from these different newsletters I think is sometimes even more valuable than just reading a single book that has, potentially, one or a few perspectives.

Kirill Eremenko: Very interesting. So, what’s your favorite newsletter so far?

Sinan Ozdemir: My favorite newsletter, I mean, so far, has actually just been the daily Medium curation, the Medium articles. I think I get such a wide variety of people saying, “Here’s BERT from start to finish,” and then right under that is, “Here’s what it looks like to spin up a dockerized container in Kubernetes for machine learning.”

Sinan Ozdemir: So, you get this really wide variety of different people talking about different aspects in data science. I’ve really been enjoying the daily Medium newsletter. Actually, I try to read at least one or two a day.

Kirill Eremenko: Wow! That’s crazy. How do you find the time for that? That is insane.

Sinan Ozdemir: I’m a morning person. So, that helps.

Kirill Eremenko: Okay. Well, it’s interesting that you mentioned that because I actually … Oh, when was it? Two or three days ago, I was sitting reading newsletters myself, and I don’t do them everyday. You’re a superhuman if you can do it everyday, but I’m subscribed to two. One is the Abundance Insider. I think you’ll like it. It’s not just about data science and AI. It’s about more technology in the world by Peter Diamandis. It’s free, and it comes once a week on Fridays, and they curate the top five developments in exponential technologies in the world for that week.

Kirill Eremenko: The other one, it might sound like a plug, but I was actually reading our own newsletter, which we have at SuperDataScience. It’s at www.superdatascience.com/dsi for Data Science Insider. Our team curates exactly what you mentioned like the top five AI, machine learning, deep learning, whatever else developments in the world for the past week.

Kirill Eremenko: The value I see in these things, why I can relate to what you’re saying is there’s so much stuff going on, and so much hype, and some things that are just fake news or incorrect. Some things are insignificant and stuff. It’s much cooler when somebody curates it for you and like, “Okay. Sinan, here’s the top article for today,” or “Here are the top five for this week that you will probably be interested in.” It acts as a filter from all of these barrage of news that’s coming at you everyday. That would be my take on that.

Sinan Ozdemir: Yeah. No. I think you’re absolutely right. I think it’s about getting as many perspective as you can without overloading yourself.

Kirill Eremenko: Very true. Very true. You should check out ours, the Data Science Insider. I think you might like it. I’ll send you the link later.

Sinan Ozdemir: Sounds good. Perfect.

Kirill Eremenko: Okay. All right. Well, Sinan, once again, a huge pleasure. Thank you so much. Very, very valuable insights, and I can’t wait. I’m going to personally attend your workshop at DataScienceGO. Sounds like very exciting.

Sinan Ozdemir: Looking forward to it. Thanks so much for having me on. I can’t wait for the fourth time.

Kirill Eremenko: So, there you have it, ladies and gentlemen. That was Sinan Ozdemir. I hope you enjoyed our conversation as much as I did. This has been one of the best podcasts I’ve had. Sinan just got such a great energy about him. I love talking to him every single time. Plus, of course, the content is amazing. How cool is that? Learned about startups, acquisitions, explainable AI, case study, used cases, how they’re hiring, about BERT, NLP, state-of-the-art things. I learned so much about BERT. It’s crazy.

Kirill Eremenko: So, yeah, lots of favorite things. If you enjoyed the talk, make sure to hit Sinan up, connect with him. I’m going to share his LinkedIn and other places where you can find him in the show notes at www.superdatascience.com/333. That’s superdatascience, one word dot com slash 333.

Kirill Eremenko: If you know anybody who’s interested in natural language processing or who is looking for a job at a cool company such as Directly, then send them this podcast. Very easy to share. Send them www.superdatascience.com/333. Very easy to remember as well, triple three.

Kirill Eremenko: Finally, if you want to meet Sinan in person, make sure to get your ticket to DataScienceGO 2020 US. It’s happening on the 6th, 7th, and 8th of November 2020. Be there, meet Sinan and lots of other inspiring data scientists. The last time in 2019, we had people fly in from 25 countries from all over the world to connect and network.

Kirill Eremenko: So, this is the place to be. Sinan will be there. He’ll be running at least one workshop, maybe two or maybe a workshop and a talk or maybe a workshop and a panel. We will see, but definitely, you’ll get to meet him there and chat to him all about NLP, and hand him your resume if you want to.

Kirill Eremenko: On that note, thank you so much my friends for being here, for being part of this amazing conversation with Sinan. Huge shout out to Sinan. Thank you so much. A huge shout out to Directly for acquiring Sinan’s startup and sparking this amazing conversation. Thank you so much, everybody, and I’ll see you next time. Untill then, happy analyzing.