Kirill: This is episode number 10, with senior data scientist Yaw Tan.
(background music plays)
Welcome to the SuperDataScience podcast. My name is Kirill Eremenko, data science coach and lifestyle entrepreneur. And each week, we bring you inspiring people and ideas to help you build your successful career in data science. Thanks for being here today and now let’s make the complex simple.
(background music plays)
Welcome guys to this super exciting and special episode of the SuperDataScience podcast. You can’t imagine how excited I am to bring you this guest. And it isn’t a coincidence that this is our 10th anniversary episode and that I invited Yaw specifically to this episode because I wanted to make it super special.
Yaw is not just a senior data scientist, Yaw is my good friend and my mentor in data science. We met with Yaw a few years ago. We worked together when he was consulting for the company I was working at. I learned so much. And just to give you a quick understanding of who this person is, Yaw has done everything. He has taught at universities. He has been an entrepreneur in the space of data. He has led a credit risk/model validation department in a bank with 20-25 people working for him. He has been a senior data scientist at Pivotal, one of the most forefront companies in data science consulting. So he’s done pretty much everything you can think of in terms of data, and this episode is truly packed with value.
In this episode, we discussed how to create models, why models deteriorate, what a churn model is, what a credit risk model is. And you’ll see how those are used inside banks. We talk about data science, computer science, and cultural change as well. So Yaw is currently doing an Executive MBA at INSEAD, which is one of the most prestigious MBA providers, and he will share his experiences from there.
Also in this episode you will learn the 3 steps to assessing a model. So Yaw shares his 3-step methodology, or the 3 things that he thinks about when he’s assessing the performance and longevity of a model that had been built, whether it’s a churn model or a credit risk model or any other type of model. Also, Yaw shares his 80/20/80 rule. You’ve probably all heard of 80/20 in terms of data science efforts on data preparation and then data analytics. Well Yaw has a modification to that, which is called the 80/20/80 rule, which was extremely interesting to learn about.
Then we talk about data exhaust and also how to continuously learn to prevent yourself from becoming a technology dinosaur. And you’ll see from Yaw’s choices that he’s made in his career path that they have always kept him on top of the game and always his skills have been in line with what’s the most brand new, what’s the most cutting-edge technology, at the most cutting-edge algorithms in data science.
And of course, in this episode we’ll talk about some tools, we’ll talk about R and how it doesn’t parallelise. And we’ll talk about Pivotal R, we’ll talk about MADlib, we’ll talk about Python versus R, and you’ll get Yaw’s take on it.
So as you can see, this episode is completely packed with value. Can’t wait for you to check it out. And now I bring to you my good friend and mentor, Yaw Tan.
(background music plays)
Welcome everybody to this super special episode of the SuperDataScience podcast. Today I have my good friend and mentor Yaw Tan with me. Hi Yaw, how are you today?
Yaw: I’m great, Kirill, it’s great to be on the show!
Kirill: Thank you so much for taking some time. Yaw right now is in Singapore. How’s the weather in Singapore?
Yaw: Bright and sunny as usual. Lovely weather today.
Kirill: Same thing here in Brisbane. We’ve got a bit of cloud, but everything’s going well. And just to let the listeners know how we met, Yaw actually works, or used to work, for Pivotal. So you were a Senior Data Scientist there, is that correct?
Yaw: That’s correct. And I’m actually shifting my roles right now, so in a month’s time, I will actually be joining a bank working on financial crimes analytics. So it’ll be an interesting new journey for me.
Kirill: That is so cool. Congratulations on the new role.
Yaw: Thanks a lot, Kirill.
Kirill: And the way we met is Yaw was doing some consulting for a company where I was working. It was Sunsuper, it is a pension fund here in Australia. And we really bonded and I learned so much from Yaw. And just to give our listeners a better understanding of what calibre of a data scientist you are, I’m just going to read out where your experience is.
So Yaw was a senior economist for Rating Agency Malaysia for 5 years. Then he was an entrepreneur for a pilot multimedia and bank tech startup for 8 years. Yaw was the Head of Model Validation for 2 years, then he was the Head of Credit Risk Management for 2 years, and then he was a Senior Data Scientist at Pivotal for 2 years. Also, Yaw was a lecturer at Monash University for 2 years teaching about International Banking and Finance.
And now, Yaw, you’re doing your Executive MBA with INSEAD. So that is a huge variety of experiences. Can you just let us know, how have you structured your career? How has it taken you through all of these pathways? And how does it feel to have exposure to all of these different industries and types of work?
Yaw: I don’t know if there was any real structure to the career. I started early. I did a Masters degree in Economics, and I thought, “I’m going to be a an economist.” And I walked into the field, and it was fun. And back then, in 2000, most of the modelling type work was in the economics domain. And that’s when I bumped into data mining, data science. And the minute I looked at technology, I said, “Oh wow, this is fantastic!” But there were no job roles, there were no companies, nobody was interested in doing this sort of work. And so maybe that’s why I started my own company. Because I was quite passionate in it, I saw a lot of value in it. And I just thought to myself well, nobody’s going to do this. I might as well just start a company and start working on this.
So one of the things I realised in consulting type work is you hand over a customer Powerpoint slides. And most of your models would be sitting in Powerpoint slides. The joke we have at Pivotal is, “Powerpoint slides is where models go to die.” And it never gets used.
Doing my own tech startup then led me to try and actually deploy build somewhere, so that the models would be used. And so that’s what I did. And after a few years, I realised, “Ok, I’ve done this role up to 8 years.” I was looking for something bigger. The company was doing reasonably well. It was surviving. But the main reason why I started a company, which was, there were no companies working on data science, that problem disappeared. People started to work on data science and modelling.
I joined a bank because they started to use these models to make decisions, and they wanted somebody on board in the committee to help them as they were making these decisions. So it was interesting instead of just building the models, instead of just buying the models, there was an opportunity to see how the models were used by the decision makers. So I jumped on it. And that lasted a few years, until big data really exploded regionally. And I wanted a more regional role to see how it was practiced around the region, and also when I saw the big data technology, I realised that this was a bit of a game changer in how they were processing things in a parallel mode, how cloud was coming in, the way we would need to build the models would be different from the small data. There would be some additionals, things that would come into play.
And so what I decided was to join Pivotal. So that’s been the arc of my career. I did the MBA primarily because I realised that at a senior leadership role, you need people who actually understand the data science, but also are able to communicate to senior decision makers. And there’s a lot of change management involved. And that’s one of the things that I realised that I didn’t have as a skill set, and I did the MBA primarily to get that skill set.
It would be good if there were people at senior management who really understood how the whole modelling process actually worked.
Kirill: Totally makes sense, and I think that’s a fantastic step, especially with INSEAD, which, as you mentioned before we started the podcast, that it’s one of the top MBA providers in the world. And it’s located in Europe, is that correct?
Yaw: That’s right. It’s based in France, and their claim to fame now is that the Financial Times ranked them as the number one MBA program and the number one Executive MBA program. So they’ve done really well.
Kirill: And how many people are in your class?
Yaw: So my section itself, it’s about 30 students. But the whole entire program across Abu Dhabi, Fontainebleau, and Singapore is about 200.
Kirill: And do you have to fly to Europe to attend the lessons?
Yaw: For some of the lessons, yes. And it’s a very international program. There is also another joint program with Tsinghua. So at some point in time, we fly everywhere. There’s one group flying to Abu Dhabi this week. Some went to Vietnam, some went to Beijing. So we take classes all around the world, with people from everywhere around the world. So it’s pretty fun.
Kirill: That’s very interesting. And so taking this MBA is definitely going to help you with some skills that you’ve identified gaps or actually just developing networking. From your background, I understand that you came into data science as an economist, right? From the economics side of things. Was it challenging to learn the other skills that are involved in data science, such as software skills? And I remember that you were really helping me out to understand Hadoop, and how that works, and apart from all the other things, such as modelling, which is probably relevant to your economics background, you still have a great deal of expertise in the technical side of things, such as Hadoop and other tools. Was it hard for you to master those tools coming from an economics background?
Yaw: I think most economists and people who traditionally trade in statistics will find it difficult. I gradually moved in, so that helped. So I always had a knack for technology, or at least an interest. And I did my own web programming, built my own websites back then. So I had some interest. But I think the big jump that helped me was, I started my own tech company. You know, when you start your own tech company for 8 years, and you’re hiring software developers, managing software developers, after a while you pick it up pretty quick. So I had that 8 years to transition and acquire all those technology skills.
So that helped me because in the role of a data scientist, computer science skills, technology and programming skills, become extremely important. And so I had a nice gradual shift from economics to data science during those 8 years that I was a tech entrepreneur. I imagine not everybody can have such a luxurious 8 years to transition.
Kirill: Definitely. So that was, as you say, a gradual transition. And then, after being an entrepreneur, you moved on to being the Head of Model Validation for a bank. Was that bank in Malaysia?
Yaw: Yes. So it was a joint role. Both Head of Model Validation and Credit Risk. Primarily, they hired me as a Head of Credit Risk. They then realised, “Yeah, we need somebody else to do model validation, because we can’t ask the guy who’s doing the model development to validate his own models.” And they just looked around and said, “Oh, you know, you’re pretty good at modelling! Why don’t you take this role!” So yeah, I had that role.
So one of the things that I recognised as early as 2005 was that whatever models we built, I was felt that it was like a can of sardines. It has an expiry date. It will expire. We never go back and check these models. And I’ve seen models that have expired, are no longer predictive, the market has changed, the world has changed. And people are still using those models to make decisions. And as a Head of Credit Risk, I’m now using these models to make decisions. So of course, I’m very worried about how well it’s been validated, has been checked. Is it still working?
So yeah, it was natural for me because as a user of the models, I was also very concerned. And because I had that ability, I could have a more intelligent conversation with the modelling development team and give my feedback.
Kirill: Totally makes sense. And I can see how that joint role would come to be. And that’s exactly what we were finding at the company that I was working at, and you were consulting for. That they had models that were deployed, but nobody had been tracking the progress of those models over, I think it was a period of 18 months, or something like that. And what I learned a lot from that experience was that models actually deteriorate over time. Whether you like it or not, the performance of models is going to drop off and I think we even found a model that was — I think it was a churn model — and we even found one that had a negative effect. So it was better to flip a coin than to use the model. Do you remember that?
Yaw: Oh yes. That’s very common. And the specific circumstances, if I’m not mistaken, was due to a regulatory change in the superannuation industry. And this is common. Whenever you see a major regulatory change, a model built before that period will tend to fail. Just because the world completely changed after that event. So it’s very important to really update the models after a major change in the economy, or a regulatory change.
Kirill: Totally. And just for the benefit of our listeners — because we have listeners of all varying experiences — when we’re talking about building a model such as a churn model, what we mean is there’s an algorithm that decides, based on a person’s characteristics, such as gender, how much they spend, how much they have in their bank accounts, their other behavioural habits, their purchasing and other behaviour, based on all those characteristics, the model will decide what is their likelihood that they’re going to, for example, leave the company in the next six months.
So that’s one of the models that we were looking at. And so usually, those models give you quite a good understanding of who the most at-risk customers are so you can prevent them from leaving and do something about keeping them. But when a model deteriorates over time, the use of this model, or the benefit of this model, is minimalised, and actually can become negative, so it might not be any beneficial to use this model any more.
So Yaw, from your experience, what would you say about model deterioration? I know this will probably depend on the industry, but how quickly do models deteriorate over time?
Yaw: It depends partly on what’s happening around the world. Certain kinds of models, say corporate credit scoring, corporate credit rating. Companies don’t change their behaviour that rapidly. They go through the business cycle. So that would last you quite a few years. For things in the retail space, or marketing space, take for example, marketing. Consumer behaviour is very erratic, changes very quickly. And those sorts of models get themselves updated every week, every day. As new data’s streaming in, those models will need to be updated very regularly.
For things like retail, credit cards, they last just a year or 18 months. And the big question that the person should ask is, has anything happened in the world recently that could cause consumers, or companies, or whoever I’m building my model based on, that would cause them a behavioural change? And that’s usually a pretty big clue. That’s just coming from a common sense perspective. There are specific metrics that you can track, specific model validation reports that you can track to monitor. Usually, I would recommend a combination of both the common sense, observing what’s happening in the world, plus also using the metrics.
Kirill: So what would be an example in terms of common sense? What would be an example of an event that could undermine a credit risk model, for example?
Yaw: Let’s say the subprime crisis, where we have people losing jobs due to the economy crashing and the unemployment rate rising. That’s going to be something that’s going to affect your model. It’s going to be a change in the business cycle, or it could be a regulatory change in what the government allows. So those usually are important events or triggers. And it’s usually good practice that you have a chronology document that you write down where certain key events are. Because as a person who’s now looking back at the history of the model and the provenance and how it’s going through different versions, by having that document which actually dates out certain key events, it helps them give a business context to what’s actually happening to the data.
Kirill: Totally make sense. That’s a very interesting example. And we’ve already discussed, or gave our listeners an example of a churn model. Could you in a few brief sentences describe how does a credit risk model work when one is used in a bank?
Yaw: Banks, as part of Basel, they had to shift to what’s—traditionally, maybe fifteen years ago, what they had was a lot of decisions were based on business judgment, human judgment, expert judgment. And they needed to move towards models, because these models would then determine how much capital—the risk of your portfolio, and what the models said the risk was, that would determine how much capital they needed to provide. And that capital is a huge cost to the bank. So where the banks then need to use these models are both a) first, to make a decision: Should I give you a loan, and b) the capital provisioning: How much capital should I set aside for prudential purposes? So it’s both dual needs.
And the regulators are extremely concerned about it from capital provisioning, and business people would be concerned about it from the decisions. So they have similar needs, but it can split. You know, businesses might care a bit more about accuracy, whereas the regulators might care more about the stability of the model. The clash between stability and accuracy—the way I would describe it is it’s like an F1 car. You want the fastest engine as possible to get to the end of the race, and you also want the engine not to blow up in the middle of the race, car to break down and things like that. So you’re trying to constantly balance between the two. The race car driver wants to win, he wants the fastest car and the engineers would say, “Let’s not push it that much. Let’s make sure that the car doesn’t blow up. And please don’t drive so fast until you blow it up.”
Kirill: Okay. Yeah, that totally makes sense and you want the car to get to the end of the race, right?
Yaw: Yes. So the banks use these models for those two reasons: the decision and the provisioning of the capital.
Kirill: So does that mean when I go to a bank and I request a credit card, that it’s not actually just the bank teller or the person I’m talking to, the representative of the bank. They look at me and they maybe look at my finance and they decide what limit to give me whether it’s $7,000, whether it’s $10,000. They actually use a model that somehow analyses my behavioural patterns or some other characteristics and then it spits out the results. Is that correct?
Yaw: Yes, that is correct. So what’s been going on is a change within the last 10 years as traditionally they tend to rely more on demographics. So when you first go into the bank there is what we call the application score. I don’t know much about you, you’re a new customer. All I know is certain demographic factors about you, and most of the assessment of my risk is based on those simple demographics for the application score. What we find is that the application score is useful as a first-cut decision, but it’s not very predictive of a person’s real risk.
What becomes more useful is over time, I actually collect data on your usage, how much balance you have, how much you transact over the card, what do you spend on, what things you look at, do you do money withdrawals. And that becomes a behavioural score and a usage. Today, what we see with the fintech world is that you have players like Alipay coming in, where they want to take advantage of a broader set of data pulled from your social networking. Anything that’s out there about you, they want to use all that data because what we’ve realised is that demographic factors are fairly blunt. What is really very predictive is behavioural type of information. And the more behavioural type of information I have of a customer, the more accurate the model can be. And the model’s more accurate, and I know that you are a no-risk individual, I’m more willing to give you a better price. I’m more willing to give you better terms. Whereas if I knew that you are a high-risk individual, I would be much more cautious. So if the model is used correctly, it could actually lead to much more optimal decisions, which are fairer for society. Of course, then if the model is not updated – that’s the issues of an expired model – it could lead to very perverse scenarios where actually it makes this whole society worse off like what we saw in the subprime crisis.
Kirill: That’s very interesting; and that is very in line also with what we found in the Australian pension fund sector, or superannuation sector. That even though these superannuation funds are huge, so the tenth in size – not even the first, second or third – the tenth superannuation fund has a total of $36 billion under management. That’s a huge amount. And the top funds are over a hundred billion dollars, really pushing the boundaries of a trillion dollars in funds under management. So they’re huge financial institutions and they’re like much bigger than banks. But at the same time banks, in terms of modelling, have an advantage because banks have credit cards that they give to their customers and then customers actually transact based on those cards. So banks don’t have that much funds under management, but at the same time they have a huge amount of this data that they can analyse and therefore assess their customers even better.
Whereas superannuation funds, they’re just pension funds where the money is just lying and it’s accumulating until somebody retires. So there is no transaction information. And even though the pension funds or superannuation funds are much bigger in size than banks, the banks have an advantage of data. And so this kind of like unspoken war between the two—when we were discussing it, we were kind of agreeing that banks, even though they’re smaller, they have this huge advantage and over time they can overtake the superannuation fund because they just know their customers better. Do you still agree with that statement?
Yaw: Yes, definitely. I think this is the shift. This is where superannuation funds are going to be disrupted. Everybody in every industry is going to disrupt it and what you just described is one example of how superannuation could be disrupted where banks realise the wealth or — are able to capitalise and leverage on the data they have, they could disrupt superannuation and take much more share. So in any business, if you are not observing of what your existential threats would be and you’re not having a good strategy to deal with this, you are non-existent. So I fully agree with that statement. And even banks themselves, they are likely to be disrupted by the fintech small companies which are leveraging on social networking data, tech data, because that’s going to tell a lot more information of a person’s personality and risk profile and risk attitude. And they will have more transactions on the social—you know, on the IT space than what you’re having on financial transactions. And that’s why you see the fintech world today where they’re trying to get as much data on things like Apple Pay, Samsung Pay, because that tells you more behavioural information and it becomes a source of competitive advantage. So you can see, everybody, small banks are thinking of disrupting the superannuations. You have the tech companies thinking of disrupting the financial services. And it’s just one big fish eating a smaller fish and then a smaller fish.
Kirill: Yeah, totally. And I think you or somebody mentioned to me an initiative that banks were thinking about undertaking – maybe this was even in Singapore – where some of the largest banks were actually thinking of installing cookies into the browsers of their users – of course, with their consent – but using that to collect data on the browsing habits of their users to better build the profile. Is that still an initiative and do you think that it is something that’s going to go ahead?
Yaw: Yes, that’s probably going ahead. I’ve been involved in a few discussions, not banks per se, but yes, I think telco companies particularly are really interested because instead of profiling a customer based on where you live and what gender, I could profile a customer based on what topics you’re interested in. If I know you’ve been searching on cars, you’re probably interested in getting a car. Or if I know you’ve just been shopping around for baby food, Mothercare and things like that, you’re probably having a child, you’ve got a life-changing event. Those topics tell me more about you and it gives me that rich information from a marketing perspective of what to market, how to market. So instead of just going based on demographics, you can extract from the URLs what topics people are interested in, know that better, and be able to offer them much better targeted promotions or suggestions of products. I think that’s pretty scary from a “1984” perspective. That leads a lot to ethical questions of how models and artificial intelligence needs to be used. That’s a large question Silicon Valley needs to ask itself. This is where the technology has moved ahead so far and what it potentially can do, and then the ethical considerations now need to catch up.
Kirill: Okay. That’s exactly what my next question was. What is your position on this ethical dilemma? Should people be protected from their data being exposed and therefore not being used in marketing and other campaigns? Or should it be completely available?
Yaw: I think the truth of the matter is that there’s so much data being collected, it’s difficult for a consumer to completely say, “I want to opt out.” So it’s an open secret. No matter what you try to do to protect yourself — I’m going to work from a position that no matter what you do or try, it is an open secret that data will be out. Now, if I come from that perspective, then the organisations that have this data, that use this data, how should they ethically behave given the fact that the data is out there and what are the considerations? And I think that it’s very much similar to, say, corporate governance from the sense of a reputational effect. Organisations that misuse and abuse this data, they should be really punished from a reputational effect on this abuse and misuse.
Just the same way we may punish organisations that are environmentally negligent and things like that, they do bad things there. And same thing, we should punish organisations that are very negligent in the way they handle consumers’ data. I think that’s one good way of bringing the market forces. And the mechanism already exists to punish organisations this way. The challenge then is what if organisations themselves are not publicly listed. Should we bring in fines? Clearly there needs to be some. The government does need to think about this and bring in some role into it, and have some role just to protect consumers if the organisations abuse. It’s not just the protection on the personal privacy, but when it gets loose, what should government do to actually promote good behaviour.
Kirill: Yeah, I totally agree with that. And I like that you mentioned that it’s pretty much impossible to protect yourselves from your data going out there. I think there was actually an experiment conducted on how can you stop this data exhaust, so we’re constantly living our lives, we’re using mobile phones, we’re using transportation, we’re walking around the city where there’s CCTV and so on. And we’re constantly leaving this data exhaust behind us and I think it was even proved that even if you go and live in the forest for a week, you will still every single day leave data exhaust just simply because there’s still satellites flying above your head, still radio signals, still some surveillance methods. So there’s no way to get rid of this data exhaust, and it’s just a fact now that it’s going to be out there and the ethical consideration should be on the legislation side, or on the corporation side of how to correctly use your data. I’m super glad we had that discussion. And I would like to move on a little bit back to your career. So you were Head of Credit Risk Management and also Head of Model Validation, a joint role. At that time, how big was your team?
Yaw: It was fairly large. About 20 to 25 people, just because I had several different things. I mean, it was a large team within the bank and had a very large set of responsibilities. Not all of them were modelling. Some of it was just day-to-day reporting, some of it was in putting in the infrastructure for analytics, for the department. It needed a couple of wide array of issues beyond just modelling, also the day-to-day reporting.
Kirill: Business as usual.
Yaw: Yeah.
Kirill: And then you moved on to a role at Pivotal, where you were the senior data scientist and a completely different role, a completely different position. You went from managing a team of 25 people to actually performing the work yourself. In a lot of our listeners’ view, it will sound like a move backwards. Ideally, people dream about moving through their roles and then becoming a manager and having a team under them whereas you chose to do the opposite. Could you explain why you made this decision?
Yaw: That’s right. I actually chose the exact opposite. I had a huge important role in the bank. Credit risk is a very key position and I decided to go back down to actually do coding. Everybody was like, ‘Why would you start…’ So what I did was, I saw that Big Data was going to be a game changer. Machine learning, all the new technology that was coming out was going to become a big game changer. And I realised that I was going to be a dinosaur if I didn’t jump and learn how to do it. I also personally felt that if I was managing a team, if I did not know the technology, if I did not have a good understanding, I would get less respect from technical people. Technical staff usually like to work for the boss who actually understands the details themselves. They may not need to execute, but they really need to have a firm appreciation for the issues. So I thought it would be good to update on my skills with the new technology so that I would not be a dinosaur.
One of the things I made sure was every year, even though I was just in a managerial role, I made sure every year I took an opportunity to build one model every 18 months. That kept my skills fresh such that I could come back down and actually take a technical role and update all my technology skills. That was going to be very important for the next 10 years because as the technology shifts, I would have at least an updated set of technology. So it was an important decision from my side to actually just bite the bullet and say, “You know what? I need to go back and do real work and relearn everything again. And I think some people may sometimes feel that all they want to do is just progress up and manage more people. But I realise that that can be a big mistake, because you will soon lose touch of – especially in the technology world – you will soon lose touch of how the technology functions. And if you haven’t gone in and actually built things and do it for yourself, you start becoming an ineffective manager.
Kirill: Yeah, totally. And it’s very easy to lose track of what’s going on in the world. Did Pivotal give you this opportunity to develop your coding skills? And just generally speaking for the benefit of our listeners, what exactly does Pivotal do?
Yaw: Pivotal was a great company for me in the sense that it’s a very hard-core technology company. It’s been in Silicon Valley for many years. It was the most pivotal—okay, let me rephrase this. Pivotal was spun out of EMC. And actually what EMC largely saw was the new world coming in, the Big Data cloud. Their new set of technologies which they then wanted to spin out as a separate company and not keep it within the old traditional business. It had a mandate. So the three business areas: Lapse, which was agile software development in cloud; and Greenplum, which was in a Big Data with data science, a big team of data scientists. They put them together because these were the new technologies that were going to transform the world. And it had a mandate saying, “Take all these practices that were operating in Silicon Valley, which a lot of the established enterprises were not aware of, and bring these Silicon Valley type practices over to the established enterprises, the big firms, and help them disrupt their businesses internally, help them to make that transition to these new economics. So that was its market positioning, bringing what was done in Silicon Valley out into the Fortune 500 type companies.
That was great for someone like me, because I had a lot of that big company background, I had small business background and I transitioned myself. Leveraging on this new data, Pivotal was exactly the position I wanted to be in. What I would say is that Pivotal really is at the forefront with regards to—it introduced a lot of new ideas and new practices of how agile software development needed to happen, how data science needed to happen. But what I also saw was when you walk into large organisations, there is a massive amount of resistance. There is a cultural shock, and it stops becoming a technology question; it starts becoming more of a cultural change management question. Most Silicon Valley companies do not have to deal with this. They can be very insular and they just have to deal with the technology issue. Pivotal wanting to bring and help Fortune 500 companies disrupt their traditional businesses has to face this change management issue to a large extent. One thing I realised is Silicon Valley doesn’t have very strong change management skills. That’s where the traditional consultants still have a very good role to play, because they’ve got a lot of experience doing this.
Kirill: And hence your MBA now that will help you develop that expertise in cultural change.
Yaw: Yeah. It’s not by design. It’s just by accident that it turns out to be a relevant skill. After coming into Pivotal I saw, “Oh, this is what’s needed.”
Kirill: Yeah, totally. Moving on to the work you did at Pivotal, because that’s where we met and that’s where personally I learned a lot from you. Seriously, I learned so much from you about R coding, about PostgreSQL, about Hadoop, about modelling. And I really want to give our listeners a little glimpse of that opportunity to also learn a little bit from you. Can you tell us a little bit about the tools that you used or maybe still use to build models?
Yaw: Traditionally it was always relational databases, and then there was this big move towards parallel processing, massive parallel processing and Hadoop, where that order of magnitude, that advantage is, instead of having just one big supercomputer, Google realised, “Well, this is not going to be possible because the Internet is just too big. I can’t fit everything into one machine. I can’t scale up and buy a bigger box. But I can scale out and add more machines and have a cluster of small computers.” Now, the challenge with that is when you have multiple computers, and not just one big supercomputer, is that you have to take your problem and you have to break that problem, you have to split the calculations across multiple machines and process that in parallel. That requires a different way of programming, a different way of thinking about it. But it also means that you can get your results much faster. And it also means that you can actually build models differently.
So what Pivotal does is we leverage on things like Hadoop, which is a Big Data platform. And what I mean by that is it actually splits the calculations across all the many small computers. It’s essentially still disk technology-based, hard disk. And the problem with hard disk is that it’s slower because there’s a lot of mechanisms involved. And so there’s a read/write, and the input/output is slower. Today, very recently, with Hadoop 2.0, there’s a leverage on Spark, which is a memory leveraging on the RAM which has lower latency. So what I spoke to you last when we met, that technology and the tools that we used have also involved. And that’s just two years ago, right? So after you learn one thing, then you realise, “Oh, damn, I’ve got to go and learn.” But that gives the advantage on the memory layers to be much faster. So when you have real-time calculations, something like an e-commerce website, or steady streaming type video, and you need to do quick calculations, that RAM gives you an order of magnitude much faster, like ten times faster in terms of speed.
Now, that’s a Big Data problem. But most of us were trained in small data just because anybody who did data science 5-6 years ago, you would have started to use things like R. Now, R itself is a great tool. It’s pretty much a direct competitor to SAS. It’s an open-source tool. And R has an advantage because academia, whenever they invent the latest algorithms, they just release out a lot of that as R package. And so you have very powerful machine learning algorithms, the latest available, out there in R. And for me that’s a big advantage over SAS because if you really want to be at the forefront, R has that technological advantage, the latest coolest things.
The problem with R is that it doesn’t parallelise. What that means is if you had a very big dataset and that can’t fit on RAM, you’re screwed because it will just take you forever to do those calculations on R. It takes a while. However, where Pivotal comes in is where we actually sort of cross that bridge. We allow you to use that R code and then we would actually parallelise it on our platform and we split those calculations for you. We call that pivotal R or another component we have is called MADlib, and that actually breaks those. So you have the advantage of using your R code, but at the same time you can actually do those calculations in parallel in a Big Data world. So that actually helps the data scientists because you don’t wait 3 hours, 5 hours, for your results to come because that can be very frustrating. It just happens in a matter of seconds and that’s a huge advantage. What’s happening now today with Spark is also that people are now moving more to Python. And the joke that runs around is that if you’re my age, in 40s, most people would be using SQL and then those in their 30s, they would be doing it in R and those in the early 20s, they would be using Python. So what language a person uses pretty much tells what age they would be.
Kirill: That’s funny.
Yaw: Yeah, it gets pretty hilarious because you can see that as they go through the interviews and you’re going through recruits. It actually shows up in that pattern as well.
Kirill: Yeah, well, I’m 27 and I’m kind of like in-between R and Python but I kind of like R more, so I think that is a testament to this segmentation that you just presented. Yeah, it’s very, very interesting. So you would say that Python is the new language of Big Data? Is that statement correct?
Yaw: That’s correct. And the advantage Python has is, first of all, it’s closer to a proper programming language with proper syntax. R is primarily functional programming so that’s good in its own way. But one of the challenges is that you actually have to deploy a model, you want it to be deployed, and Python gives better support when you want to move your model from development, which is great for the data scientist, but you now want it for production grade and actually get used. Remember how I talked about model validation, where the models will expire? In a digital marketing space, it can expire very quickly, over a period of a few weeks. So being able to move from development to production quickly is a huge advantage because you just get correct decisions. And that’s where a proper programming language like Python has an advantage over R.
Kirill: All right. And it’s good that we moved slowly back to models because I was also going to ask you a bit more about these parameters. So you mentioned that there’s two sides to model validation: the common sense side, so has anything happened in the marketplace that might have disrupted the model, so just switch on your common sense and think that maybe things have changed, and it’s time to reassess your model. But also you mentioned parameters. So what are these parameters? Without going into too much technical detail, what are the parameters that people who are interested in modelling, they should look out for or they should research and understand better that would help them assess the performance and deterioration of their models?
Yaw: The standard metrics would be accuracy ratios, ROC, what they call Receiver Operating Characteristics. These are very technical terms but at the end of the day one of the things that you want is metrics that tell you how accurate your model is. If your model is meant to predict whether the customer defaulted, then the question was, “Were you right?” and “What was the accuracy rate?” That’s one aspect. The other thing you want to look at is the significance, the t-stats, the p-values, where it’s basically telling you “Did you have enough data? Was the result you found just by coincidence, or is this reasonably robust?” Back to the issue that we talked about earlier on between a balance between accuracy and robustness. So those are two standard ones.
There’s a whole bunch of metrics that go along. The third group really is from an economics point of view, which is “Was this decision financially profitable?” So it’s useless if we had the most accurate model, the most robust model and it doesn’t make a big change to the organisation because it’s too small of an impact, like if it only changes 0.001% of the customer population. Then whether you got it right or whether you got it wrong, it doesn’t make much of a difference. You want to spend your time on the areas which have huge economic impacts. So, for example, you’re thinking of rolling out a model. You think it’s going to have a huge impact to the business and grow the business. And it turns out it only had a small effect. Then sometimes it’s just worth killing the model because it’s not worth maintaining it. So those are the three things that you want to have in consideration. Different business problems may have specific metrics that go along with it, but broadly speaking these are the three considerations.
Kirill: Yeah, that’s a very valid point. So was your model accurate, did you have enough data, and did it actually have a business impact. I totally agree with that. And it reminded me that you mentioned when you were working at that bank in the Credit Risk Management/Model Validation role, that at some point you had somewhere like over 20 models that you were managing simultaneously. Is that correct?
Yaw: Yes. 20 was on the low end, actually. The target was to raise it to 40 over the next few years. And some banks have much, much more.
Kirill: Is that reasonable? Does it actually add value to have so many models?
Yaw: It definitely provides a lot of jobs. So partly it’s driven by regulation. And I think, yes, it’s going to be a situation where—so the variations between the models aren’t that big. You can think of reusing it. So what typically happens – and I’ve had this conversation with many organisations – is that they run these little POCs (Proof of Concept) prototypes and they see one or two models that work well. But these processes that you introduced to handle these two or three models, a small modelling team, what happens is it doesn’t work and it falls apart when you starting having 20, 40, 50 models. In the future that’s where every organisation is going to head to, just a large volume of models. And they will need to think through for the future how they’re going to manage this and this is where having good proper systems and proper planning will help later on. Because what works for just two or three models will not work for when you get to 30, 40, 50. And I think that was the problem that the organisation faced because they skipped the steps of setting down the infrastructure. They just wanted quick results. And then once you got past a critical break-even point, it was just chaos trying to manage everything manually.
Kirill: I can totally imagine that. And maybe in the future we’ll have some sort of computer algorithms or AI managing these models and that will simplify things. Yeah, that was a very interesting part of this episode and now I’d like to move on to some more questions about your experience and your thoughts about data science. First one will be what has been the biggest challenge that you ever faced as a data scientist?
Yaw: When I was young I used to think that scrubbing and cleaning and getting the data was 80% of the problem. And 20% was just actual modelling. And that’s what everyone always states out there. I’ve redefined it differently now that I’m older. So 80% of the problem is still scrubbing and cleaning the data, 20% is the modelling. And then the other 80% is actually getting buy-in of the model. Convince people that your model is correct. So, just when you think you’re done, you realise, “Oh, my God. This is even harder.” And most data scientists are not well-trained in the buy-in process. We’re numbers people, you know, we show it, it’s logical, everyone should agree. And no, the world doesn’t work that way.
Kirill: Love it. Love it. I’m going to call this “The 80-20-80 Rule of Yaw”.
Yaw: Yeah. So that’s where the other remaining second thing—yeah, I think the harder challenge now is the buy-in process because I think—I once had a quote from a CEO saying, “Are you trying to tell me that for the last 30 years we were practicing and looking at the wrong factors?” And the thing was that the models suggested that these were additional variables which made the difference, which actually were predictive, and no one was looking at it. And the CEO could not accept the fact that they have been looking at the wrong factors all these years. And it is a cultural shock, right? It is a mental shock to the individual. It will take a while before they—he kept wanting to see proof after proof after proof. And even though every single proof showed that we were right, mentally it was a very difficult adjustment for the individual.
Kirill: What is your recommendation on how to go about these situations?
Yaw: There is no absolute rule to this. Having strong stakeholder support internally is one. Having the data is also necessary. But in the end, one of the things I now realise is that it’s a very emotional decision for the individuals. And being able to allay those emotional concerns is something that’s important. Because they have to have this emotional shift. It’s not the mental shift. They can get the data, they see the data, they can make that mental shift. It’s that emotional shift. And as data scientists, we are not really trained to work in that sort of skills and it’s good to have somebody in that team who knows how to help the customer make that emotional, psychological shift; and the data scientist just needs to appreciate that’s where that promise is not showing more results, showing more data. It’s helping that individual make that emotional shift.
Kirill: Yeah, very deep, very profound thought. And a lot of data scientists, and aspiring analysts, they don’t even know about this area, or they don’t focus on it enough and I can attest to that. That that is one of the hardest things. There’s so many people who can crunch numbers, so many people who can drive insights. But it’s actually the people that can convey those insights to an audience and get them to, as you say, buy into this truth that you’re portraying with data, because data tells the truth. Those are the people that are most valued and become the most successful data scientists. Moving on to our next question—so we talked about challenge. What is a recent win now that you can share with us? Something that you’re most proud of in your data science journey?
Yaw: I think one of the most recent wins was when I actually helped a very large organisation, one of the top global insurers. Actually, they had a challenge where the CIO wanted to bring data science, machine learning into the organisation. And to do that, he needed to have the different stakeholders buy into that process. And they brought us in to make that transition. And this was a very long project. It was challenging from all areas. One of the biggest challenges was what I just described, that emotional shift. It was a multi-country engagement, or several countries. And it was multiple departments. So at the end of the day, what the CIO wanted was a buy-in across the organisation. I was happy at the end of the day that they actually made that shift. They actually had several countries and several departments actually raise their hands saying, “Yes, we are interested in starting their own internal projects.” So I was successful in actually implementing what Pivotal wanted, which was to bring and disrupt an existing business in a large, established organisation. And it was very, very challenging. I didn’t have those necessary skills at the start. Towards the end of the project, I managed to acquire more of those necessary skills for the change management; but yeah, maybe one of the reasons why I’m so aware of it is just because of that journey I had to go through.
Kirill: Okay, that sounds like a huge project, and congratulations on that. That must have been quite a difficult process to get the buy-in from a multi-country organisation. Now that we’ve learnt about your experience, and you have by far a huge amount of experience in different areas of data science, both the entrepreneurial side and economics background, managing a team, and actually then working in Pivotal performing the coding, what is your favourite thing about being a data scientist?
Yaw: That’s easy. When you actually find something interesting, when you hit that insight, it’s just like that aha moment. It’s like an inventor or a scientist having made a discovery, or an inventor getting his invention to work. The satisfaction of doing a good job and having something that’s been built and it’s something that you created, that’s usually the best part of that moment for me. Creating something, and you know that you’ve done a good job, you’re proud of that creation. That’s usually the best part.
Kirill: Yeah, and that’s a lot of fun. I can see how that can be in any of your experiences, whether it’s being a senior data scientist at Pivotal or being the head of a credit risk department at a bank, or doing the entrepreneurial side of things. In any aspects of data science that you’ve been involved in, that aha moment is always present. So whatever you’re doing, data science allows you to create things and allows you to drive insights and have that aha moment and I can totally agree with that. It gives you that great feeling—probably for the rest of the day, you feel so enlightened and so proud of yourself that you’ve found something that was so hidden inside the data. From where you stand now, from everything you’ve seen in data science, where do you think this field is going? What should our listeners look into to prepare for the future of data science?
Yaw: I wish I knew the exact answer. So clearly, as immediate, I could say that more models are being deployed and this field is really—you have one side which is really moving towards deployment of models, which is much more computer science, programming-driven. That’s one huge direction I see, because nobody just wants to have a model anymore. And you have some data scientists who just get very interested in the implementation and the programming aspects. The other side of it is also working on the communication. Because more and more senior stakeholders want to come in, more and more people from, say, marketing want to get involved. And they really don’t understand this whole language that data scientists are talking about. So just being able to understand and cross-communicate with the business people, dealing with the emotional issues I have just described, that’s also another big challenge. And I see data scientists having to get pulled deeper into these two worlds. And unfortunately there’s no time to learn all three, both the data science, the computer science and the business/human aspects, so usually you find data scientists gravitating towards one side versus the other.
Kirill: And which side would you say you gravitate towards now?
Yaw: This is a tough one. So Pivotal clearly on the computer science/programming deployment approach. And then having done the MBA is clearly on the other side. So I switch hats every few hours. But at the end of the day, I think my core skill would really be on the business side of things. I’m not so much on the — well, I am not too bad on the programming, but yeah, I gravitated perhaps, having made a huge investment in the MBA, perhaps more to the business aspects.
Kirill: Interesting. And you stand as a great example of somebody who even despite all your success in data science to date, you continuously find new ways to develop your skills to be an even better data scientist. Going forward, what is your career from where you’re standing now? What does it look like now? So you’re joining DBS Bank, and what do you see for yourself in the space of data science?
Yaw: I think the new organisation which I’ll be joining, DBS, they’ve realised that banking is probably going to be disrupted by a lot of the fintech companies, a lot of the Silicon Valley companies. So this is fun for me because joining an organisation that’s progressive, they understand the threat that’s coming, and they’re trying to figure out how to deal with this by leveraging more on machine learning. I think that’s where the skills will come in of how do you marry a business threat from a strategy perspective with a technological advantage, and how do you bring that to bear? I think those are probably skills which I want to leverage on and add more expertise. So bringing not just data science to the implementation, but really bring in and invest in a business strategy. That will be probably where I should progress on in the next role.
Kirill: Yeah, it sounds like a very exciting role, and it sounds like, more than anybody, you’re prepared for it, to tackle that challenge of finding the ways that a bank can be disrupted and maybe—like they say, if you want to survive as an organisation, take the smartest people and get them to disrupt your own organisation and then find how to fix those problems, right? It sounds like that’s what you’ll be doing.
Yaw: Hopefully.
Kirill: Looking forward to it.
Yaw: Right. It was great to be on this show with you, Kirill, and I wish you best of luck with this endeavour. I think it’s great that you’re doing this podcast. It’s definitely value for society.
Kirill: Thank you so much. And just for our listeners, I’m sure there are going to be so many that are just captivated by your career and where you’re going. How can they follow you, or contact you, or just follow your career to see where it takes you?
Yaw: Right. So I’m fairly active on LinkedIn. I check it several times a day. I like that platform. Yeah, if they could just contact me via LinkedIn, I think that’s usually the best way. And I’m usually quite amiable to work with and get to know other data scientists. I tend to ignore the recruiters but if it was a person who’s a data scientist, and up-and-coming and interested, I’m usually always happy to share and connect with them.
Kirill: Wonderful. We’ll definitely include the URL to your LinkedIn in the show notes. And one final question: What is your one favourite book that you can recommend to our listeners that can help them become better data scientists?
Yaw: One of the lesser known books which I enjoy, and I wanted to just raise this out, was this book called “The Logic of Failure” by Dietrich Dorner. Where it was interesting was he was describing how screw-ups happen, and mistakes happen, and the decisions made in making all of those big mistakes, a lot of it was just the misinterpretation of the data, the misinterpretation of how things actually happen, and I think, maybe I just want to call up one point, which was feedback loops. A lot of planning people think of it in a very linear world. And they think “Okay, step one, step two, step three.” But the real world is much more complex. There are lots of feedback loops. Once you do something it creates a cascading effect. The world is not linear. And I think that’s the crux of what the traditional models have. We didn’t have the computing power, so we kept all the equations simple, linear, Bell curve. But where data science is saying it’s different is because the world is much more complex. And his point of feedback loops is still not much known, but I think it’s very important because once you do something, it will feedback and so—once you deploy a model to take advantage of a pattern, you actually change the pattern. The pattern will change because you’ve taken a business action. And that’s also the point of the model validation I raised earlier, where the patterns will change. And that was discussed in the book. And I think that’s interesting because you have to realise that if you are taking advantage of a pattern, you will change that pattern.
Kirill: Very interesting. That’s something you don’t think about often and it really made me pause for a second, that applying a model will change the pattern that the model exploits and therefore the model will need to be rethought, hence the feedback loop. Thank you very much, that’s a great suggestion. “The Logic of Failure” by Dietrich Dorner. We’ll definitely include that in the show notes. And I just wanted to say again, thank you so much, Yaw, for taking some time out of your super-busy schedule to share this knowledge and wisdom. I’m sure lots and lots of our listeners are going to find so many gems in the insights you shared today. Thank you so much.
Yaw: All right. Thank you very much, Kirill.
Kirill: So there you have it. I hope you really enjoyed this episode, because Yaw shared so much of his knowledge and expertise in data science. And I’m so happy that Yaw was a guest on this podcast, because this allowed me to show you a glimpse of all of that mentorship and knowledge that I was so fortunate to receive when I was working with Yaw. And, of course, I will aim to invite Yaw again onto this podcast sometime in the future so he can share even more knowledge because just one hour is totally not enough for him to explain everything that he can share with us about data science. But this should also give you an idea of who you should look for as a mentor for yourself in data science, what calibre of people, what type of people that will guide you through this very complex and very broad field which is data science. They will help you get through it. They will guide you. They will put you in the right direction and hopefully this episode gave you a glimpse of that, gave you a feel for who you should be looking for as a mentor in data science.
You can get the show notes for this episode at www.www.superdatascience.com/10, so just a number 10. It’s our anniversary episode. And while you’re there, make sure to subscribe on iTunes or Stitcher. Also, you will find all of the links to the books and Yaw’s LinkedIn. So go ahead and follow Yaw on LinkedIn so you can see how his career progresses in the future. And I can’t wait to see you next time. Until then, happy analysing.
