SDS 219: How Kaplan uses Data for Education - SuperDataScience - Big Data | Analytics Careers | Mentors | Success

SDS 219: How Kaplan uses Data for Education

Kaplan Data for Education

How does the future look for education?

In this episode of the Super Data Science Podcast, we take a deeper look at how data science, AI, and reinforcement learning can use Kaplan data for the education sector. Make sure to tune in to know more!

Subscribe on iTunesStitcher Radio or TuneIn

About David Niemi

David Niemi is the Vice President of Measurement and Evaluation at Kaplan Inc. where he oversees efforts to improve the quality of measurement across all business units, evaluate the effectiveness of curricula and instruction, and study the impact of innovative products and strategies.


An individual spends at least 16 years in the educational system to get his bachelor’s degree. This is not counting the time you spend on your master’s, doctorate, and other certifications, may it be traditional or online. Truly, we’ve established long enough in this society that education is the key to a successful future.

Kaplan, one of the biggest names that offer higher education programs, is set to improve the educational systems and educators’ learning techniques, according to our guest, David Niemi. In 2017 alone, there were around 491,000 students, from different parts of the world, who used the platform. It’s high time that we make use of the big data available. From analyzing students’ learning behaviors to remodeling course outlines (based on a user’s data) to boosting students’ performance, there are too many possibilities that data science can help the education sector.

David’s responsibility at Kaplan is very crucial in making sure the educational programs are catering to each student’s needs and learning capacity. In his book, Learning Analytics in Education, he, together with other experts, comprehensively lays out methods, examples, and trends to further the use of analytics.

As educators, we must ensure that there’s always significant progress in a learner’s performance. Learning is not at all easy, as David points out, so it’s advisable always to review if a student is still keeping up or a course will still be useful for his future. Online courses shall make fitting recommendations and user-friendly interface. The goal is always to make education work for everyone.

There’s a lot of opportunities that can change the future of learning as new developments like AI and reinforcement learning is seen to help us understand each student’s background, activity, motivations, etc. Hindrances in providing the best course can be avoided in the future by using Kaplan data for education. Discover more on how much promise learning analytics provide from David today.

In this episode you will learn:

  • What is Kaplan? (05:13)
  • How do you access university materials, courses, tutorials, etc.? (08:20)
  • Tips and hacks to learn better analytics from the book. (12:55)
  • “Know what’s in demand.” – Kirill (18:09)
  • “One way to get started is just to get started.” – David (19:28)
  • Education will stay as a top industry in the future. (24:15)
  • Identifying the learning outcome. (32:42)
  • Challenges in learning analytics. (38:17)
  • AI and reinforcement learning in data analytics. (43:44)
  • Study modes: part-time vs full-time. (52:26)
  • The future of Learning Analytics and General Education. (54:45)

Items mentioned in this podcast:

Follow David

Episode Transcript


Full Podcast Transcript

Expand to view full transcript

Kirill Eremenko: This is episode number 219 with Vice President of Measurement and Evaluation at Kaplan, David Niemi.

Kirill Eremenko: Welcome to the SuperDataScience Podcast. My name is Kirill Eremenko, Data Science coach and lifestyle entrepreneur. Each week we bring you inspiring people and ideas to help you build your successful career in Data Science. Thanks for being here today. And now, lets make the complex simple.

Kirill Eremenko: Welcome back to the SuperDataScience Podcast ladies and gentlemen. Super excited to have you back on the show. And today, we've got a very special guest joining us, David Niemi, who is the Vice President of Measurement and Evaluation at Kaplan. So, first things first. What you need to know about Kaplan if you haven't heard of this organization before is that, this is a very large company in the space of education. They were founded in 1938. Over the past 80 years, they have grown significantly. They service over 10,000 business to business clients worldwide, and we're talking about large organizations here. In 2017 alone, 491,000 students used Kaplan to prepare for different source of exam. So, that's 491,000 students worldwide, just in 2017.

Kirill Eremenko: So, as you can imagine, this is a very large organization. What David's role at Kaplan is, is to oversee efforts, and improve the quality of measurement across all business units, and evaluate how well the students are able to learn, how well Kaplan is delivering its trainings. And so, as you can imagine, this is going to be a very exciting podcast because on one hand, we all love to learn, we're all endless learners here, so we can pick up some very valuable tips, and I've picked up some valuable tips from our conversation. On the other hand, why I love this podcast is because it's an industry example of applied Data Science.

Kirill Eremenko: So David uses data to perform analytics in the space of education. So, he analyzes the learning journey, collects data points, and applies Data Science techniques in order to extract insights and understand from a data stand point, how the learning journey can be improved. So, very exciting podcast. Whether you want to pick up some education, or tips, or whether you want to see how Data Science is applied in a specific industry, which is learning, whether it's online, or offline, this is a podcast for you.

Kirill Eremenko: And by the way, the education industry is booming, and it's only going to keep growing, and that's another thing that we talked about in the podcast. So without further ado, I bring to you, David Niemi, Vice President of Measurement and Evaluation at Kaplan.

Kirill Eremenko: Welcome back to the SuperDataScience Podcast ladies and gentlemen. Today I've got a very special guest on the show, David Niemi, calling in from L.A. California. David, how are you doing today?

David Niemi: Oh, very well thanks.

Kirill Eremenko: We were just-

David Niemi: It's been a nice day. Usual great Los Angeles weather.

Kirill Eremenko: 70 Degrees you mentioned, right? 31 Celsius.

David Niemi: Yep.

Kirill Eremenko: And we were just talking about the snow. When was the last time you went up to the mountains to see the snow?

David Niemi: Oh, it's been a while. Probably 10 years. I mean, we can see it, during the winter months. Usually a little later around January, you can see the snow on the nearby mountains. But yeah, I haven't been up there for a while. This reminds me. Maybe I should go up this year.

Kirill Eremenko: Yeah. It's always nice to change it up a little bit. Do you notice how I've heard the snow caps are reducing in size because of global warming? Is that something you notice just from looking at the mountains?

David Niemi: Well, there's less. Yeah. I think that's true. There's just less snow in general, and that's partly a function of rain fall in California. When it rains in the area I'm in, down in the low lands, it's snowing up in the mountains. And that's just happening less, and less frequently now.

Kirill Eremenko: That's a shame, isn't it? We don't want to lose that completely.

David Niemi: No, absolutely not.

Kirill Eremenko: Anyway. Moving onto more exciting topics. David, you are the Vice President of Measurement and Evaluation at Kaplan, and I'm super pumped to have you on the show because I have personally ... as I've mentioned to you before podcast, I have personally used Kaplan many times through companies where I used to work professionally. Many companies in the world actually use Kaplan for their internal training. And that's as you've correctly pointed out, that's not the only case where Kaplan is used. And basically, I'm super excited because we not only ever use your system, you guys know so much about education, and so many of our listeners are passionate about education. So, I'm really looking forward to discussing these topics. And, to get us started off, for those, all our listeners who are not really familiar with Kaplan, could you give us a quick over view? What does the company do?

David Niemi: Well, it's basically an education company. It started out as a test preparation company, by Stanley Kaplan decades ago, actually ... helping students prepare for college admissions tests, and we still have that going on in the unit called Kaplan Test Prep, and they prepare students for all sorts of different life ensured tests, and different things that people need, certification tests, and so on. But besides that, we have around the world financial training language schools for students who are preparing ... let's say to go to universities in Australia, or the UK, or the United States, and want to learn English. And, lots of other different kinds of programs. Helping students get into universities. That's kind of around the world program.

David Niemi: So, there's a lot going on. And, one of my roles is to help make sure in all of our different educational programs we're actually using everything we know about learning research, and also everything we know about how to measure learning. How to tell whether students are actually benefiting from all of our programs.

Kirill Eremenko: Very interesting. And so, has Kaplan been measuring learning for a long time now?

David Niemi: Well yeah. It's been sort of the core part. One of the things I'm working on is how we can make our measures technically more valid. Because it's actually not easy to measuring learning. If you think of learning as acquiring, they're developing skills that you didn't have before, if you're going to measuring that, you're going to measuring whether your program has taught students anything, you have to measuring effectively what they know at the beginning, and also at the end. And you have to do that with equivalent kinds of metics. So, it's not actually an easy thing to do. So, we're constantly working on improving how we do that. But yeah, it's a critical thing, and critical thing we have to do in all of our programs.

Kirill Eremenko: Gotcha. And just to paint a bit of a picture of Kaplan, because I've mostly ... or the only way I've encountered experience Kaplan training is online. Is it a fully online training platform? Or do you have in person training as well?

David Niemi: Both. The online university that I mentioned to you earlier, Kaplan University was a fully online university. It was actually purchased not to long ago be Purdue University, which actually wanted to get into building up its online programs, and serving kind of a wider population than they typically do. It's a very selective high end university, and they wanted to be available to people all over the world, who might not be able to go to Purdue. So, that was a totally out of line thing. But there are lots of programs that happen in office. A lot of the test prep stuff that goes on, I mean some students actually want to go and work with an in person teacher when they're studying for admission tests, and so on, and a lot of the training programs, a lot of the financial, and language training programs happen in person. So, it's kind of a mix around the world of online, and in person.

Kirill Eremenko: Okay, gotcha. And so, it's not just that I can access the Kaplan training materials through a company like for instance you have a partnership with a big corporate organization, and employees access Kaplan, but I can do it as an individual? How would somebody find your university or materials, and access your courses whether they're online or offline?

David Niemi: Well, you can go to the Kaplan site. and pretty much, eventually find your way to everything. There's so many different programs around the world. If you're in Australia let's say, you can find Kaplan Australia companies there, that's primarily financial, and some language training things going on in Australia. But, you can find your way there by going to the Kaplan main site, too.

Kirill Eremenko: Okay, gotcha. I totally understand that. Okay and so, another topic, you recently authored a book. Learning Analytics in Education. Experts explain how to use data to understand and increase learners success. Very exciting! Congratulations on the book first of all, that's exciting, I can imagine.

David Niemi: Thank you.

Kirill Eremenko: How did you feel about the writing process?

David Niemi: Well, in the end it came out really well. I'm happy with the book, and what we ended up saying in it, which I think is really important. It actually started as a Gate Foundation, the Bill and Melinda Gates Foundation project, several years ago. The idea was to kind of lay the groundwork for an integrated feel of learning analytics that would bring together experts from many different perspectives. Learning scientists, educational measurement people, psychologists, cognitive researchers, as well. And also people on the more data analytics side. Data scientists, AI people, and so on. With the educators who were going to be people who actually would have to use the results of learning analytics, and do something about it, and without getting them into the picture, we're liable to do what often happens is, the technical people will come up with all sorts of data that educators can't figure out what to do with. So, that was kind of the initial idea.

David Niemi: There were a bunch of different people who ended up writing down some ideas, their thoughts about their aspects, their understandings, their parts of the field. I eventually decided it would be really interesting to get that published as a book. So, that's what we did. It took a while, it was a very kind of lugubrious process working with people who were really pretty famous, in their individual areas, and so that means very busy. Kind of herding them all, keeping them on track, and getting it done was a big challenge. But I'm really happy with how it came out.

Kirill Eremenko: Interesting. I'm fascinated by this field. I never knew actually ... I suspected it existed, but never actually met anybody who worked in this space. So, would you say that learning, this book is more for educators, or teachers, and people who want to create the online or offline curriculums and courses and better help their students? Or is it for students as well?

David Niemi: It could be for educators who are interested in learning something about learning analytics, and what the different aspects of it are. I think its primary audience is probably more people who are actually going to do learning analytics, so the Data Science people, the learning science-

Kirill Eremenko: You deal with data analyzing data about learners?

David Niemi: Yes. And that's what course, the learning analytics kind of makes that clear. So it's basically analytics that focus on learning and how to improve it. And there's other kinds of analytics going on in education like how do we retain students, or how do we get more students to apply? And so on. Those are more kinds of the business sides of it, and it is important to keep students in your program, and you want to help them persist and stay in the program because they're not going to learn if they don't. But, that's not enough. I mean, we also want to make sure they're actually getting some real value out of the time they're spending in our programs. So, that's the learning analytics part, too. How do we tell what they're learning, and who's struggling, and what we can do to help the people who may be having trouble.

Kirill Eremenko: Okay. Good. I understand. So that book sounds definitely like something I would pick up, and I'll order one for myself because, I want to understand learning analytics better. However, for listeners of our podcast who aren't in the field of learning analytics, let's focus on getting some insights from the book that will help them learn better. I'm sure there's some tips and hacks that you've identified that will help somebody who's learning something. Just kind of like have more accountability about their learning, because there's so much information right now in the world, that it's so easy to pick up a course, to pick up some skill that you want to learn, and then not do it efficiently, or in general, just let it die off and never learn. Do you think we can do that and you can share some insights from your book?

David Niemi: Yes. I actually wrote a chapter on using analytics to improve academic persistence. Which is basically how to hang in there, even when you're struggling. And, this is kind of a key thing, too you know? Because learning isn't always easy, particularly when you're getting into a new field, and so when you start to run into challenges, that's problematic for lots of students. And just to speak of people in schools, and we'll kind of get back to your question.

Kirill Eremenko: Yeah.

David Niemi: You know, some of us probably were pretty successful as we're moving though lower levels of our schooling, and figuring out how to cope with what might not have been great instructions.

Kirill Eremenko: Yeah. Or irrelevant.

David Niemi: Yeah I mean, we could use the text book. We could use our math text book, and figure the math out almost for ourselves, even if our teachers were not that great.

Kirill Eremenko: Yeah.

David Niemi: But, if you're in a situation where you're trying to learn new things for yourself, in your career, there's ... to me, there's a couple of key things. And this is sort of applied to education in general, too. One thing is we waste a lot of time in schools teaching students things that we don't really need, and we'd be much more effective teaching the things they do need like better math skills, or how to write a coherent paper that actually presents evidence on something and persuades somebody of something. We're really neglecting a lot of students when we don't teach them those kinds of basic things, or focus of other stuff that they're never really going to use. So, making sure you have an understanding of what people working in your field really do, and what the skills are, and I would say get that by talking to those people.

David Niemi: What is it you know how to do? What kinds of things do you think about? What skills do you use in your every day life, and how do I get those kind of skills? Because now, you want to make sure if you're going to spend time kind of teaching yourself, or taking classes or whatever, it's going to be things that are actually going to benefit you somewhere, in a real life situation. And you know these people are interested in something, in working and learning analytics, working in education, which is really about trying to teach other people something, and help them learn, there are some really good programs now in different universities, Sanford I think has recently started one kind of connected to the book about how to analyze educational data, and kind of gives you some basic background in learning, and so on. So, we can talk more about that, too. Just got to get into the field of learning analytics if you want to apply your Data Science skills to actually improving schools, and so on.

David Niemi: But anyway, making sure that you know what it is that people who are really good in the field you're interested in, actually do, and how they got those skills. I think would be a key thing for people. For anyone who wants to progress in the field. And, there's actually a program now thorough Metis, which is a unit at Kaplan, too. Kind of teaching Data Science skills to people, but they're working all the time in trying to figure out where are the jobs right now? What skills do people need for those jobs? So, this may seem obvious, I don't know whether it is or not to your audience, but lots of people just stumble on that. Trying to get themselves ready for fields for which there aren't going to be any real jobs, or jobs that they're actually going to be interested in. And then, if you do, if you are trying to study various things. Thing I mentioned before is a big challenge of what to do when you start to struggle.

David Niemi: One of the big things, we know from research is, that's typically ... it could be a result of a couple things, but most of the time, it's probably because you don't have some knowledge background that you need to understand the content you're dealing with. So, you have to hang in there and figure out, what is it that I was missing, if I'm having trouble with some tactical stuff here, that I don't understand, it's most likely not because I'm a stupid person, but because I just don't have the background. I need to master that. And if you look at people who eventually do become successful at any field, it's much more a result of the time and effort they put in than just being naturally smart. But that's a big thing that a lot of people get discouraged by right away when they run into something that seems hard. They conclude it's just beyond them, just not going to be able to figure this out. If you really want to do it, you've really got to analyze, well what is it here that's making ... maybe I missed some other prerequisite course I should've taken, and I go back and do that first. That kind of thing.

Kirill Eremenko: Mm-hmm (affirmative). Wow. That is extremely insightal. Thank you. I'll just recap on those for points mentioned. A great tip is before you get started in learning, go and find out the key things that people in the field actually do, and point number two is, how did they get those skills in the first ...? They're like, you can replicate their journey. And also, being conscious of what's in demand. If this is a podcast for data scientists, so assuming that everybody has already decided that they want to be in the field of Data Science, but still understand, machine learning going to be in demand. That specific type of machine learning that you're learning, or in that specific industry, is it going to be in demand, or is something else going to be in demand. Because, you made a great point that, why study something so hard, and be so persistent about something that won't exist in the near future. We already know there are professions that 10 years from now, probably won't exist. Same goes for sub domains within a profession, like Data Science.

Kirill Eremenko: Look out for trends. See what is actually happening. What will be in demand? Maybe it's not in demand now, but things like robotics process automation. Or forensic analytics. Are those things going to be in demand? geodemographic segmentation, things like that. And finally then what is your comments on why people struggle? Is that the lack of knowledge? Or, do you just need to realize that you need to sit down and persevere? It's sometimes scary, isn't it? When you're starting out your learning journey. If you look ahead and you're like, "Whoa. There's so much I have to learn. There's just no way." Just like, maybe you understand you can do it, but you also feel that kind of fright, or a bit apprehensive about how much there is to do. How do you recommend for people to deal with that when they see the volume of learning that they have to go through?

David Niemi: Well, I'll actually recommend something from my person experience. But there's research on this, too. One way to get started is basically, just to get started. And don't feel like there's so much to do here, why would I even try? It's sort of hopeless. Well of corse you'll never get anywhere if you have that kind of view. But, just getting started, I mean, for people who have anxieties about things, or afraid to try things, the best thing you could actually do is take some action. Get started on some aspect of that huge amount of things that you have to do. And I had this experience when I was working on my dissertation, which was actually in learning science, and having exactly that feeling.

David Niemi: I had taken on such a big project that involved so many students, and teaching about 50 teachers, how to teach students about understand rational numbers, and then administer a bunch assessments, and all that. So that took a long time itself, setting up the study, then analyzing it, and writing about it. And there were a lot of times I just felt I don't see how this is ever going to get done. But if I need to slow myself ... if I just spend 10 minutes a day on this, it might take me 10 years, but it will get done. And so, I mean, that's basically what we would recommend, the research recommends to teachers of any stripe trying to help students who are having trouble getting started.

David Niemi: And you can do some analysis, how long you think it's going to take, and so on, and then figuring out well, I could be at some point in eight months if I spend only 20 minutes a day might get me there. So, don't think about the whole task. Just think about getting started on something. And it could be you'll end up going in a different direction, but the steps you take will lead you in that other direction. Just like ... I often tell people who can't decide what their career is going to be or what ever, "Well, get going on something. It's better to get going on something and try to do well on that, because that will lead you. Even if it's not the thing you want to do, it will lead you to the next better thing in a more effective way than kind of sitting around and thinking about well, I'm not really sure if I want to go work at Starbucks, or what ever." Well, do it. If you don't have any other idea, do that first. And then that will take you somewhere else. That kind of thing.

Kirill Eremenko: Wonderful. Love it. And I also really, I side by your recommendation to invest a certain amount of time, like 10 minutes per day. Because I initially thought of saying, "Yeah. A good thing is break it down into baby steps, and take one step at a time." One way of doing that is saying, "Alright so, I want to study the field of machine learning." For example. And there's so many different things. Well, baby steps. Today I'll do simple introgression, tomorrow, logistic regression, the day after, and so on. But even in that approach, because you don't know how long it'll actually take you, you might procrastinate, and when you get to the day when you have to study logistic regression, you might be like, "I don't know if this is going to take me 10 minutes, or if it's going to take me 10 hours." So, by allocating yourself 10 minutes per day, no matter what you do, you have to do those 10 minutes, right? That way you guarantee to your own brain, to your own self, that that's all you're committing to. Even if it's a complex topic, you're not going to spend more than 10 minutes. And that way, it allows your brain to be more easy going with it. Right?

David Niemi: Well yeah. It's more doable. For most people. And it's exactly as you said, if it turns out something you tackle is too much to master in 10 minutes, who cares? You'll take it up the next day. It might take you three weeks, or what ever, but you will get there as opposed to sort of trying to say, "This is going to take me 10 hours. I can't spend that much time today, so I think I won't get started on anything."

Kirill Eremenko: And even if you're doing those 10 minutes, and you get internet a state of flow, you get super excited about it, and you don't want to stop, as long as you can afford not to stop, you can go for 20 minutes, or you can go for an hour. But, you don't have to. That's the different.

David Niemi: Right. And you're putting your whole emphasis into the effort. Putting in time and effort to master something, and not worrying about, jee is this hard for me? And maybe it's easier for other people, and they're just smarter, so maybe I should give up on this. That's what kills progress from people.

Kirill Eremenko: For sure. Okay, and speaking of the field of education analytics. So, using Data Science and data analytics in the space of education, or learning analytics, why I think so is because as we move into the future, one of the industries that is indeed going to remain alive most likely is education, because we see a lot of jobs being transformed. A lot of jobs being replaced. Automated by robots, I was recently reading an interesting study that according to the World Economic Forum, the number of jobs that were done by robots in the world compared to humans is 21% were jobs done by robots in 2017, and by 2020 is going to be more than ... I think it's going to be 49% of jobs that are going to be done by robots. Somewhere along those lines. So basically a radical shift. And then as we go into the future, more and more jobs are going to be done by robots.

Kirill Eremenko: And so, people eventually need to reeducate themselves. People are going to need to learn new things, people are going to need to find new passions, new careers, and so on. So field of education is definitely here to stay. And therefore, learning analytics is a really powerful tool to have, or to know because there's going to be lots of jobs based on learning analytics. So, what would you say, let's dive into that for a little bit. What are some of the data points that you collect? What are some of the powerful types of data that you collect in learning analytics to help make those conclusions derive insights or predictions that you're making in your job?

David Niemi: Well, right now education is kind of in an interesting position where at least in most countries now, early education is primarily in person, although there's more and more online stuff going on in kindergarten through 12th grade schools, just talking about the US for example. So, the data picture there is kind of different. The schools do have quite a bit of data about students. In fact, they have a lot more than they used to, and they have lots of test scores, and all that, but it's in kind of archaic database set up, student information systems, and so on. So, it's not right now being used to generate information on a day to day basis that teachers and students could use to improve their own learning. But still. So the challenge there is, how do we figure out the things the teachers could actually do in the classroom to kind of assess how their students are doing? And also have the information they'll need on how to act on those data. And then occasionally we'll provide them test scores, or what ever, and tell them what to do about those, and so on.

David Niemi: But that picture, that old traditional picture is really changing now that more and more students are studying online. Particularly in universities now, we've mentioned a couple examples before of universities kind of going all in on online programs, and you probably heard about the big MOOCs, and universities that are putting a lot of their course content online for free and all that. That is really opening up the analytics opportunities. In a number of different ways. And one of the big ways is, when you have a whole bunch of people, when your enrollments are kind of open, and anybody could go into a program, you're going to get people who have all different sorts of backgrounds, which is going to mean, some of them are going to do really well with what ever the content is ... let's say it's an AI course, or something you want to teach ... because that's a really good example, I don't know if you heard about a researcher a while ago who started an online AI corse. Something like 50,000 people signed up for it. The number who actually finished it was minuscule. I'm thinking it was 10 people. [inaudible 00:28:24] than that.

David Niemi: But, everybody was really interested in it, but it's people who are coming with no prior knowledge, no background, or what ever. So, if you're going to make online education work, you need to have a way to figure out what kinds of people are starting the program. What are their backgrounds? And what do they need to know that not know right now in order to be successful in this program? And how do we give them that additional information? By the way, they may have a bunch of other problems, too. They may have financial issues, so they're struggling to feed themselves, and their families day to day, also it's other things going on. So, it's going to be critically important to really make education work for everyone online. To know who the students are, and as much as possible about them. I know there's lots of ways we can talk about it, how you might collect that.

David Niemi: So, if you've had students be in an online programs for 12 years, in their early education, and now they're going into college, you could potentially know a huge amount about them that could make it possible to really help them much more effectively as they move for their college education, and even beyond. Into the work place. Now this raises some kind of privacy issues, too. You might have so much information about students that someone could use that information in harmful ways, but that's another issue, too. How do we back the data? How do we protect the privacy to the students we're dealing with in the case where we're trying to use everything we know about them to help them learn more effectively and efficiently, which does imply that we've done a lot of analytics, so we know which kinds of programs, which kinds of teaching, and what kinds of support being teaching work best for which types of students. So, that's the optimal future, using analytics at its best to kind of help everyone.

David Niemi: And some negative possibilities there, too because we don't want to get into things like ... in schools now, who are likely to fail, and then telling teachers, "This 40% of your students are likely to fail." That tends to have a really negative effect on the students and the faculty. What we want to do is, "Heres what kinds of help each of your students needs to know." And we'll put them in four different groups. "This group needs this, this group needs that, and some other group needs something else." As opposed to "Here's the students that are probably going to drop out in six months." It's much more about, "Heres what you should be doing for these students." So, that's how I see the use of analytics, and kind of is one of the fundamental themes of the book, too. How do we make it move in a direction where the information being provided can actually be used to effectively help students?

Kirill Eremenko: Gotcha, and I totally agree with that point. It's also about how you communicate those insights. I read a recent study where their researchers went to a school and went to a few classes, and picked randomly, students absolutely randomly, and said to the teacher that these are ... we did a test, and these students performed higher so they're your top students. But, there was no actual test. That was all done at random. And indeed, after a while the statistical significant number of those students that they picked at random, were doing much better because teacher was now focusing on them, giving them more attention, more knowledge. So indeed, you have to be careful about how you communicate. Yeah but, very interesting field. So you ... we can collect data on how people learn from very early on, and that's true like you hear in Australia, one time I was talking to someone and I found out that their kids in school, they actually no longer write. They just use iPads for everything.

David Niemi: Yeah.

Kirill Eremenko: I mean, I think that's a very progressive approach. Kids don't even know how to write in that school, so. Kids just know how to type. And if you think about it, that's the only thing you'll need in life, the writing anyway. But yeah, so, there basically everything is digitalized. Also data ... Okay. That's more kind of like the early education. What about for profession? For people who we don't have that data because in their generations, and years ago, everything wasn't digitized. What kind of data points are you focusing on there?

David Niemi: Well then it becomes ... again you know, if you're going to do to professional training for people who are already working at a career, let's say, and there are several Kaplan programs that do this, too for people who are financial planners, let's say in Australia, has become a big thing, I think in many different countries.

Kirill Eremenko: Yeah.

David Niemi: Helping figure out what to do with their money. Well, there are different levels of certification you can go through in that field. So, there are people kind of working constantly to improve their knowledge so they can get higher and higher levels of certification, and so on. So, I mean there are a couple of critical things there. One is ... and unfortunately I have to say this is not true in many of these fields right now ... you have to define very clearly what it is, what your learning outcomes are, for the people that you're trying to train, and then you have to measure where they are with respect to those. How far are they away from achieving a specific outcome? What skills are they missing? And the most efficient way is to do some assessments upfront, some testing upfront, figure that out for all students, and then just give them the skills that they actually need to move forward and advance to the higher level of what ever it is they're trying to achieve.

David Niemi: Unfortunately none of that is working all that well in many fields, partly because people, even certifying boards have not been great in many different fields, about determining what it is that financial planners actually need to know, and be able to do to be effective, and best serve their clients. So, that's a big park of the work, too. And that's true for all levels of learning. Just getting clear what it is we're trying to teach people in a way that sort of says, "At the end of this teaching, heres what we want them to be able to do. Now let's test them and see how much of that they can do already, and just train them up on the parts they can't do when they start."

Kirill Eremenko: I'm totally loving this podcast session because we're attacking it from two different angles. Anybody listening to this on one hand, we're talking about an industry specific application of Data Science, which is in the space of learning. And on the other hand, you can apply all these things to your own learning. Anybody listening to this pod cast by definition is already interested in learning, otherwise you wouldn't be listening to this podcast. And this tip, heres another tip, right? Identify learning outcomes. On one hand, if you are a data scientist in the industry of ... definitely super important to incorporate that into your analysis, into one of the data points ... collecting. But, on the other hand, if you are just learning something online, you now no longer need to apply that insight identifying the learning outcome. You no longer need to apply it to a cohort of people or to a group of students, or to an education facility, or a course, you need to apply that to yourself.

Kirill Eremenko: You're taking a course online, what are your learning outcomes? Or, not just taking a course, you want to learn something. Sit down and identify what exactly are the outcomes you want to do. So, do all the same analytics that you would be doing as a data scientist in the industry, but rather than doing it, for many people, for a whole segment of customers, do it for yourself, because you have all the insights anyway, right? We all know how to learn. We all know what we're good at, whether we prefer this type of media, that type of media, what times of the day we learn best at, what type of instructors we prefer. I think that can be even more specific if you're doing it, applying all these insights to yourself. What do you think David?

David Niemi: I think that's really a great insight. And it reminds me of what we were saying earlier about how really everyone's going to need to be a life long learner. People have talked about that, but it's really becoming more and more true. You see when you might go in the old days, might've retired when you were 55 or 60, but people are working until they're 90 now. And they're having to learn new careers, adjusting their knowledge. So, I mean to me that's sort of an exciting prospect. You keep on learning. But, knowing how to do it more effectively, will be a huge advantage for anyone who's able to do that. And one of the fun things about being involved in learning analytics, is if you're a data analyst, you're connected with learning people, and you are finding out kind of from your own data, your own analysis you're doing what things work best for people studying what types of things. So you may say, I'm having a little difficulty figuring out how I'm going to learn this, but I was just doing some analysis sort of shows, if you do these four things, you'll be more likely to succeed than if you don't. So, you can kind of learn from your own work, too in this field.

Kirill Eremenko: For sure. And-

David Niemi: [inaudible 00:37:38] as a learner.

Kirill Eremenko: Yeah definite. And speaking ... it's kind of like I always think I love professions like that, for instance a psychologist ... Or, let's say a professional like we said financial planner, or psychologist, people who can use their profession in their own life, to their advantage. If you're a psychologist, then when you're talking to people at a bar, or at a restaurant, you can use your profession there and then to identify patterns if somebody is lying to you, somebody's ... truth, where is if you are a stock broker, not so much. It really is that in day to day. Okay speaking of things that can help your learning, what are some of the problems in student learning that from your experience, are best ... you found solutions to, or are easy to solve with learning analytics. Do you have any examples?

David Niemi: Well, I don't know if there are any problems in education that are actually easy to solve. But there are definitely problems that leaning analytics can help with. And we've been talking about some of those. The chapter that I mentioned that I wrote on-

Kirill Eremenko: Persisting.

David Niemi: Persisting at tasks, yeah. One of the things we've actually learned from analytics is it's more effective ... People had the idea a while ago that since a lot of students drop out of college like the drop out rate is around 50% overall if you look at all different institutions in the US, about half the students don't make it through to a college degree.

Kirill Eremenko: Wow.

David Niemi: Depending where they start. The percentage is much higher for very selective universities like Harvard, and places like that because they're selecting students who kind of already know how to succeed, and they have the skills they need to deal with it, and so on. But most places are not doing that. So-

Kirill Eremenko: Harvard has a less lower drop out rate?

David Niemi: Yeah. They're, I think ... I don't know ... 88% or something.

Kirill Eremenko: Succeed?

David Niemi: Succession rate, yeah. Make it through and succeed. And actually you would expect it almost to be higher than that. So, I'm not positive about that number, but it's very high.

Kirill Eremenko: Yep.

David Niemi: And, some open programs that we were referring to before, online open universities, it's maybe 20%. In many community colleges, which are two year colleges that basically take anyone whose graduated from high school ... in fact you don't even have to had graduated from high school, you can start taking ... so it's kind of anybody who can go there. Their numbers might be 20, 30% students make it ... get out of the community college, and actually end up graduating. So, some people have looked at those numbers and said, "Well what we need to do is basically give people pep talks and tell them about the value of a college education, and how much more money you'll make, and all that. So maybe in our orientation programs, we'll try to motivate people to stick with it in their programs."

David Niemi: Sometimes there have been kind of small effects from doing that, but mostly those types of programs cheering people on haven't been that effective. What's more effective is figuring out how to help people day to day, deal with the tasks they have to do that day, or the assignment they have to complete that week. In the face of maybe there's alt of crazy things going on in our lives, or what ever. So this is where the analytics can come in. If we can detect when people are starting to tail off a little bit in their performance, and maybe even have some bots that are ... chat bots that are talking to them and saying, "Jee. Looks like you're not doing much today. What's going on?" And getting some feedback that says, "Oh my god. We've just had this horrible event in our family, someone died." Or what ever, And it's really effecting me." And so on.

David Niemi: At this point we probably wouldn't want to try to automate the response to that, but at least someone who could then connect with the student, and support them, could help them. But anyway, the point is, the analytics should be used to kind of monitor how people are doing on a day to day basis, and provide support when it's needed to help them keep going. That is proven ... at least from all the research we've seen so far ... much more effective than some kind of up front canned messages about what the value is, if you're succeeding in the end, and okay, now get started. You're on your own for the rest of it.

Kirill Eremenko: Interesting. Very interesting. And I would say that's where the advanced forms of analytics that we have now available to us come in. Once you ... it's such an individual thing, right? You almost look at, well tracking not rather than a group of people, tracking an individual person.

David Niemi: Yeah.

Kirill Eremenko: I don't know. Measuring their heart beat in a hospital would be analogy of this. Everybody has their own heart rate, and you want to see when it's starting to drop, okay. That's a bad sign, or when their blood pressure is changing. But here, you want to ... we'd use things like advance machine learning, or even to the extent of deep learning, artificial intelligence reinforcement learning, to build a virtual profile of how the person learns. Kind of like a virtual avatar of their learning journey to ensure that they're sticking to it. For instance, somebody might be learning at a pace of ... I don't know ... 10 units of content per day, somebody might be learning at a pace of 5 units of content per day, And for ... or per week, for either of those people, that's their normal rate. You can't compare a person to a person. But on the other hand, you can, you should, and as you mentioned, you should compare the person that's tracking. Are they getting better? Or are they getting worse? Are they stable in their amount of content consumed, and how their learning's progressing? And yeah. That's very exciting that advanced forms of data analytics come in. What kind of data analytics have you seen in the field? Has it gotten to the state of AI and reinforcement learning yet?

David Niemi: Yes. I mean those are sort of on the research edge of things, but I have to say first of all, the way you just described things is kind of a fantastic description of the field of learning analytics and where it's heading, and what some of the opportunities are. These are topics that are discussed in several different chapters in the book. But, as you said, it's really ... if we can make it work to figure out individually what's happening for each student and then, how to respond, is a key part of it, too.

David Niemi: And that you could get from your big data sets where you actually are looking at lots and lots of different people who maybe have similar profiles, similar educational background, similar motivational stuff, and you can be asking questions along the way, too like, "How confident do you feel about what you're doing right now?" And then responding to people say, "God, I don't know whether I can handle this or not." People get over confident too, so that's another interesting sort of thing. People think, oh man this is easy. I'm not going to spend any time at all with this, so they sort of under perform because they're too confident about their skills.

David Niemi: So, you have all these different things going on for students, and we have the ability now to ... sometimes it would be surveys. Actually we're asking people questions about themselves, but other times it will be we can just watch, we can directly observe what they're doing. Some people are experimenting with using the cameras in your computer, and facial recognition things tell what your emotions are at a given point. Now that's ... some people have concerns about the privacy issue there, but if the goal is to do things like help people, and say, "Wow. Are you really feeling okay about this? Or does it seem like something you're struggling with just kind of by looking at your facial expressions, and so on."

David Niemi: So, the opportunity is tremendous to do all sorts of things for students that we might do if we individually, as very skilled teachers were helping them. But, there's no way teachers in classrooms with ... You're a high school teacher, you have six classes, you got all together 300 students, there's no way you could help all of them in a way that we could be helping them online with the right kinds of analytics, some AI support for ... Even doing things like evaluating complex student work. Students write essays, or give speeches. Right now, we're pretty bad in education at having students write a lot of stuff, and giving them really good constructive feedback because it just takes too long for human people to do that. But, a lot of people testing out approaches to giving, not just a mark, or a grade, or a score, but really constructive feedback about what's missing in the argument you're trying to make, and why it's not as convincing as it could be.

David Niemi: So, lots of interesting things going on that we haven't really figured out how to scale up yet, but that is kind of starting to happen. Bigger and bigger tests figure out how to. We got 10,000 people studying around the world, studying the same course at the same time, and every 10 weeks you get another 10,000 people. Tremendous opportunities to learn more about what would help those people in that course.

Kirill Eremenko: Mm-hmm (affirmative). Fantastic. I love your idea you mentioned about using facial recognition. For me personally, if that's going to help me learn, hell yeah I'm all for it. I don't mind recording my face while I learn some topic and then getting an analysis report or the AI getting better at understanding when I'm going through ... for example, what would be very valuable to me is if an AI could alert me at points in time when I'm feeling tired. You don't really notice, it's not like you're feeling full of energy in one moment, and the next moment you feel super tired. It kind of happens gradually, and usually for me, I pick that up a bit late. I've picked that up already when I've spent an unnecessary 30 or 40 minutes of working, or learning while I'm tired. I would rather the AI, through facial recognition, tell me when I'm three minutes into that and be like, "Kirill, time to get up and go for a walk, or go have a nap, or something like that." And that way, I would get more out of my day for sure. That's a no brainer.

David Niemi: You get more out of your day, and a huge amount of research suggests more out of your learning efforts, too. Because it's really ineffective to kind of over do it on the same topic for too long. You're much better spacing your learning practice. We probably all had this, too where we were struggling with something. Even a dumb thing like a crossword puzzle or something. You're just wracking your brain [inaudible 00:48:48]. Walk away for a while and come back, and it comes easier for you. But actually your brain takes things better in small chunks spaced out than in the old I'm going to just cram in all this information in 24 hours or something. And you might be able to do that, but mostly you're going to forget that in the end. It's not going to last. So yeah, you would be ... I mean, that's a great example, too of how it might be used. We'd be better off having the program, and maybe even more likely to take a break if you get that feedback from an AI system that says, "Wow really looks like it's time." If it's doing it by looking at you, that's probably even better. But it might do it just by kind of time. Looking at how much time you-

Kirill Eremenko: Like cars have, right? Some cars say if you drive for three hours, then take a break. Go get a coffee.

David Niemi: Yeah. It could do other things like probably read ... I mean, it is hard to kind of learn some kind of things. And there is sort of productive effort that you have to put in. And so if you read the difference between someone who's really bring intensely productive versus somebody who is just undergoing a kind of cognitive overload, and not making progress, they're just wracking their brains and saying, " Okay. Take a break on this." Or, "Look at this from a different angle." Or something like that.

Kirill Eremenko: Yeah. That sounds very cool. And with these advanced techniques, you don't even need to measure the persons' heart rate, and you don't need to connect any sensors, you can do all that through a video camera, rendering just yeah. Basically you're right. The only concern here is the privacy issue. The whole concern people are going to be thinking like how is my data going to be used? Is it ever going to be leaked, and so on. That's kind of the main ethical question [crosstalk 00:50:40].

David Niemi: Yeah. And that's a big thing for education of younger people who ... now everybody kind of assumes there's no privacy online, and Facebook or anywhere. But, in the US anyway, there are federal regulations protecting the privacy of younger students. So, you do have to be careful about that, but I'm thinking for watching someone interact with a math program online is probably not that risky for students. I mean, I suppose if they started crying, and talking about horrible things their parents have done, or something, that could be a privacy issue, but. So you'd have to figure out how to kind of deal with those things. But that is the big issue. Not to be neglected, and there will be big policy decisions to be made about that in the future.

Kirill Eremenko: Yeah. And so, what are you experiencing right now in those ethical considerations from the data that you have? And people becoming more open to it? Or is the opposite happening with what you're hearing about Facebook, that hack that happened, and so on. Are people closing up more?

David Niemi: Right now, we're not close enough to having enough data about students that it's much of an issue. So, we are doing ... researchers are testing things under controlled conditions where they're going through institutional review board ... which are basically boards to protect student rights, and so on. It's going to become more of an issue when some of this stuff really begins to look promising, and get used on a larger scale. So, sofar, that hasn't started to happen. But, it will. We can anticipate probably much faster than we imagine.

Kirill Eremenko: Got cha. And I wanted to run this by you. I like what you mentioned that, it's better to learn in chunks, rather than get an overload. I recently read that it's actually also a good idea to learn multiple things in parallel. For instance, we're all kind of used to reading one book. We finish it, then you get a new book, and so on. Well actually, there's studies showing that it's better to read five books at the same time because you'll activate different areas of the brain. One might be creative, one might be about history, one might be about mathematics, or what ever. And then, you also use those books to analyze the same topics from different perspectives. And if you think about it, that's how we learn in school. We don't learn history for a month, then we learn biology for a month, and then mathematics for a month. We all learned these things in parallel. We kind of move away from that as we grow up. Even as professionals, we tend to take one course on a specific topic until we finish it, then we take the next one.

Kirill Eremenko: What are your thoughts on that? Is that a good insight? Or is it a misleading insight? Is it a good idea to learn multiple things in parallel? Or is it better to do in sequential?

David Niemi: Well, the research so far hasn't been conducted for all topics being learned. Any five different topics. But it's really interesting and suggestive, and I think something that learners should be thinking about, and something that when we get more data on more people, studying more things, we'll be able to say something with more confidence about that. But, so it has been tried in some areas with some populations of students, and is really where I think every learner thinking about for themselves. So, rather than spending 12 straight hours just studying some aspect of coding, so they take a break and try to learn something else along with that. My guess is for most people, that probably is going to end up being a better approach. But, there's not enough research right now to say exactly what the rule is. If they're learning five things, do them each for one hour. There hasn't been enough studies. But some really interesting ... the one you picked up on, and some other really interesting ones suggest that yes. We should be doing more work to test this effect.

Kirill Eremenko: Interesting. Okay, alright. So, we talked a bit about artificial intelligence in the field, and how that's going. What are your other thoughts on the space of education? I think we both agree that education is going to stay, and people are going to need to find ways to educate themselves more, or reeducate themselves, and we are ... artificial intelligence is slowly entering the field. Any other ideas you can share with us? Where do you see this field of learning analytics going in the future? And in just in general education?

David Niemi: Well, this is not going to sound to forward looking an idea, but. One of the things that's really holding us back in learning analytics right now is the lack of good measures of learning. It's partly because it's not easy to do, and partly because people just don't want to invest in doing it. But, for example, through kindergarten through 12th grade, we do not have ways that allow us to measure how much students have learned about anything over that span of time. There's lots of little tests and so on. And we could conceivably start to put all those data together.

David Niemi: But, this would take some really smart work by Data Science people working with learning people to build measurement scales that show us on some specific topic in ... well, let's say understanding and application of rational numbers in mathematics. How much progress do students make when they start out in kindergarten learning about dividing up pieces of pie, or something, among different students, and what fraction does each student ... to using rational numbers, and really complex equations, or calculus, or so on.

David Niemi: We just don't have it. We haven't built those measurement scales yet. So, that's a really important piece of work that needs to be done. And the same thing would happen in ... We were talking earlier about professional training early on ... we need to be able to figure out how are we measuring learning from where ever a person is starting, to the end point of expertise in that field. And are there several stages that people go through? Is there a bunch of tiny little steps? So, this is really a big thing holding us back, because if you want to test analytics approaches, or instructional approaches, or what ever, and your intent is to improve learning, if you can't measure learning, you don't know how effective any of your analytics or anything else really is.

David Niemi: And this in itself, I think, is something in some of the newer approaches, including AI and so on could help us to do. But, this will require people who really understand psychological measurement to work with people who've got new and more interesting approaches to analyzing tons of data we have about what students have done. It may not all be good measures of learning, but it's all there, and we want to figure out, what is this actually telling us about how much progress students are making? So, this is a big area for me. I think better uses of things like AI to do what we don't have enough people to do right now, to connect with students, to provide support like we were talking about ... you've given a couple good examples earlier but things like students who are just struggling with something and may not know what to do about it.

David Niemi: You can ask them a series of questions, there's actually lots of good psychological research about what might be the causes of people failing at a specific task, not being able to get started. It might be they don't have good strategies for learning, they don't have the right knowledge. You could have ... you can be asking them questions about that, and actually making recommendations on that. And, over time, with enough data, with enough people, you could be improving the quality of the questions and the recommendations you're making. That's a tremendous opportunity I think to me because we're just not doing anything effective in that area in education right now.

Kirill Eremenko: Got cha. So interesting. It's like a closed loop model where you make recommendations and then ... like Amazon. It makes ... or Netflix ... it makes recommendations on what to watch, you watch it, and if you do watch it, and if you do go through a certain amount of that video or movie, that means it was a success, and that means it learns from that, and it makes better recommendations further on, and then those recommendations are now also fuelling it.

Kirill Eremenko: It kind of learns from its own recommendations, a closed loop model assistant.

David Niemi: Exactly right.

Kirill Eremenko: But as you've correctly pointed out to set that up, you need to first ... this field in general, needs to first understand how to measure learning. You can't manage what you can't measure, right?

David Niemi: That's right.

Kirill Eremenko: That I see from this conversation, I can see that's one of the big challenges, and it reminds me of what in Data Science we have structured data or ... oh no, not structured, we have ... what was it called? We have data where it's like words, and text analytics, and so on, and then we on the other hand, we have just numeric data that is already ... yeah structured, and unstructured data. So we have structured data that is already in tables and so on, it's easy to analyze, and is unstructured data which you have to come up with ways to analyze it. Similar thing here, you don't have computer games. You don't have a progress bar above person saying how well they've learned a skill, right? It's very individual for every person. You have to find out what questions to ask them, how to measure that from qualitive data, quantitative data. It's very challenging but I think it's a very exciting field to be in.

David Niemi: Yep. I completely agree. And there are some people doing Data Science work in many university now with the university level data, but they're still not dealing with all these basic issues, measuring learning and so on that we're coping with, but I ... if I had to make a prediction, I think data analytics is just going to sort of explode into whole new realms in education in the next five to 10 years, so. Anybody who wants to get into that, I think it's going to be a really exciting area to be in.

Kirill Eremenko: Fantastic. David, thank you so much for sharing insights. It's been such a good and exciting conversation for me. It's hard to imagine, but we've actually been almost an hour talking, and I'm totally [inaudible 01:01:38] this has been fantastic. Thank you so much for coming and sharing these insights. Before I let you go, could you tell us where our listeners can follow you, get in touch, or specifically where they can purchase your book? What are the best channels to get it?

David Niemi: Well, the book is on Amazon now. So, you can pick it up there. Learning Analytics in Education. And I'm on LinkedIn, That might be one good way for people to connect, and Facebook, and other ways. But I'll ... if you're interested, I can give you an email here, too that you can ... [inaudible 01:02:12].

Kirill Eremenko: Okay, sounds good. Your LinkedIn is great, and we'll include it on the contact details on the showing on the page as well.

David Niemi: Sounds good.

Kirill Eremenko: Awesome.

David Niemi: Well, I was hoping it would be a fun discussion, and boy it has. So, thank you.

Kirill Eremenko: For sure. Thank you very much, David, for coming on this show it was a massive pleasure.

David Niemi: Thank you.

Kirill Eremenko: So there you have it. That was David Niemi Vice President of Measurement and Evaluation at Kaplan. I hope you enjoyed this conversation as much as I did, and picked up some very useful tips for yourself. Whether it is to apply to your own education, or whether it is in or about the space of online education in doing analytics in the space of online education. I personally loved this podcast for that, that it had both components. My favorite take away was to identify learning outcomes. Before you learn something, you need to sit down and understand what are my learning outcomes that I'm after because that way you set yourself a target and it's easier to go towards. And of course there are plenty of other tips. If you're interested further in this space of analytics, and the space of education, in this specific industry, then make sure to pick up David's book which is called Learning Analytics in Education, and it's available on Amazon. You're going to find all the show notes for this episode, including the URL to David’s LinkedIn at There you can also find the transcript for this episode.

Kirill Eremenko: And on that note, we're going to wrap up. But before you go, if you know anybody who is interested in the space of education who can benefit from learning about Data Science in analytics in the space of education or somebody who is a life long learner, and would be interested in the tips that David shared here, then make sure to forward them this podcast and share this information with them. And on that note, I look forward to seeing you back here next time, and until then, happy analyzing.

Kirill Eremenko
Kirill Eremenko

I’m a Data Scientist and Entrepreneur. I also teach Data Science Online and host the SDS podcast where I interview some of the most inspiring Data Scientists from all around the world. I am passionate about bringing Data Science and Analytics to the world!

What are you waiting for?


as seen on: