SDS 640: What I Learned in 2022

(00:19):
Wow, what a year 2022 was, maybe the biggest year for machine learning and AI ever. And for the SuperDataScience Podcast in particular, it was definitely our best year yet. We had incredible guests. I’m proud of the conversations and content covered in every single episode this year, and I appreciate the feedback you’ve provided and questions you’ve had for guests in order to make that happen. The SuperDataScience Podcast team has also grown, allowing us to produce even slicker episodes with more meat on the bones. Specifically, we added three folks to the team this year. The brilliant data scientist Serg Masis came on as our researcher. If you aren’t familiar with his work already, you should definitely check it out. We had him on for a long episode at the start of the year, episode number 539, and then we also had him for a short Five-Minute Friday episode recently, that was episode 634.

(01:13):
So with his role as researcher on our show, he digs into guest backgrounds super thoroughly and comes up with amazing topics to discuss with them. He bridges guest backgrounds in order to come up with questions that they might be the only person on the planet that can answer them. And so I hope you enjoyed hearing those questions and those answers over the course of the year. On our writing team, in addition to Sylvia Ogweng, who has already been writing for us for a long time, we added another writer, Dr. Zara Karschay, and so she’s been a second writer on the show, enabling us to produce exquisitely professional episode summaries, show notes and social media posts for you. And third, the third person that we added to our team this year is the indefatigable, Natalie Ziajski. She joined me full-time as my operations manager to keep all of my plates spinning across the podcast and my other professional commitments. 

(02:02):
For the podcast, for example, this has allowed us to build up a much, much deeper guest and episode pipeline than ever before, and also to increase the richness of our offering and engagement across platforms like YouTube, Twitter, LinkedIn, and even a developing TikTok channel. So with incredible guests and incredible new team members, it’s perhaps unsurprising that the show has continued to enjoy tremendous growth. If you compare the most recent quarter with the same quarter a year earlier, the number of downloads of the show has grown by 90%. So it’s very nearly doubled. Individual guest episodes now conservatively garner at least 35,000 listens. So thank you for listening, for watching, for engaging, and for letting your friends and colleagues know about the SuperDataScience Podcast. We put our heart and soul into all 104 episodes each year, and we do it for you. It means a ton to us that you commit some of your valuable attention each week to the program. Thanks again. It’s truly the great honor of my life so far to serve you on this show. 

(03:05):
All right, so that gives you an update on what we’ve been doing with the show over the past year. And now to cap off 2022 like I did to cap off 2021, for today’s episode I’ll cover the five big lessons that I learned over the course of the year. Namely, I can’t do everything at once. Orders of magnitude more parameters produce unbelievable AI models. The 24-hour news cycle is exhausting and unsatisfying. Working in-person is way more fun, and logging nutrition is effective and paradoxically liberating. So we’ll go through those five one by one, starting with the first one, I can’t do everything at once. So I already went through to kick off this episode, the tremendous success we’ve had with the SuperDataScience Podcast. In addition to that, my machine learning company, Nebula also launched its first product into a private beta in the autumn, in the Northern hemisphere autumn, so just a couple of months ago. 

(04:08):
And that’s a lot, but there’s lots of other things that I hoped to do this year. So things that I started doing at the beginning of the year and that I had been doing in previous years. So this includes one of the things that I regret the most is that I wasn’t able to keep up this year with my weekly updates to my YouTube channel and the corresponding Udemy course. So starting a couple of years ago, I started publishing video tutorials on my YouTube channel, and after getting a rhythm to it, I was able to do it at a weekly cadence for over a year, maybe 18 months, maybe two years, I can’t remember exactly. And all that YouTube content, it’s mostly on math topics specifically geared towards machine learning. So things like linear algebra, calculus, probability theory, and everything that goes on that YouTube channel we also put into a Udemy course. And that Udemy course, it’s got everything that’s on YouTube, but in addition, it has full solution walkthroughs that aren’t available on YouTube. 

(05:15):
So that’s the only difference. You get all of the teaching content on YouTube or Udemy, but if you want these math for machine learning solution walkthroughs, it’s just my Udemy course that has that stuff. Anyway, the Udemy course has been doing incredible. It has over a hundred thousand students, which is crazy to me, but I haven’t been able to update the course since the Northern Hemisphere spring. So yeah, I don’t know, that’s just, it’s disappointing. I get lots of students reaching out, lots of people commenting on YouTube videos, lots of people adding me on LinkedIn and saying, when’s there going to be a new video on YouTube or in the Udemy course? And I keep saying, I hope it’s going to be a couple months, and I really do. I hope that early in 2023 with the podcast now having this great guest pipeline and this recorded episode pipeline, hopefully I can turn a bit of my attention towards creating those videos again, having a great pipeline of videos and having that weekly cadence again, for all of you that have been enjoying those videos.

(06:17):
Another thing that I haven’t been able to make progress on is the corresponding book. So I signed a contract with the publisher Pearson to write my second book, the Mathematical Foundations of Machine Learning. And so this will specifically cover the linear algebra and the calculus content that I cover in my YouTube channel or in the Udemy course, but made into a book format. And I’m so excited to get started on that because writing a book, it really forces you to get super deep into understanding the content in a way that I think even creating video tutorial content I don’t get quite as deep on, and people love a book format. There’s something really magical about a well-written book. 

(07:04):
And so I can’t wait to be able to get around to writing that book. But again, just something I haven’t been able to make much progress on in 2022. So yeah, I can’t do everything at once. The podcast has doubled in listeners. We’ve got this amazing guest pipeline and my machine learning company has its first product in beta. These are huge achievements, but in doing those things, yeah, I just wasn’t able to do everything at once. So I’m trying not to be hard on myself, but with so many earnest students and my publisher out there hoping for progress on the videos, on the YouTube videos, the Udemy course, the book, it’s hard not to feel like somehow I should be able to muster the time or the energy to somehow miraculously get that stuff done on top of everything else. But I really am already, as we’ll get more into later in the episode, I’m already heavily overweight on how much time I spend working, and my life is out of balance related to that. 

(08:06):
So yeah, just accepting that I can’t do everything at once has been the big first lesson for me in 2022, trying not to be hard on myself as I accept that. The second big lesson for 2022 is that orders of magnitude more parameters produce unbelievable AI models. So we’d started to already get a taste of this in the last couple of years. So models like GPT-3 that have orders of magnitude, more model parameters than its predecessor GPT-2. It had lots of emergent properties that the authors of that algorithm, so people like Melanie Subbiah who was on this program in episode number 559, she was one of the first authors on the GPT-3 paper. 

(08:55):
And she expressed to us in that episode that the kinds of capabilities that GPT-3 had, people didn’t anticipate that, the authors didn’t anticipate that. It’s mind blowing the breadth and often human level capability that these generative models have. And 2022 was a super, super crazy year. So leveraging what we call foundational large language models like GPT-3 with orders of magnitude, more parameters, those have been used, those foundational models have been used, no, augmented with other models appended to them, new training data sets. And we witnessed a big bang in the emergence of generative models with staggering expert human level creative capacity on tasks as diverse as artwork. So for example, DALL-E 2 on long form conversational text, for example, ChatGPT. And then perhaps the most staggering development for me in 2022 was the language-based gameplay negotiation capabilities of CICERO by Meta AI research. 

(10:13):
And so the CICERO algorithm is able to play a complex game called diplomacy where you need to form alliances with other players using language, and then you need to basically stab some of them in the back in order to win the game. And the CICERO algorithm is able to play diplomacy online at the 90th percentile or higher. That’s really mind blowing to me, and I’m delighted to announce that I actually just found out, so it might be a little while, at least a few weeks before the episode is live, but I just got email confirmation from one of the engineering managers working on CICERO that he’ll come and do an episode in early 2023 to break down this huge achievement. But in the meantime, if you can’t wait to hear about that huge achievement, you can go back and check out episode number 569 with Dr. Noam Brown. He talks about lots of different gameplay AI algorithms. His particular focus historically was poker, so that’s talked about a lot in episode number 569. 

(11:28):
But in addition, he is one of the key people on CICERO. And so if you go to any of the Meta blog posts about CICERO, Noam Brown is one of the talking heads explaining how that algorithm works. So you can check that out for now. And then, yeah, we’ll have another Meta engineering manager come in, in early 2023 and talk about specifically now that the CICERO’s paper is out, they’ll be able to dig into a lot of the detail there for us on air. So super cool. So I think that this kind of thing is going to continue, these unbelievable AI models, I mean it when I say that 2022 was probably the most extraordinary year for AI ever. There were other important years, like 2012 when we saw the emergence of deep learning period, specific applications like the AlexNet architecture out of Jeff Hinton’s lab at the University of Toronto. 

(12:30):
And that was big, but that was one specific relatively narrow application, and it showed how important deep learning is. But now, 10 years later in 2022, we have these huge, huge, huge foundational large language models that are capable of so many human level capability generative tasks. Yeah, I already went through artwork, conversational text, and language based gameplay negotiation. It’s just wild. So the breadth and the human level quality of these applications using natural language, I did not anticipate anything like this at the beginning of 2022. And so I’m holding my breath for what will happen in 2023 with the release of next generation large language models like GPT-4, which is rumored to be released early in the year, maybe February. 

(13:25):
And so with that, you can expect existing human-level generative capabilities that we saw this year to become markedly more refined and even more realistic and human-like. We’ll see, probably new human level generative capabilities emerge that we didn’t anticipate. We’ve got more coming up on that in the very next episode, episode number 641 when we go into 2023 data science trends with the brilliant Sadie St. Lawrence. So plenty more coming up on how wild it is when we add more parameters, orders magnitude, more parameters into AI models. All right, so that was my second lesson of the year. 

(14:05):
The third lesson for me was that the 24-hour news cycle is exhausting and unsatisfying. So when the COVID pandemic hit, I went from being somebody who only read the news in the physical Economist newspaper that would arrive at my doorstep once a week. So once a week I’d get this physical magazine and I would read through it often from beginning to end. And so I’d get a lot of depth on stories. When the COVID pandemic hit, I was like, what in the world is going on? And I was glued to the news on my phone and looking at rates of COVID transmission and testing and just trying to understand how this was going to change the world and the economy and my businesses and how I needed to adapt. And so yeah, I just became glued to the news during the COVID pandemic. 

(14:57):
And just as the pandemic was coming to an end, Russia invades Ukraine. And that to me, again, I was just glued to the news cycle. I was like, what is going on? I didn’t expect war in the West in my lifetime. I thought that was something that just wouldn’t happen. I think maybe even my parents’ generation, they never saw war at this kind of level in the West. And yeah. So the shock of that, I’ve been since absorbed in the 24-hour news cycle since. So COVID pandemic right into that war, and I’m just left exhausted by constantly, when I have a break or I’m on the subway, my default is to go check the news, what’s the latest? And I think that isn’t good for my mood, just like social media isn’t good for my mood, and it leaves me feeling fatigued. 

(16:03):
So in 2023, I hope to be able to shake my addiction to the 24-hour news cycle and get back to what I was doing before just reading the physical, the paper Economist that comes once a week. And then hopefully that’ll also open up some time so in those moments when I’m sitting on the subway or waiting for some food to finish cooking or something, instead of going to the news, hopefully I’ll go dig into a great book, which is something, since the pandemic hit, I’ve barely done any reading of books at all, and that that’s another huge joy in my life. So 24-hour news cycle, exhausting, unsatisfying, reading books, super satisfying and joyful. So we got to find a way to turn that around in 2023. 

(16:54):
All right, lesson number four is that working in-person is way more fun. So this plays on one of my lessons from last year, which is that remote working works. So I admitted last year that I was surprised that remote working works. And it does. It absolutely does. I’m still blown away at how efficient my machine learning company is largely being remote. The SuperDataScience Podcast also, except for Natalie and me working together most days, everyone else is remote all around the world. So people like Ivana, our podcast manager, or Kirill who founded the SuperDataScience Podcast, I interact with these people more days than I don’t, and I’ve never met them in real life. So remote working can definitely work, but working in person is way more fun. 

(17:45):
So at my machine learning company, for example, this year with so many people being vaccinated and COVID being less of a risk, we’ve been having offsites again, or actually for the first time, because prior to the pandemic, we were an all in-person company. So we’re having offsites for the first time in our now remote-first company. And so that includes things like social activities with other data scientists on my team. So a couple of data scientists on my team and I with a bunch of other people from the company. We went on a big trip to Cabo in Mexico this summer, Northern hemisphere summer, and wow. Yeah, so great to spend time with all those people. Sometimes I just have meals with data scientists on my team, with the ones that live with me here in New York. And then yeah, as I just mentioned, Natalie, who is that full-time hire that I mentioned at the top of the episode, my operations manager we’re in-person most days. 

(18:51):
And that is such a treat. Yeah, just being able to have a bit of banter, bit of laughter, it isn’t the same for me, at least over Zoom. It doesn’t have the same positive impact on my mood. And so, yeah, working in-person is way more fun. That’s my fourth lesson. And so I’m looking forward to more conferences. Yeah, it’s another thing that happened this year is I was able to go in-person to conferences again, like ODSC West, we recorded a whole bunch of episodes at ODSC West in San Francisco this year, episode number 628, 630, 632, 633, and 634. All of those were recorded at ODSC West, and it was so much fun. Also recorded a bunch of episodes in-person at my apartment or on stage at other conferences this year. So that Noam Brown episode that I mentioned earlier, that was episode number 569. We recorded that at ML conf in New York, and we also did an amazing episode with Hillary Mason, episode number 589 at the R Conference in New York. 

(19:56):
So yeah, just being able to get out there and see people in-person at conferences or working with people in-person, it’s really, I don’t know, for me personally, it’s brought a lot of fun and joy back into working life for me. And critically, it’s also meant that I’m more excited than ever to do work and tackle problems in a way that at least, yeah, with Zoom meetings, it isn’t for me the same. And that isn’t to say that I still stand by my lesson from last year, that remote working absolutely works. And there are huge advantages to remote working that I talked about a year ago. But yeah, loving this when I can get it, the level of in-person interactivity has made a huge positive impact on my mood, and I’m grateful for that. And looking for more of it in 2023, more conferences and hopefully seeing more of you listeners out there at conferences as well. It was fun getting to meet a bunch of you at the conferences this year. 

(21:00):
Okay. And then lesson number five, my final lesson of 2022 is that logging nutrition is effective and paradoxically liberating. So around the Northern hemisphere summertime, maybe in the springtime, I can’t remember exactly, I could go back and look, but it doesn’t really matter. I started logging macros. So the grams of fat, the grams of carbs, and the grams of protein, of every single thing that I eat about five or six days a week. So previously, I used to do this every once in a while. So for the preceding few years I was doing this for maybe a week, a quarter, I would log all of my macros. 

(21:42):
And I’ve discovered by doing it consistently, it paradoxically saves me time. So specifically, I use the MyFitnessPal app. There’s lots of apps that you can use for logging macros. The MyFitnessPal one, I don’t know, it’s pretty easy to use. It’s not perfect, but whatever. It’s pretty simple task. And that app works for me. I think it’s probably the most popular app for doing this kind of thing. And so there’s this paradox because while obviously it takes time to enter the macros as you eat everything over the course of the day, it saves me mental time because otherwise I spend this time wondering, oh, have I eaten enough today? Am I just hungry because I’m bored or lonely? And when I’m logging everything, I can see, oh, yep, I have this calorie budget for the day and I’m at 80% of it and it’s 9:00 PM at night. So yes, sweet. I get to have another 500 calories. Great. I know that this hunger is real, and I say hungry. This is the other really cool thing about logging macros is that I never have really felt hungry. 

(23:03):
So in the say six months that I’ve been doing this rigorously, I never feel hungry, but I also never feel stuffed. So my energy is more consistent, and it’s also led to consistent results. So my body fat percentage peaked above 17% in November 2020, and now a little over two years later, I’m down below 11%. So having gone from over 17% to under 11% in two years, that isn’t… If you saw me month to month, there’s probably not visible change, but over the course of two years, it’s a big, big difference. And so I think it’s like how get rich quick schemes are always scams, any kind of platform for changing your physical composition, any kind of process for changing your physical composition. If somebody promises you that it’s going to happen very quickly, it’s probably not true, or if it is true, it’s going to be quite unpleasant and the results might not be lasting. 

(24:11):
But with things like logging the exercise that you do and the amount of calories burned, as well as working with a nutritionist to figure out what number of calories you should be having every day, or maybe like I do logging macros every day, you can see tremendous results slowly. But I think that those will also then be lasting results. And this show, I’m not a medical professional, so hedging everything that I’ve just said with that, but hopefully that sounds like reasonably sound advice from somebody who isn’t licensed to give it. And yeah, so that logging nutrition has not only been time saving for me mentally, for my attention, and not only led to this decrease in body fat percentage, but it’s also meant that combined with consistently going to the gym five to six times a week over the last year, and a serious commitment to mobility, that’s been a big thing for me. 

(25:18):
So as a relatively big person, mobility can be an issue and can be liable to injuries. But I’ve been using an app called Pliability at least six days a week to just stretch out my whole body. Pliability has, it’s a yin yoga platform targeted at athletes originally, specifically targeted at CrossFit athletes. So yeah, so between lifting consistently, going to the gym consistently five to six days a week, eating well, and having this commitment to mobility, and then also just in the last few months, getting involved in Team CrossFit competitions. So having teammates that I don’t want to let down, including actually Natalie, my operations manager, is one of my teammates that I’ve had. And so being in competitions with the team, wanting to be ready for those competitions, and then working out with teammates, it’s just made working out so much fun. It’s been easy to stay consistent, to push yourself a bit. Study show that whether it’s running or weightlifting or whatever, if you do it with other people, you will do it for longer, not only in the individual workout, but also in terms of commitment to that sport. 

(26:29):
So as a result of all those factors, nutrition, weightlifting consistency, mobility commitment, and working together with other people at weightlifting, I’ve had a crazy, crazy year for personal records. So smashed records in all the key power lifts, deadlift, back squat, front squat, bench press. In Olympic lifts, I had enormous PRs this year on the clean, the jerk, the snatch, on classic CrossFit workouts like Grace and Fran. And also just speed things like a 400 meter run. So peaking on all those things because of the consistency, it’s simple. It’s like color by numbers, so anything related to your physical fitness or health nutrition, it’s all about being consistent and everybody knows that it’s hard to be consistent. But I do also have frameworks to help out with that. So you can check back to episode number 538 when I talk about habit tracking. And so that can be a key to developing the habits that you want around whatever, whether it’s fitness related or profession related, or personal life related, whatever. All right, so that’s it. Those are my five lessons for 2022. 

(27:53):
I can’t do everything at once. Orders of magnitude more parameters produce unbelievable AI models. The 24-hour news cycle is exhausting and unsatisfying. Working in-person is way more fun, and logging nutrition is effective and paradoxically liberating. I’d like to highlight though, that there’s one big lesson that I did not learn in 2022. I already alluded this to this already. So in 2021, one of my five lessons was all work, no play makes Jon a dull boy. And yeah, I’m still pretty dull, not a lot of play. So despite getting way better at delegating in 2022, I didn’t really learn how to work less. So demands at my machine learning company, Nebula, as well as at the SuperDataScience Podcast scaled up dramatically. Meaning that even though I wasn’t able to do things like continue on my YouTube videos, the Udemy course, my book, I’ve been working just as much as ever. And so while I’ve had a tiny, tiny, tiny amount of travel and time with family and friends this year, it’s not been nearly enough. This is no way to live. 

(28:58):
It isn’t a life. It’s productive. And I absolutely love seeing the results, the fruits of these labors, and being able to create a podcast in particular for all of you folks. But this is no way to live a life. So many weeks in the year went by with no social activities except at the gym or the bit of banter at work that I now have with people in person. So yeah, little time in 2022 for personal development, playing musical instruments which I love. Enjoying the arts, I barely do that ever despite loving it so much. And I already talked about how reading for leisure is something that I really enjoy. And, yeah, since the pandemic hit, I’ve read one book. Specifically, that book is 4,000 weeks. So you can hear me talk about that book in episode number 606 this year. 

(29:53):
And I also did a follow-up episode a little bit later on episode 618, which also discusses topics from that book, 4,000 Weeks. And 4,000 Weeks, it’s how many weeks you have in your life. And I read it because it seems so perfect. It’s about, you have a limited lifespan, you’re mortal. And so you can’t just work all the time. Productivity techniques are a trap. They just lead to more and more and more work. And life isn’t just about work. Life is also about just enjoying being alive, the simple things and connection with people in a personal way. So read the book in hopes that I’d find some way to get out of this trap of all work and no play makes Jon a dull boy. But I haven’t figured that out yet. So maybe in 2023, somehow, I don’t know, 2023 doesn’t look like I’m going to figure it out either, because I’m going to continue to pour all of my time and energy into serving you and growing our machine learning company Nebula. 

(30:58):
And so in all likelihood, 2023 will be even more work, but it’ll even be a bigger year for this podcast. So that’s the good news for you guys. So we’ll be stretching ourselves to find even bigger names than ever before in data science and machine learning and AI, and cover even more mind-blowing topics and have even more deeply practical conversations for you. I’ll continue to have more experts join me on Friday episodes, which is something we just started experimenting with regularly in 2022. Expect to see more of that in 2023 with guests coming on Fridays and digging into fascinating topics outside the narrow confines of the field of data science, but nevertheless, topics that I am confident will inspire and support you through your career and your life. So combine those things with perhaps 2023 being the biggest year ever in terms of AI innovations and applications we’re surely in for an exciting year. 

(31:57):
So thanks to everyone who works on the podcast. I mentioned everyone that we hired newly this year already at the top of the show, so Serg Masis our researcher, our new writer, Dr. Zara Karschay, as well as Natalie Ziajski, my operations manager. But in addition, I’d like to take a moment here to thank the people who have been working on the show all year bound. So I already alluded to Ivana Zibert, our podcast manager. She’s just incredible. So all year round, a 104 episodes a year, she’s on the ball with every aspect of the production of all these episodes. And they’re always, in my view, at least tremendously high quality about as good as podcasts get out there. So amazing to have Ivana captaining the podcast and making sure that everything happens to an extremely high standard. So thanks to Ivana. Mario Pombo, he’s doing all of the audio and video editing for all 104 episodes all year round, and he just does an incredible job. 

(33:00):
He’s so thoughtful about ways that we can be improving the way that we do things all the time, and being extremely consistent in delivering exquisitely professional high quality all year round on those 104 episodes. So amazing thanks to Mario as well. And I already mentioned our first writer, so in addition to Zara Karschay whom we added this year, we already had Sylvia Ogweng, who’s been working with us for a couple of years. And so again, thanks to Sylvia for doing an amazing job writing and creating podcast pages, show notes, social media posts that are awesome, so easy to read and give great summaries of what’s going on in each episode. And then finally, thanks, of course, to Kirill Eremenko. So Kirill founded the show and he hosted the show up until I took over two years ago. So the first four years, were all Kirill all the time, and I continue to meet with him regularly. 

(34:02):
He has a big influence on the direction of the show, and it’s always such a joy to talk to him. We typically book hour-long meetings and end up talking for several hours, and despite one being either early morning or late evening for one of us, because he’s based in Australia and I’m in New York, but we make it happen. And yeah, it’s such a joy to work with you, man, and thanks again for the opportunity to host this program. It seriously is, yeah, the greatest honor of my life so far. So thanks everyone listening, and thanks Kirill for giving me the opportunity. Thanks everyone on the show, Ivana, Mario, Sylvia, Serg, Zara, and Natalie for supporting me while I make these episodes. And, of course, everyone at Nebula for supporting me, giving me the time to taking time out of my day there to be making this program for all you out there. 

(34:59):
All right, so last year when I lamented that I hadn’t been playing musical instruments enough, I ended the episode by playing a song, and playing a song on the guitar and singing along. And I intended on having that be something that I would do every year. But sometimes things happen beyond our control. And I had a freak accident putting away a barbell recently and broke the end of my right index finger, which you use for holding a guitar pick. And so I can’t play guitar right now. I’ve got a splint on that finger. So I recently came across this story, this legend from Hinduism that I thought was really inspiring. So I thought I would end the year with that instead of a song this year. 

(35:46):
So there’s two characters in this really short legend. One of them is Rama. So Rama is a major Hindu deity, and then the other character is Hanuman. So Hanuman is part monkey, part man, and he’s a devotee of the deity Rama. And so Rama asks Hanuman, what are you? And Hanuman says, when I don’t know who I am, I serve you, when I do know who I am, I am you. So I don’t know, I found that really beautiful. It connects me to you and to everything else. And it makes me think about how when we’re not aware of that, when we’re not aware of that connection that we all have with each other, we can at least focus on service to each other and to the greater good. All right, that’s it for 2022. Keep on rocking it out there, my friend in the new year, and catch you on another round of SuperDataScience very soon.