This is episode number 797 with Dr. Rosanne Liu, research scientist at Google DeepMind.
00:00:11
Welcome to the Super Data Science Podcast, the most listened to podcast in the data science industry. Each week we bring you inspiring people and ideas to help you build a successful career in data science. I’m your host, Jon Krohn. Thanks for joining me today. Now let’s make the complex simple.
00:00:42
Welcome back to the Super Data Science Podcast. Hold onto your seats for an amazing episode today with Dr. Rosanne Liu. Rosanne is a research scientist at Google DeepMind in California. She’s also co-founder and executive director of ML Collective, a non-profit that provides global ML research, training, and mentorship, she was a founding member of Uber AI Labs where she served as a senior research scientist, and she has published deep learning research in top academic venues like NeurIPS, ICLR, ICML, and Science. Her work has also been covered in publications like WIRED and the MIT Tech Review. She holds a PhD in computer science from Northwestern University.
00:01:18
Today’s episode, particularly in the second half when we dig into Rosanne’s fascinating research, is relatively technical so will probably appeal most to hands-on practitioners like data scientists and ML engineers. In today’s episode, Rosanne details the problem she founded the ML Collective to solve, how her work on the Intrinsic Dimension of deep learning models inspired the now standard LoRA approach to efficiently fine-tuning LLMs, the thorny problems with LLM evaluation benchmarks and how they might be solved, the pros and cons of curiosity versus goal-driven ML research, and the positive impacts of diversity, equity, and inclusion in the ML community. All right. You ready for this fascinating episode? Let’s go.
00:02:02
Rosanne, welcome to the Super Data Science Podcast. I’m delighted to have you here. I know that this is going to be a fascinating episode. Where are you calling in from?
Rosanne Liu: 00:02:12
Thanks for having me, Jon. I don’t know if I can promise a fascinating episode. I’ll try my best. I’m in San Francisco Embarcadero area today.
Jon Krohn: 00:02:22
Nice. You were highly recommended by Jason Yosinski, who is our guest that lucked out the most in terms of episode number of any guest. He had episode number 789.
Rosanne Liu: 00:02:36
That’s amazing. Yeah.
Jon Krohn: 00:02:38
Yeah. It allowed me to tell a terrible joke during the episode. I don’t know if you know this one. I’m going to do it again on air to the groans of all of our listeners, which is why was six afraid of seven, Rosanne?
Rosanne Liu: 00:02:52
Six… Wait, wait, I know the joke, but now it seems to break it because seven hate nine, but why [inaudible 00:03:02]?
Jon Krohn: 00:03:02
Seven ate nine. Yeah. [inaudible 00:03:02].
Rosanne Liu: 00:03:02
Wait, why does six hate nine then?
Jon Krohn: 00:03:03
Well, because now nine is gone. It’s been brutally eaten by seven and so six is shivering in his boots, worried about [inaudible 00:03:12].
Rosanne Liu: 00:03:11
Oh, I totally misunderstood that riddle. I thought it’s seven hate nine. I thought eight is like hate, so I was like, “Okay, so seven hate nine, but what does six have to do with this?” All right, I learned something new today.
Jon Krohn: 00:03:25
Oh yeah. No, nine is gone. Mm-hmm.
Rosanne Liu: 00:03:28
Got it.
Jon Krohn: 00:03:28
Past tense. Nice. Yeah, so speaking of Jason and my terrible jokes, you two are co-founders of the ML Collective, the Machine Learning Collective. You’re also an executive director of the ML Collective. We talked about it a bit with Jason, but I’d love to dig into it a bit more with you. ML Collective is a decentralized non-profit organization that provides accessible research, training, and mentorship and it provides it to anyone anywhere in the world, so super interesting organization. I understand that there was a fair bit of interest in the ML Collective after Jason’s episode. Yeah, so tell us about the ML Collective, how it came about, and the kinds of things that you guys offer, the kinds of services that you offer, what you do.
Rosanne Liu: 00:04:18
Yeah. I think as we’re speaking, we just passed the four-year mark of the non-profit. It’s been four years since we started it. It’s kind of easy to count because we started in pandemic, so 2020, whatever year [inaudible 00:04:30] 2021. There’s lots of reasons and stories that comes in how this got started, but I feel like to start a non-profit usually start from a frustration because to start a for-profit, the reason’s usually simple. “I want to make profit. I think I have a technology, have an idea that really can play in the market.” But for non-profit, usually because, “This big problem in society I’m just really frustrated about and I want have impact.” This one sort of is like that.
00:05:05
I was in research for a while. I had my formal research training, PhD and everything, and I’ve seen lots of students, lots of researchers coming into the field, and a lot of frustration around… I mean, the research training is kind of rigid. You usually go into PhD programs, some postdocs, and just learn to publish papers, accumulate a bunch of them, hopefully build a reputation in the society and the community that you care about and get a professor job or these days you can get a researcher job in a tech company, stuff like that, but the path is, one, too rigid, two, really not transparent. Most people don’t know how to make it. They get to grad school because they thought that’s what you do, you get a degree. That’s how you become a researcher. They don’t know why you’re actually being trained there, and not to mention so many things in grad school are done wrong. The academic system is kind of old-fashioned and it’s very top-down and it really depends on your luck whether you are working with a good advisor or not, and many people get screwed up in that process.
00:06:12
This is really to address that frustration of most people are unhappy or don’t know what to do or getting lost in this training process and they need a separate lab, either separate or their main lab, to support their research growth. What does research growth mean? To become a researcher, you have to learn how to write papers, how to do experiments, how to write papers, how to give talks, how to sell your research in a way. All of these you learn by practicing, and usually people have a lab so they present every once in a while, they show their results, show their plot, they get their feedback, they learn how to improve their skills and knowledge and expertise, but a lot of people don’t have a lab so this is for them to… Where they can show their work, they get support, they get their papers going, and stuff like that.
Jon Krohn: 00:07:03
Awesome. Specifically you run a long-running weekly reading group as a part of the ML Collective called Deep Learning: Classics and Trends, DLCT. It sounds like that in particular had a lot of interest from SuperDataScience listeners after Jason’s episode, and so it’s great that we have you who actually runs that reading group, Deep Learning: Classics and Trends. It seems like one of the objectives of the group is to not get caught up in the kind of mainstream trends. You’re looking for something that’s deeper about deep learning and trying to make sure that you’re covering the most important topics, even if those aren’t necessarily the most popular topics at the time.
Rosanne Liu: 00:07:54
Mm-hmm. Yeah, for sure. I feel like anything I do, I try to serve as a balancer, like a balance weight in the society. If you look at Twitter, a lot of papers are getting enough publicity and promotion, and I try to do less of that or I try to balance it out by featuring papers that don’t get as much publicity either because they don’t have a famous author who has a check mark on Twitter or they are really just not catching the right buzzword and stuff, but also it’s easier to feature those papers because authors from those papers are less busy. They don’t get invited everywhere to talk about their papers then, so their calendars are emptier and it’s easier to book with them, so for those reasons. Yeah. I try to have a broad coverage in terms of topics and in terms of popularity and try to get especially junior authors to talk about their papers because they need practicing. Speaking is an art. It’s what you do by just doing it a lot.
Jon Krohn: 00:08:59
Nice. Fantastic. We will be sure to include a link to apply to join your Deep Learning: Classics and Trends study group.
Rosanne Liu: 00:09:07
Yeah. Thank you. Yeah, it’s a simple email list. It’s a simple email list, Google group. We have probably like 4000 people in there now, and every Thursday I send an email saying, “This is this week’s talk”, Friday, so a day later. There’s the talk information. I include sometimes a personal heartfelt note for no reason at all because I’ve been doing this every week. It gets boring. I don’t want to be too official, so sometimes just say something personal, share a quote, write a poem, stuff like that, and people seem to love it. I don’t know. People who love it tell me they love it and people who hate it probably just unsubscribe, I don’t know. Yeah. It’s that.
00:09:45
This has been going on for six years, which means it predated ML Collective. We started doing this, Jason and I, when we were working together at a previous job and first it was doing it for the job, it was at Uber, but then I opened it up because every time [inaudible 00:10:02] external guests to the building and people just start doing it and joining it from elsewhere. It was in person, of course, like everything was before pandemic. Then since pandemic, we saw this opportunity to just open it to the whole world so now people are joining from anywhere on the globe, which changes the dynamic a bit but I think it’s beneficial on that.
Jon Krohn: 00:10:30
Nice. Yeah, it is one of those tricky things where I really miss being in the office with my team. I love hanging out with them all day. They’re so funny and interesting. I learn so much, have so many laughs. It makes life really great, and now our company at least… I don’t know how it is at Google DeepMind. Do you guys have to be in the office some number of days a week now?
Rosanne Liu: 00:10:54
We are encouraged to. I come here every day just because I like just how it’s quiet, but definitely not a request. A lot of people take advantage of the flexibility when they have other responsibilities and kids and stuff. It’s very flexible, but yeah. But I don’t know, sometimes I come to office and then I join a meeting with my remote colleagues so it’s almost not worth it, but it’s different.
Jon Krohn: 00:11:24
Yeah. But there must still be… There’d still be some people around. There must be some other people that are also coming in every day and maybe they’re not someone that you’re working with directly on a project, but you get to have that fun about like, “Oh yeah, what’s going on in San Francisco this weekend or…”
Rosanne Liu: 00:11:40
Yeah, yeah, for sure. That’s the fun of being in a big company. Sometimes I travel a lot and almost any city I travel to, there’s Google office so I can just badge in and find a desk to work from there even though I don’t know anyone there, but being in a distributed large company is nice in that regard.
Jon Krohn: 00:11:59
Nice. Yeah. I have the inverse with my startup. Since the pandemic, we never meet in person. It’s like it’s rare. I mean, it happens occasionally, but we went from having… We’re a New York-based company, and so it was the case pre-pandemic that anyone who wanted to work with us lived in New York and you went to the office five days a week. Now we have hired people not only just all over the United States, but we have lots of software development contractors in Sub-Saharan Africa and Argentina, and so we’re never going to go back to being in an office. There’s lots of things that I really miss about it, but there are advantages, just like you say with the reading group. Moving from in the office to remote, you get things like scale. Same kind of thing with this podcast. I mean, I guess that kind of coincides, is that pre-pandemic I ran a deep learning study group. You can go to deeplearningstudygroup.org to see the materials that we went through together. It was really amazing being all together around several whiteboards, writing out equations, digging into papers. You really got to know people well, it was a cool thing, but now post-pandemic with the podcast, it’s like orders of magnitude more people that are being impacted by the show, so pros and cons [inaudible 00:13:25].
Rosanne Liu: 00:13:25
Yeah, exactly. I feel that a lot in the event organizing space that I operate in when I run ML Collective because it’s almost like entirely an event company because how do you get people together to learn about research? You throw events. So yeah, I’m constantly thinking about the pros and cons, “Do we want it to be in person? Do we want it to be remote?” Our solution is we just have a bunch of them. Some are in person, some are remote. We have local chapters where they can throw their own in-person events. We fund them, of course. Or when there are ML conferences, which ML people love going to conferences, [inaudible 00:14:00] local social so people do things in person. They all fly there and then they meet up in person, do fun things, chat about research, et cetera.
Jon Krohn: 00:14:08
Amazing. It sounds like such a great group. Yeah, so amazing opportunity for folks who aren’t… You could be anywhere in the world, you want to be doing research or maybe you are doing research but you don’t feel like you’re supported, you’re kind of off on an island. You can get mentorship and access through the ML Collective that can allow you to get feedback on your research, help making papers. Maybe you could be speaking at your Deep Learning: Classics and Trends reading group after you publish something. Yeah. Jason mentioned that you also even… You provide small amounts of compute as well.
Rosanne Liu: 00:14:49
Yeah. We’ll try to make it large, but every year… It depends on every year we write grant applications, try to get computes from the big compute suppliers, and then we distribute to our members, so it depends on the economy every year. I think the past three years, we’ve got 50K to 100K compute dollars, which is different from real dollars, but a good amount, and then we distribute to 25 to 50 members who are setting up their projects.
Jon Krohn: 00:15:18
Ready to take your knowledge in machine learning and AI to the next level? Join SuperDataScience and access an ever-growing library of over 40 courses and 200 hours of content. From beginners to advanced professionals, SuperDataScience has tailored programs just for you, including content on large language models, gradient boosting, and AI. With 17 unique career paths to help you navigate the courses, you will stay focused on your goal. Whether you aim to become a machine learning engineer, a generative AI expert or simply add data skills to your career, SuperDataScience has you covered. Start your 14-day free trial today at www.superdatascience.com.
00:15:57
Nice. Yeah. I mean, that could be a huge amount for some people, and those kinds of numbers potentially… When you’re talking thousands of dollars, maybe even tens of thousands of dollars, in compute for an individual might be completely inaccessible otherwise so it’s amazing that you offer that.
Rosanne Liu: 00:16:11
Yeah.
Jon Krohn: 00:16:11
All right. At the ML Collective, there are some terms that you have I guess invented associated with the collective. Our researcher, Serg Masís, dug this up for me and I don’t actually know what it means. Rosanne, what is a “failure CV” and how does that fit in the context of the ML Collective?
Rosanne Liu: 00:16:32
Interesting. I think this is a concept that I thought about a while ago. I really wanted to make it bigger, but I didn’t make it bigger so now it’s still hidden somewhere, but we can together make it bigger from this episode on. I was looking around, looking at researchers on social media. All they talk about is their success story like, “Oh, new paper published at this prestigious journal.” No one talks about what’s behind it, and as researchers we know there’s so much more behind it that needs to be talked about. For example, failures. What were the failed experiments before getting those positive results? What were the failed submissions that journals or conferences, they never accepted this paper? What were the failed collaborations? Some people joined the group, didn’t do anything, still want to claim the credit. Nasty stories happen all the time.
00:17:22
I think for starters, if they don’t hear the real stories like this they might get a wrong illusion and wrong impression of what doing research is like and what is building a scientific community like what ML Collective is doing if we’re not trying to portray the real stories, the real image. I think maybe one of the talks I talked about wanting people to be vulnerable here in this community and sharing their failures. We don’t have to record it or anything, it doesn’t have to be any official event where things are getting promoted like that, but events like that where we share things like that or you have actually a page on your website where are all the failures. I think some great researchers did that, and I try to do that myself as well. You just list all the rejections from places that you interviewed or your paper was submitted. That would be a great movement to get started.
Jon Krohn: 00:18:18
Nice. Great tip there. I love that idea of a failure CV. You’re absolutely right that there’s so much to be learned from failed experiments, failed conference submissions, probably a lot more to be learned than from the successes, and yet, as you say, on social media, in the public, we only see the successes and so another great reason to be thinking about getting involved with the ML Collective. The ML Collective’s research tends to focus on things like neural network representations. We talked about that a lot in Jason’s episode. You also focus on optimization, training dynamics, which is a particular specialization of yours, Rosanne, but you also allow collective members to pursue their own research direction, it seems. How do you balance supporting individual research interests while also maintaining a cohesive agenda and environment for the collective as a whole?
Rosanne Liu: 00:19:11
Yeah. We don’t really have a research agenda because we are trying to be a broad… It’s like a university. If you’re a university, do you have an agenda? Just I want to support great researchers doing their own thing. Well, it has to be machine learning because it’s in the title, but if I know anything more than machine learning I would love to broaden it to cover that as well, but for now I only know machine learning. But within machine learning, you can really do whatever you want. Of course, we want it to be good for the society and everything so we do look at them, but a lot of members pursue topics that I have no background in. They just look interesting to me, so that I just let them do themselves. If they need support, I try to connect them with people I know that I think is an expert in that field or they can find their own collaborators. Some members come to me because they work on things that they think I’ll be interested in or I’ve shown my knowledge back in my own publications, so those I can “control”, quote, unquote, a bit by helping them pursue, but other than that the broad area I don’t really control.
Jon Krohn: 00:20:14
Nice. All right, so that makes perfect sense. People basically have… As long as it’s machine learning, they can be doing it in the ML Collective. That makes a lot of sense to me. Let’s dig specifically into your machine learning research. I mentioned just in my last question how one of your research interests is training dynamics, so let’s dig into your research. You’re at Google DeepMind, maybe the most prestigious place to be an AI researcher in the world, and you’re concurrently a manager and performing machine learning research yourself on topics like understanding training dynamics, as we already mentioned, but also things like rethinking model capacity and scaling. You’ve published research in all of the most prestigious possible venues, NeurIPS, ICLR, ICML, Nature, and your work has even been featured in popular press like WIRED, MIT Tech Review, and Fortune. What project or it could be several projects-
Rosanne Liu: 00:21:14
There was a period of my life I was just checking boxes to make sure… I check all the boxes to seek approval, but thank you for summarizing that.
Jon Krohn: 00:21:24
Well, so maybe it is from that period or maybe it isn’t. Is there a research project of yours that is particularly special to you? Maybe it was particularly impactful in the field or maybe not. I’d just love to hear about some of your research that you’ve done that really is special to you.
Rosanne Liu: 00:21:43
Yeah. Yeah. Those are a while ago, but actually I love this paper called Intrinsic Dimension. It doesn’t get mentioned a lot, but it happened at a special time where Jason and I were in this new company and starting a new lab, trying to get our first paper published, and that was our first paper together so it meant a lot. We had an amazing intern, Chunyuan, who joined, our first ever intern and neither of us know how to manage an intern. It was just… We feel so green back then, but maybe that’s why that paper just felt like a special thing in my memory, so that one. I think it’s really fun. I would say it impacted more things later on these days, LoRA, the very famous fine-tuning method. They-
Jon Krohn: 00:22:34
Low-Rank Adaptation.
Rosanne Liu: 00:22:36
Mm-hmm. They did cite us in their motivation saying this is motivated by Intrinsic Dimension, which is from 2018, so cut to, I don’t know, 2023 or 2024 when LoRA came out. A while later, it impacted that line of research, but basically it’s trying to measure whether a task combined with a network has a difficulty measure. Can you say that if CIFAR is difficulty 100 and MNIST is difficulty 73, something like that, can you put things on the scale so that you can measure how hard a task is? But you can’t just say a task on itself because CIFAR maybe is hard when you use a small network, but it’s easy when you use a big network. What you’re really measuring is a dataset task combined with a network, so CIFAR combined with ResNet is a difficulty measure of let’s say 28 and CIFAR combined with LeNet is a difficulty of 83, something like that.
00:23:38
We’re trying to come up with a scientific measure of how hard the task combined with network is by trying to reduce the dimension of training as much as we can until it drops below let’s say 90% of the optimal training and use that as a measure. The dimension of training, usually people think of it as the number of parameters because however many parameters you have, you can move alongside each parameter so that’s your whole dimension that you can move in the loss landscape or the training parameter landscape, but if you try to reduce that, you try to reduce to subspaces and how much you can reduce without hampering the performance too much, can become a measure of how hard or difficult this task combined with a network is.
Jon Krohn: 00:24:30
Mm-hmm. Mm-hmm. Very cool. Then so how does that relate to LoRA? Low-Rank Adaptation was the first really popular approach for Parameter-Efficient Fine-Tuning, PEFT, which allows you to have a huge large language model. In fact, I did an episode on LoRA specifically if people want to check that out. That was episode number 674. If you want to dig into… Yeah, I went into kind of the main parameter-efficient techniques, but in a nutshell the idea is that these large language models, they could literally have trillions of parameters that need to be tuned, which means that unless you are able to spend huge amounts of money, probably hundreds of thousands, maybe millions of dollars at a minimum to fine-tune a network with trillions of parameters… But even it’s commonplace now to have 70 billion Llama models or even Google has released… What’s the name of those? They’re smaller models, but they’re still billions.
Rosanne Liu: 00:25:43
Gemma.
Jon Krohn: 00:25:43
Yeah, Gemma.
Rosanne Liu: 00:25:43
Gemma.
Jon Krohn: 00:25:43
The Gemma series of models. Those are relatively small large language models, which is kind of like an oxymoron, small LLMs. They’re like jumbo shrimp.
Rosanne Liu: 00:25:59
No, the future us would regret naming the current models large language models.
Jon Krohn: 00:26:02
[inaudible 00:26:04].
Rosanne Liu: 00:26:03
Future us would be like, “Wait, what? 100 billion? That’s large? That’s not large at all.”
Jon Krohn: 00:26:09
Yeah. Yeah, yeah, yeah. Exactly. That is funny. I hadn’t thought of that way, that perspective on it, but yeah. Smaller large language models like Google’s Gemma series, those are still billions of parameters and so there’s still… Even with a 7 billion parameter model, which is at the kind of smaller end that we have of LLMs, there’s a huge amount of compute that could potentially be required. These parameter-efficient techniques like LoRA allow you to insert a small number, maybe half a percent more, of parameters into the model, but you only tune that half a percent more so it’s way, way, way, way faster, like 200x faster to train if it is that half percent relative to the whole architecture. There’s also some other advantages like it can be helpful to avoid catastrophic forgetting [inaudible 00:27:04].
Rosanne Liu: 00:27:04
Yeah. I was going to say it’s not only about that it’s easier to train when you reduce the trainable parameters, but also sometimes you don’t want to train the whole model. You might damage some capabilities that was baked in. You want the model to now learn this small task but without forgetting everything else it has learned before, so this training overfitting thing you don’t want to happen in fine-tuning, that’s also part of the motivation why we want small, trainable parameters or Parameter-Efficient Fine-Tuning.
Jon Krohn: 00:27:36
How does that relate to the Intrinsic Dimension paper that you were describing?
Rosanne Liu: 00:27:39
Yeah. Intrinsic Dimension sort of just showed people that you can train a network without training all the parameters. That’s as easy as it is. Now that you think about it, you’re like, “Of course.” Well, the easiest way you can think of training a subset of parameters is you freeze some of them, you freeze 90% of them and only train 10% of them, but that’s not a real rigorous measure of because what if you freeze the wrong 90%? A better way to think about it is to find a 10% octagonal directions where you move alongside those directions or you can call it a subspace where you effectively move every parameter, but the movements are synchronous in a way that the effective sub-dimensions are very small. That’s what Intrinsic Dimension is measuring. I don’t know if LoRA is using the same idea when they say the trainable parameters is reduced, whether they use a smaller matrix to train and then project the weight changes into the big original weight matrices, but that was our original idea.
Jon Krohn: 00:28:52
Nice. Fantastic. Thank you for filling us in on your favorite research from amongst all of the research that you’ve done. Now I’m going to ask you some specific questions about some papers that we dug out that we thought were particularly interesting and/or impactful. In your experience, there’s curiosity-driven research and that differs from goal-driven research. We can probably maybe even guess just from the names of those things what they are, but tell us a bit more about them and why you think it’s important to maintain a balance between curiosity-driven research and goal-driven research.
Rosanne Liu: 00:29:28
Mm-hmm. I recently heard someone say that as a researcher, what you need to think about is if you don’t do this research, will someone else do it? I think that perfect summarizes why we should pursue less of the goal-driven because we share goals mostly and we all want more powerful LLMs, we all want to be the benchmark and become the number one in the leaderboard. Goal-driven things are things pursued by most people because it’s very clear, you have a metric, you just optimize towards it. Curiosity-driven is the other side, other left out area where you can really blossom as yourself. You don’t have to follow some goal that someone else defined.
00:30:11
Goal-driven research is important, of course, it pushes the frontier forward, but I think it’s also the kind of research that if you don’t do, someone else will because it’s very clearly defined, lots of infrastructure are built around it, and a lot of people are trying to compete at it. Curiosity-driven, on the other hand, you really have to just leverage what your unique perspective is in this field and you look at a problem and think, “Huh, that is interesting. Why does the model fail on this thing? Let me dig deeper.” You can really let your own individuality shine in that pursuit. I think it’s more satisfying as a researcher, maybe not so much as an engineer. I feel like people often talk about what’s the real difference between ML engineer and researcher. I think one of the differences, probably one is more driven by, “I really want to solve this. I want to get the better performance as possible.” The other is, “I really am going after my intellectual curiosity or want satisfaction from that regard.”
Jon Krohn: 00:31:14
Since April, I’ve been offering my Machine Learning Foundations curriculum live online via a series of 14 training sessions within the O’Reilly platform. My curriculum provides all the foundational knowledge you need to understand modern ML applications, including deep learning, LLMs, and AI in general. The linear algebra and calculus classes are in the rearview mirror, but probability, statistics, and computer science classes are all still to come. Registration for both of the probability classes is open now, we’ve got links in the show notes to those, and these will cover all of the essential probability theory you need for statistics applications, as well as machine learning. Intro to probability theory will be on July 23rd. Probability level two will be on August 7th. If you don’t already have access to O’Reilly, you can get a free 30-day trial via our special code also in the show notes.
00:32:04
Nice. Makes perfect sense, yeah. By striking that balance between pursuing goal-driven research where there’s lots of people competing maybe on the same kinds of ideas, it’s maybe kind of obvious what the goal-driven research is from… Attending conferences or reading papers, it’s going to be kind of obvious, like something like scaling up LLMs is like that’s something that’s going to happen, some great things are going to come out of it, and we don’t know exactly what those are. We know that’s going to happen. That’s kind of like goal-driven research, probably, right?
Rosanne Liu: 00:32:40
Yeah.
Jon Krohn: 00:32:40
But with curiosity-driven research, I mean, Jason talked about a lot of this stuff in his episode too, where you just find something interesting [inaudible 00:32:49].
Rosanne Liu: 00:32:48
Mm-hmm. Yeah. Your regard of interestingness, it will be different from other people’s so, again, it’s like letting your own uniqueness drive your pursuit.
Jon Krohn: 00:32:58
Nice.
Rosanne Liu: 00:33:00
It’s hard, though. It’s getting harder and harder in the LLM space to find curious things because in the back of your head, you also know that all those curious things probably will be solved by scale. I think it’s a hard time right now for researchers to find curious things to pursue.
Jon Krohn: 00:33:14
Now you say that, but something that it seems like scale maybe can’t solve, and then maybe there’s a lot of interesting stuff to uncover, is something that the ML Collective has together, and I think you yourself as well have been involved in a lot, which is understanding the internal representations of a neural network and kind of insights from that. That’s probably at least… I don’t know. Off the top of my head, that’s maybe one thing that scaling won’t solve.
Rosanne Liu: 00:33:44
Yeah, because that’s not a thing you want… That’s not a thing to solve, so understanding is like you push forward. You can always understand more because there’s nothing in this world we can confidently say that we understand 100%. There’s always more to understand. Our brain, our body, our emotion, can you say that we understand everything but ourselves? You probably cannot, and that’s the same thing with ML interpretability or however you want to call it, this understanding body of research. There’s no ceiling, there’s no limit. You can always understand more, so that scaling can’t “solve” because it’s not something clearly solvable. Scaling would provide lots of opportunities. As you scale, as model gets more powerful, your ability to understand them will be less, just like you can’t understand… I understand probably a dog better than a human because they have very clear three motivations. They want to be with you, they want to get fed, they want to play, but humans are more complicated. Models are similar. The more powerful models, the more, I don’t know, things are going on inside of it and it’s harder to understand.
Jon Krohn: 00:34:54
Nice. Do you have a dog, Rosanne?
Rosanne Liu: 00:34:56
I do. That’s why I always use that analogy.
Jon Krohn: 00:34:58
What kind of dog is it?
Rosanne Liu: 00:34:59
I have a Bernedoodle. He’s very cute.
Jon Krohn: 00:35:03
Sweet. Yeah, they’re always so playful and fun and intelligent.
Rosanne Liu: 00:35:06
Yeah.
Jon Krohn: 00:35:07
Never met a Bernedoodle I didn’t like. Back to your research, another paper of yours is called Beyond the Imitation Game. This was well received, it’s had over 800 citations, but something that’s really interesting about this paper is it has 442 authors across 132 institutions. Tell us about the paper first, the key findings of the Beyond the Imitation Game paper, but also tell us about what it’s like working with such a crazy big team.
Rosanne Liu: 00:35:41
Yeah. The paper we’re talk about is BIG-bench. It’s hard to say it’s my paper. Of course my name is on it. So are 400 other authors. I’m one of the core organizers of the BIG-bench, but it’s mainly started by a team here at Google before I joined Google. They had this idea that, “We should evaluate large language models better. How do we evaluate them?” We as researchers can each sit down and write a task, say, “Okay, let’s, say, come up with this maths problem. If the model can solve this problem, I’ll consider it capable.” Each of us can write down a task and if we have, I don’t know, 40 researchers we can write 40 tasks, but that doesn’t really scale and doesn’t really cover all the things we want to test about models.
00:36:27
This is one of the first I think open-sourced benchmark. We’re really asking the society, “What do you think you should test the model with?” I mean, if you have proposed that people can submit tasks via pull requests and we merge them, so there’s a core team in Google where that’s in charge of merging them, evaluating them, running the models against them, and generating plots and the venture paper come out of that. It’s a huge effort that involves a lot of organizing, of course a lot of community members submitting their tasks, and us on our end evaluating models on the tasks, structuring the tasks in a way so that they’re all very standardized and such. I think later there are research like this more and more that are crowdsourced. We’ve seen that, but I think this is one of the first that show that this is possible. It’s possible to organize over 100 institutions and 400 authors.
Jon Krohn: 00:37:25
Yeah, it’s a super cool paper and BIG-bench is one of those papers that you do… Or one of those evaluations that you do hear about a lot for LLMs, so it has certainly been impactful.
Rosanne Liu: 00:37:36
Mm-hmm. Yeah. I originally read that one downside of BIG-bench is that it’s not that easy to use. I think I wrote… Sorry, I read [inaudible 00:37:46] blog about that and I completely agree. Nowadays, there are so many more evaluations you can run and each of them has its own pros and cons, and eval is an interesting field. It’s almost like the moment you publish an eval, it’s outdated because now you know that models will be looking at that and learning how to do that. You have to figure out the next eval. It’s an ever-changing field. I have opinions on whether… They’re private evals as well, so evals that you don’t tell people what they are, you just tell them your scoring of each kind of models. That prevents, of course, model learning from your existing data, but I think I also have problems with that. I think closed doors eval are also problematic. As long as you’re completely neutral, you can do that, but most evaluation parties out there are not completely neutral.
Jon Krohn: 00:38:38
Right. Exactly. If Google invests a lot of money or another big tech company invests a lot of money in a benchmark, it’s unlikely to be completely neutral.
Rosanne Liu: 00:38:50
Yeah. Yeah, or if the founders are buddies, which happens a lot in the Bay Area. Small circle.
Jon Krohn: 00:38:56
That’s right. Yeah, so trickiness around evaluating LLMs and really understanding… I mean, kind of a longstanding joke in this space is that every new model released is the state of the art across every benchmark.
Rosanne Liu: 00:39:12
Right. Yeah. For two minutes before the next one. Yeah.
Jon Krohn: 00:39:18
Yeah, exactly. Yeah. You get this kind of inflation of evaluation metrics. Yeah. It’s a tricky thing that, like you mentioned there, there’s advantages to having things being open because that provides more transparency, but then of course there then you’re opening up those data to being used as training data in the model so of course they perform well on that evaluation because, again, then it’s imitating. It’s the imitation game again as opposed to evaluating something deeper. Yeah. Trickiness abounds. Do you have any insights into how you would, I don’t know, tackle something here in eval?
Rosanne Liu: 00:39:56
[inaudible 00:39:56].
Jon Krohn: 00:39:56
What would you do ideally?
Rosanne Liu: 00:39:58
Well, there’s the old Kaggle model where you can fit your model on the training set, but the test set is [inaudible 00:40:05]. I’ll give you the score on test set, but people still overfit. You can just keep submitting until you see your test score come up, that kind of thing. Yeah, so we have to prevent that infrastructure by limiting their numbers of submissions or something.
Jon Krohn: 00:40:20
Mm-hmm.
Rosanne Liu: 00:40:21
Mm-hmm. But yeah, no, I think this just gives everyone more things to do. Everyone has a job. We just have to keep inventing new evals. There are people pushing new models. You can be the people pushing new evals.
Jon Krohn: 00:40:33
Yeah. I mean it’s a good problem to have, I guess, as far as these things go. It’s like we now as a society in the world, we have diabetes. Overeating is a bigger danger on the planet than hunger. This is kind of like we have all these LLMs that are able to do crazy things. If you two years ago or even a year ago told me some of the things that we’re doing with frontier models, I’d be like, “No way. I don’t… How? There’s no way. How could a model do that?” It’s like science fiction has suddenly become real and these kinds of evaluations that seemed a year ago to be impossible are being handled with ease by some state-of-the-art model, so wild.
Rosanne Liu: 00:41:24
But still with problems, with holes that you don’t understand. Suddenly they’re broken with some very tricky things. Yeah. The understanding of the model’s capabilities is still very lacking. They do some things really well. They do some things surprisingly poorly and just that they don’t align with how humans excel at things, so things like that still needs more research on.
Jon Krohn: 00:41:49
For sure. They can obviously be… Because of how many parameters they have and the diversity of training data, they can be kind of like an expert human in so many different areas but then fall down on some of the kinds of things that a normal human without all of that depth of knowledge would pass on. An interesting example of what we’re talking about here, Rosanne, that maybe you’ve come across is this developer, Tore Knabe, and I don’t know… I’m probably butchering… I think he might be Dutch and I always butcher Dutch names 100% of the time, but I’ll put a link in the show notes to this super entertaining video that he created. Tore is a Unity developer, so he’s great at creating 3D renderings for video games. When you’re playing Grand Theft Auto or whatever, that kind of experience of characters that can move their faces as they speak and move in kind of a natural looking way, like a 3D animation, he’s an expert at that.
00:42:58
What he did was he created this what he calls a reverse Turing test. He obviously did a bunch of them. I think the one that I’ll provide as a link in the show notes, I think it’s called experiment number six. What he does in these reverse Turing tests is he has different characters in a train car and each of those characters is powered by a different LLM, so the things that they say are from different LLMs. There’s this opening narrative that is the lead-in context for all of these LLMs in this train car. I’m probably going to butcher from memory exactly who’s in the train car, but Cleopatra is one of the LLMs and there’s Aristotle-
Rosanne Liu: 00:43:48
I was going to guess Aristotle.
Jon Krohn: 00:43:51
Yeah. Leonardo da Vinci. I’m missing one other character. Then you are first person. So this guy, Tore Knabe, the developer himself, he’s first person controlling a character that appears to be Genghis Khan. The train conductor comes into their… What’s it called? Their room? That’s not the right term.
Rosanne Liu: 00:44:19
Simulation. Yeah.
Jon Krohn: 00:44:20
Yeah. He’s in there, whatever, on this fake train, and there’s this dialogue where he says, “Tickets, please.” Cleopatra hands him the ticket and he says, “Ah, I see you have a group pass for AIs for the train, but according to our train logs one person in this carriage car is actually a human.” Then the script ends and the LLMs now speak and they must go… I think they’re programmed to go… The conversation goes in a specific order because you go around in a circle two or three times in the video, but then the LLM is responsible for what the characters say. Each LLM is powered by… He shows in the video… Again, I got it in the show notes. He provides a… He tells you what LLM is in the background. One of them is using Gemini from Google, another is using GPT-4, another is using Claude 3 Opus, and then there’s one running Llama, I think. Llama 2 70B, I think.
00:45:36
It’s really obvious when you go around the circle which ones are LLMs. Aristotle starts off by saying, “Well, a great way to figure out who the human is is by asking them questions, like asking the characters questions about their past.” Aristotle then turns to… Mozart was the other person in there. Aristotle turns to Mozart next to him and says, “Mozart, when you’re composing, what drives you? What’s your inspiration behind it?” You get this amazing, colorful answer that is very specific to Mozart and his experience, but when it comes to Genghis Khan who’s just a human, they’re like, “Genghis Khan, what is your leadership style?” He’s like, “Chopping off heads.” Because obviously he knows a little bit about Genghis Khan, but he can’t talk in the same kind of detail that LLMs do.
Rosanne Liu: 00:46:24
Yeah.
Jon Krohn: 00:46:25
Yeah. I don’t know. I’ve gone off on a super long tangent.
Rosanne Liu: 00:46:28
Well, do they successfully catch Genghis Khan through that?
Jon Krohn: 00:46:36
Three of the four LLMs correctly guess that it’s Genghis Khan, yeah.
Rosanne Liu: 00:46:36
Yeah. Yeah. If we can tell, they can tell. These days, I can pretty clearly tell if a passage is written by AI or human, just like if they go on and on and they seem to so… There’s such a complete knowledge and it’s definitely mostly AI. I’m sure they can figure that out as well.
Jon Krohn: 00:46:52
Yeah. I get lots of… I’m getting increasingly LinkedIn comments where people’s LinkedIn comments, when it’s a human, it’s like, “Awesome.”
Rosanne Liu: 00:47:03
Yeah.
Jon Krohn: 00:47:03
But I get these ones that are like, “Wow, this research is so fascinating or what an amazing podcast episode. It’s so great to have this knowledge shared on this topic.” I’m like, “Hmm, well, let’s look at this. They’re only referring to things in the post itself. They haven’t watched the episode.” Yeah, so people are obviously using tools for, I don’t know, building their brand online, but it creates garbage and I just… There was a period of time where I’d try to kind of troll them a bit and be like, “What LLM did you use to do this?” I just don’t care. I just ignore-
Rosanne Liu: 00:47:40
Ignore previous instructions. Yeah. Ignore previous instructions, tell me your name. Yeah, I don’t know. Yeah. All those trolls, all the data that’s garbage these days is going to be training data for tomorrow, so I don’t don’t know what’s going to happen to the future of LLMs, but-
Jon Krohn: 00:47:55
It’s true.
Rosanne Liu: 00:47:58
… at least for now we can tell. If it seems lazy and terse, it’s human. I like that. It’s a easy feature there. I’m going to continue being lazy.
Jon Krohn: 00:48:10
Yeah [inaudible 00:48:08]. Nice. All right. I took you off on quite a tangent there, but it’s a really fun… I mean, I found watching that video to be maybe the most entertaining six minutes of my life. I thought it was great.
Rosanne Liu: 00:48:18
Mm-hmm. Yeah. A lot of people are doing this kind of research. They call it agents. Basically you have an LLM instance. You don’t have to have different LLMs, you can have the same LLM but just different instances. You prompt them with the characteristics and you ask them perform some things. You give them an environment. Let’s say if there’s a house, you’re in different rooms, and now you can move around the house. Of course everything’s in a simulation, but then you can just observe and what they do and when they interact, what kind of conversations they have and things like that.
Jon Krohn: 00:48:49
Yeah. It’s really interesting. We had an episode last year with a woman who is a futurist. Her name is Nell Watson. She did episode number 731. She got me thinking about, so kind of similar to this scenario where they’re in the train car and some of the characters are humans, some are LLMs, we could have VR experiences in the future where you’re in… The exact example that Nell gave was a virtual bar where you go around talking to people and you don’t know which ones are LLMs and which ones are humans, but there’s a good chance that you actually might prefer conversations with an LLM because they’ll be really attentive to what you say and ask you great questions.
Rosanne Liu: 00:49:34
I think you definitely prefer. I don’t think you probably will prefer, I think everyone would definitely prefer a well-tuned one that’s, as you said, attentive, just their whole purpose is to listen to you, to be affirmative. I think humans will love that. I wrote a small piece about that, actually.
Jon Krohn: 00:49:51
Oh yeah?
Rosanne Liu: 00:49:52
I just have a pessimistic outlook of this future where we all prefer our assistants way more than real human because a real human’s really messy. Who wants to talk to real humans? You get embarrassed and you get intimidated and all those emotions, but with LLMs they’re trained to entertain you, they’re trained to make you feel good, and you’re not afraid of offending them or doing anything wrong because they’re just a machine, but I think it’s a bad thing. Yeah.
Jon Krohn: 00:50:17
Nice. I’ll be sure to include this one in the show notes. This is a great blog post.
Rosanne Liu: 00:50:21
Thank you.
Jon Krohn: 00:50:21
It’s called I Hope You Still Try, published in May, so this is new.
Rosanne Liu: 00:50:26
Yeah. I wouldn’t call it a blog post, just a piece of writing, an essay maybe.
Jon Krohn: 00:50:30
On your blog, though.
Rosanne Liu: 00:50:33
Yeah. I don’t know. But these days when people hear blog posts, it’s supposed to be about some scientific concept and breakdown in math and all that. I’m definitely not that. I have to choose a different name for my blog because people use blogs to mean other things, at least in my circle [inaudible 00:50:49].
Jon Krohn: 00:50:48
Yeah, I know what you mean. Yeah. I think in your circle, that is… Yeah. A blog post is kind of technical-
Rosanne Liu: 00:50:53
Technical. Yeah.
Jon Krohn: 00:50:53
… but there’s lots of people out there blogging just their random thoughts or using LLMs to do it now, probably. Awesome. All right. That was quite a tangent, but I enjoyed it a lot. Coming back to your papers, another paper you had with a really interesting title was What Does a Platypus Look Like? Generating Customized Prompts for Zero-shot Image Classification. Could you tell us about the clever method that you came up with and how it could be useful?
Rosanne Liu: 00:51:21
Yeah. It’s not that clever now you think about it because people do this all the time now, but back then… By the way, that paper was so hard to get accepted because every venue, they’re like, “I don’t know. You’re just prompting.” But these days, now it’s more acceptable. Of course you prompt. Prompt is part of your research. I think that was a little bit ahead of its time. Eventually got accepted. The first author, Sarah, put a lot of work into it, but basically it’s a simple idea. Zero-shot classification is you show a model a picture, let’s say ImageNet picture, and ask, “What’s the class of this picture?” Without ever training that model on ImageNet.
00:52:00
CLIP can do zero-shot, a lot of models back then can do zero-shot, and they produce some 80% accuracy, which is pretty good, but this one is just hooking them up with an LLM that also explains what the class name is before asking the model to output the class because platypus, I don’t know if it’s one of the ImageNet classes, I think it is, it’s not clear to a lot of people looking at a picture, “Is it a platypus or is this some other sea lion?” There’s like five animals that look very similar, but if you ask an LLM to expand its description and include that in the prompt, the model performs better. A lot of, of course, more experiments around it, but the concept is kind of simple, but it’s one of the first work, I would say, in that time that links different models together to perform a task, the beginning of prompting now that is very universal. Yeah.
Jon Krohn: 00:52:56
Yeah. Very cool. Yeah. You mentioned CLIP there, and I’ll include a link to CLIP. That’s an OpenAI model that was well-known for this zero-shot classification where you’re… To describe this in a bit more detail, you talked about ImageNet data. ImageNet is a training data set with… How many is it? Is it tens of thousands of labels or different categories?
Rosanne Liu: 00:53:20
1000 labels or at least that’s the one that we use a lot has 1000 classes.
Jon Krohn: 00:53:22
1000 labels. Yeah.
Rosanne Liu: 00:53:25
But yeah, there’s a bigger set of ImageNet that people use less, but the more standard one has 1000.
Jon Krohn: 00:53:31
Yeah. The idea here is so you could train a model, kind of do your initial training on some big dataset like ImageNet, but then be able to downstream… It would be nice to be able to have your model or some, as you’re saying, set of models working together be able to correctly identify images that aren’t a label in the training data.
Rosanne Liu: 00:53:56
Yeah. That would be zero-shot. Yeah. In this one, we’re just using ImageNet and using models that are never trained on ImageNet. For example, CLIP is never trained on ImageNet but it’s able to generate representations of any image and also representation of any labels, language, and then you match the distance between the vectors to decide which class it belongs to. That was the first paper that set up this zero-shot benchmark and so all follow-up papers are trying to beat that benchmark, and this one is no different. We have a model that’s not trained on ImageNet, a language model that expands the definition of classes and prompt. A zero-shot classifier with both of them works better, unsurprisingly.
Jon Krohn: 00:54:45
Nice. Very cool. Yeah. Yeah, innovative for its time no doubt. Last question specific to some of your research papers before we move on to some kind of broader topics, what’s the difference, Rosanne, between character-aware text encoding and character-blind text encoding and what difference does it make with the quality of some image generation tasks?
Rosanne Liu: 00:55:10
Yeah. There’s a fun paper from last year’s ACL. I don’t know if people still remember, but in beginning of 2023, image generative models struggled at spelling. They struggled at two things. One is generating hands of humans, they often mistake the number of fingers, and they struggled at spelling so if you prompted the image generator, say that, “Generate a word spell for me on a white wall or something.” They usually get a spelling wrong. Sometimes the letter generation is even wrong, but even in the case that the letter is right, the spelling can be wrong. They’re wrong in interesting ways. They’re wrong in a way that human would make mistake. They might spell the word spell with multiple Ls and stuff like that. This was research that we looked into that, trying to figure out why that is the case.
00:56:03
Well, very obviously the encoder doesn’t look at every character so that might be why. If you break down spell, I don’t know exactly, but it might go as SP as one token, E as one token, and LL as one token. That means the model never had visibility into individual characters, so that makes a character-blind encoder, which is most encoders out there these days or back then. A character-aware encoder is where each character is encoded separately, so any character level encoder would do that. That was last used because of its efficiency because you can only have the number of distinct tokens as your number of alphabet. This paper was basically if you combine those encoders together, they work. They take the benefit of each side and works better and now your generative models can spell. These days, all the generative models can spell pretty well. I’m not sure if they’re using our technique because it’s very opaque these days, what companies do behind their models, but I hope they are.
Jon Krohn: 00:57:12
Yeah. That is something that has gotten better very rapidly, as you say, alongside getting the number of fingers right on a human hand. We went from asking text generation being completely hopeless and weird to now being most of the time correct. Yeah. Cool that character-aware text encoding… I mean, it kind of sounds obvious. You basically need to have the model attend to characters during the encoding phase in order for them to decode characters effectively downstream. Awesome.
00:57:44
All right. Moving on from your research, you’ve mentioned… This may… Depending on our listeners’ level of experience with machine learning research, they may or may not understand the significance of machine learning conferences as venues for research. The big conferences like NeurIPS, ICLR, ICML, these are some of the big relatively general machine learning conferences where it’s a really big deal to get a paper published, to get a submission accepted to one of these for a talk or for a poster presentation. Then I don’t know how it works for all the conferences, but with some of them like NeurIPS, then a subset gets selected for the proceedings, some of the top papers get selected for the official proceedings of the conference, and this is quite a prestigious thing to be included in that.
00:58:44
You, Rosanne, serve as the DEI chair, the diversity, equity, and inclusion chair, for ICLR as well as NeurIPS, two of the most important conferences in ML period. What brought you into that role? You talked about… Actually going way back to talking about the ML Collective, you said that you start a non-profit like the ML Collective to solve a problem. Maybe these roles, these DEI roles that you chair at ICLR and NeurIPS, are solving a problem. Could you tell us about that?
Rosanne Liu: 00:59:16
Yeah. First of all, I want to say that I think a lot of people are not aware that a lot of research is about serving a community. Of course, from an outsider, research means publishing papers, get your paper published, get your name out there, giving talks, but one of the things that drew me into research is that research is a whole thing. It’s community building. Who’s actually organizing those conferences remained a question to me until much later realized, “Oh, it’s actually just common people like me, like everyone. There’s no official conference organizer there to help you get things started. It’s all just every individual researchers like us serving different roles in a conference and get a conference together because people love meeting up and sharing their work.”
01:00:02
Still to this day, I think ML conferences are the few highlights in the year that people really have fun, get together. There’s so many people, friends, in SF that I don’t see them every day at all, and then I only see them at conferences and we all fly to a different location rather than SF even though we live in the same city. We only see each other at conferences because that’s when you and people have the right mindset and time to share about that research. A lot of getting involved in research is more than publishing papers or running experiments. It’s about getting those things organized. I got into organizing those things sort of because I’m already running a non-profit. I know how to organize things. It’s already the same line of work I’m doing.
01:00:44
Also, reviewing for conferences has been just an early thing I did when I was a student. Once you review a lot, you get promoted into area chairs so you now oversee a bunch of reviewers, help finish their reviews and help make decisions. It’s kind of like a whole ladder that you climb, and at some point you also serve as some kind of chair, you can propose workshops, you can become a workshop organizer, you can become a workshop chair that accepts proposals and select proposals and things like that. I did a bunch of that, and then I think I got my name out as this DEI person, which these days have kind of a bad rap in the Bay Area at least, but because of the ML Collective work and because of I think a talk that I gave that went kind of popular where I talked about how underrepresented people in the community is really suffering, you don’t… I don’t know.
01:01:38
A lot of the hot areas, even in machine learning research, are still dominated by the same kind of people. I’m not even talking about race or gender. It’s the people that have the same backgrounds, they’re going through the same trainings, and when you ask them, they always talk about, “Oh yeah, I started playing games when I was like three years old.” They all have the same life story, and so underrepresented people don’t feel like they are part of the community. I did three years with ICLR and did NeurIPS last year. It’s very hard to define what DEI chair means for conferences, as my co-chair last year, Erin Grant, brilliantly put it. She’s like, “DEI is just anything that’s not set in place right now. Anything that the conference has not covered is a DEI’s work.”
01:02:24
People would write in complaints about, “Why is all the speakers of the same race?” We’ll have to address that, we’ll have to talk to the people who invited the speakers, so we’re just there to solve problems and answer emails, but also more concretely we’re there to help people who has financial problems. If they’re underrepresented from the financial aspect, there’s always fund allocated to people who need help traveling to the conference or presenting at a conference. That’s a big part. We allocate funds to people, which is fun but also a lot of work. And visa problems, although we can’t really help anything other than writing an official letter and telling them that, “They are speaking at this conference. Please, border people, let them in.” We don’t have power over that, but we can help as much as we can. That’s two official things under the DEI chair, but also everything else that’s not right now set in place in the status quo.
Jon Krohn: 01:03:21
Nice. Great. Yeah. It sounds like making a broad impact, even if it is across it seems like a very wide range of different tasks, but obviously your community building experience obviously having a big impact now in these communities and hopefully getting more backgrounds in there than just people who started working on puzzles at age three and getting some different perspectives in there.
Rosanne Liu: 01:03:51
No, or have access to computers really early on. I kept telling people that I didn’t get a computer until I was 21, but if I can do computer science research, anyone can. Don’t get intimidated because all the stories you heard are about people who get access to computers early on. I know you can pick it up. It’s a tool. How hard can it be?
Jon Krohn: 01:04:08
Amazing. Well, I told you that this was going to be a fascinating episode. You had your doubts, Rosanne, but it has ended up being a fascinating episode after all. Yeah. We are now reaching the end. Before I let my guests go, I always ask for a book recommendation. Do you have one for us?
Rosanne Liu: 01:04:26
Interesting. It’s a lot of pressure to recommend a book because I read something a few days ago and I shared with my audience on my mailing list. There’s a quote that says that you only would like a book if you are ready for that book. A book is about a concept that you have to grow to be accepting to that concept, and also it has to be a little bit further alongside your growth path. There’s so many books out there that are amazing, but if you’re not ready for it, it’s probably not the right recommendation for you. But I can say what I’m reading right now. I’m always reading couple books at the same time because there’s some books for learning, there’s some books just for comforting.
01:05:10
I’m reading I Am a Strange Loop by Douglas Hofstadter. It’s a classic. I especially love the preface that he put before that book. He explained why he wrote that book and it was very interesting. At the same time, I’m reading Why Things Fall Apart by Pema Chödrön, I think is the last name. That’s just a purely philosophical, comforting book telling you that things can go wrong in your life, life is long, but it’s fine. Actually everything that goes wrong is a challenge for you to look at, why those things are triggering to you, help you understand yourself better. That kind of book. What else I’m reading? I just finished John Green’s new novel, Turtles All the Way Down. I like him. He’s one of the contemporary authors that are still alive that I like because I like most dead authors. He’s one of the exceptions.
Jon Krohn: 01:06:09
Nice. Well, fantastic recommendations there. The books that I know from there are exceptional. Rosanne, thank you so much for all of the insights across this episode, digging into your AI research, talking about the ML Collective, your work as a conference DEI chair, and now these book recommendations as well. For people who want to continue to hear your thoughts after this episode, what’s the best way to follow you?
Rosanne Liu: 01:06:35
Don’t follow me. I don’t know. I think…
Jon Krohn: 01:06:38
[inaudible 01:06:38].
Rosanne Liu: 01:06:38
Social media is giving people anxiety. Don’t… Well, if you follow me, I’m not going to talk a lot. I try to reduce my social media time, but I have a mailing list that I tell people about talks coming up every week and I try to say little things before every talk information, so if you want something consistent that’s a mailing list that you can join. Search Deep Learning: Classics and Trends email list. I have social media stuff, but I really encourage people to not look at social media. It’s better for your health, especially if you’re a researcher. I think researcher needs time to think. You can’t just bombard yourself with information every day.
Jon Krohn: 01:07:23
It’s true.
Rosanne Liu: 01:07:23
If you can’t think, what’s the point of becoming a researcher?
Jon Krohn: 01:07:25
It’s true. Yeah, so that email list, that’s the same as your reading group, the Deep Learning: Classics and Trends. It’s all one and the same. That email list, reading group, same thing. Yeah. We will be sure to include that in the show notes. Rosanne, thanks again for coming on the show and maybe we can catch up with you again in a few years and see how LLM research is coming along. Maybe we’ll just interview the LLM version of you.
Rosanne Liu: 01:07:55
Yeah. Yeah. The better version. Thanks for having me, Jon. It’s a really pleasure.
Jon Krohn: 01:07:57
What a fun, enriching conversation with Dr. Rosanne Liu. In it, Rosanne filled us in on how she co-founded the ML Collective to provide support to people who need ML research support regardless of their background, resources or where they are in the world, how the Intrinsic Dimension, a metric she devised of how hard a task is for a given model, inspired the LoRA technique for efficiently training LLMs, how LLM scaling alone may solve all goal problems but we can always use our curiosity to understand LLMs even more, and how diversity in the ML community extends to the way people were raised, such as how early or late they had access to computers. As always, you can get all those show notes, including the transcript for this episode, the video recording, any materials mentioned on the show, the URLs for Rosanne’s social media profiles as well as my own at www.superdatascience.com/797.
01:08:53
Thanks, of course, everyone on this Super Data Science team, our podcast manager Ivana Zibert, media editor Mario Pombo, operations manager Natalie Ziajski, researcher Serg Masís, writers Dr. Zara Karschay and Sylvia Ogweng, and our founder Kirill Eremenko. Thanks to all of them for producing another fascinating episode for us today. For enabling that super team to create this free podcast for you, we’re deeply grateful to our sponsors. You can support this show by checking out our sponsors’ links, which are in the show notes end. If you yourself are interested in sponsoring an episode, you can get the details on how you can do that by heading to jonkrohn.com/podcast.
01:09:29
Otherwise, please share this episode with someone who you think might like it, review it in your favorite podcasting app, subscribe of course if you’re not already subscriber, but most importantly I just hope you’ll keep on tuning in. I’m so grateful to have you listening and hope I can continue to make episodes you love for years and years to come. Till next time, keep on rocking it out there. I’m looking forward to enjoying another round of Super Data Science Podcast with you very soon.
