Jon: 00:02
This is episode number 808, our In Case You Missed It in July episode.
00:19
Welcome back to the Super Data Science Podcast. I’m your host, Jon Krohn. This is an In Case You Missed It episode that highlights the best parts of conversations we had on the show last month. My first clip is from episode 803 with Daliana Liu. Daliana, who has amassed hundreds of thousands of followers on social media thanks to her sage guidance, coaches technical people on finding a career path that will withstand rapid industry changes. I wanted to know the technical, the “hard” skills, that Daliana advises her data-scientist clients to learn.
00:40
In this clip, I got him to speak from his experience as CEO of the climate technology startup he founded, Windscape AI. This is a great case study if you’re planning to launch your own AI models commercially.
00:51
In the coaching that you do, you have a tremendous amount of experience as a data scientist. And so I’m curious where you see the field going. So we’ve talked a lot in this episode about soft skills. Yes, very important. But I’m just curious, in addition to the soft skills, when you’re doing career coaching, are there technical skills that you highly recommend people have in data science? Maybe this is related to what’s coming, you know, with generative AI being here, for example, with agentive AI coming and becoming more and more important, are there particular skills, technical skills that you recommend to our listeners?
Daliana: 01:29
Yeah. I think again, previously we mentioned data science roles can be very different in different companies. So it depends on what you do. For example, my friends are building forecasting models, so their work is not highly impacted by generative AI or large language models, although they do use some type of forecasting. You can use transformers. And there are other types of machine learning engineers. Maybe they work for a small, medium-sized company. And maybe they want to take advantage of some LLMs, identify some low hanging fruit. Maybe you can build some chatbot, or question answering tools for if you are a 2C business.
02:29
So I think again, the most important thing is to identify the business use case, but you have to understand what are the technology? What’s the possibility out there to identify the use case. So if you are in a more generalist role, and then you feel your company has a lot of documents, you can improve either the business use case. Or I think a lot of times for data scientists it’s to improve the internal process. For example, maybe your stakeholder cut a ticket. Maybe you can create some GPT bot to tag, put it into a certain category. I think right now, a lot of AI/LM tools are more like an intern you have. It can help you solve a lot of problems, create some project, but it’s not good enough to rely on it as an advisor. So you’re still the person to make decisions.
03:31
And also, my friends from hiring agencies tell me, so actually people are not hiring the title of AI engineer, because it’s very specific. At end of the day, AI skills is just going to be part of engineering or part of machine learning engineers, but they still want people to have those skills. So when they’re hiring software engineers or machine learning engineers, if you have experience leveraging, say OpenAI API to build an app, knows when to fine tune or use a RAG, that’s also important. Because a lot of times now when companies are hiring people, again, this role is new, this skillset is new. They don’t exactly know what skillset they’re hiring into. So you are an advisor. You need to tell your hiring manager what you think they should do with their data, with their business use case.
04:36
So be familiar with OpenAI or Anthropic, those APIs, fine-tuning. But based on my conversations with practitioners, the RAG use cases of chatbot are more common than fine-tuning, because not every company has their own data, or their data quality may not be good enough. I think sometimes RAG is good enough. And also, when you think about building those AI solutions, it’s connected to a database. So get familiar with some vector database for example, on Pinecone, and those things can be useful. And if you are interviewing for large tech companies, I think for data scientists, machine learning roles requirement for software engineer is higher and higher. So previously, I think people asked questions about data manipulation in Pandas, or maybe like a simple lead code.
05:41
Now I heard at data points, sometimes people will ask Medium, or even hard lead code questions. Which I think makes sense, again because models are becoming better and better. A lot of times, companies are thinking about scaling the solution, reducing the latency. So you need to really be proficient in your software engineering skills. And for product data science, I don’t think it has changed that much. Product data scientists or data analysts, BI type of roles have less requirements for software engineering skills. Some might still ask you lead code, simple questions, but they want to make sure you know SQL, and you can do similar type of data manipulation in pandas. I do have some data points. People tell me the questions about experimentation, AB testing is getting a little bit more difficult. Sometimes it could be causal inference. I think it depends on the company your interviewing.
06:56
Again, if right now your work is pretty demanding, it doesn’t really get into the AI or LM side. Don’t feel anxious to focus on the tools that will make your current work easy, but in your spare time there are so many things to learn. To learn things, it doesn’t mean you have to sign up for a big course. You can read blog posts. Just be aware of the possibility. So whenever you feel this specific tool or technology can be useful in your current workflow. And then you can learn at that time, it’s not going to be late.
Jon: 07:36
Staying knowledgeable about the new and emerging technologies makes so much sense to me. In an industry like tech and AI, it’s so easy to drop the ball because people are developing new products and services the world over. Daliana’s advice is the same as mine: Keep on reading and learning. Thankfully, as a listener to this very podcast, you’re already staying on top of the ball.
07:58
From hard to so-called “soft” skills, my next clip is from episode 805 with the Pulitzer Prize winning journalist and many-time New York Times bestselling author Charles Duhigg. Charles offered some constructive advice about the art of making great conversation and how speaking to clients and team members about their lives outside work helps warm them up before a meeting.
08:21
Nice. All right. So, then once we’ve established that, I guess we’re in one of those three categories. And then, maybe there are one or two top tips you have for each of the categories?
Charles: 08:33
Yeah. Once we know what kind of conversation we’re having, and we lean into it a little bit, so what do we do next? What we do next is, oftentimes that’s what sets us up for a good conversation, but then what happens next that’s critical is that we have to prove to each other that we’re listening. And sometimes we prove that we’re listening just by asking a follow-up question. Someone says like, “Oh, I went to my kid’s graduation.” “Oh, what did that feel like?” “Oh, it felt amazing. I thought about my parents.” “Oh, what did your parents tell you before you went to college?” Follow-up questions show that I’m listening.
09:12
But there are other times that we are discussing things that are more difficult to discuss, and these are known within psychology as conflict conversations. A conflict conversation is like when we disagree with each other and we’re discussing our disagreement, when we just come from different perspectives, when we’re talking about something that’s hard, and even if we agree with each other, it’s just a difficult thing to discuss. If there’s any tension in the conversation, it usually becomes a conflict conversation.
09:38
One of the things that happens is that we have an automatic instinct in the back of our head in a conflict conversation to suspect that the other person is not actually listening to us, but is merely waiting their turn to speak. We’ve all been in that situation. And even if they ask a question, they’re like, “Where did you go on vacation?” And you tell them and you realize within 15 seconds, they don’t care where you went on vacation, they just want to tell you about their vacation, the yacht that they rented. We have to somehow overcome that suspicion.
Jon: 10:11
A quick question for you. That conflict conversation, that isn’t a fourth category, is it? Any of these three categories could be…
Charles: 10:19
Anything can become a conflict conversation. If we’re disagreeing about where we ought to go on vacation next month, it’s a conflict conversation that is rooted in practicalities. If I’m telling you how I feel and you’re feeling defensive, that’s a conflict conversation that’s rooted in emotion. A conflict conversation is basically just defined by whether there is tension. This happens often. If you go to a party and you start a conversation and you’re not certain how to end it, you’re feeling socially awkward, that’s a conflict conversation, because it’s just creating tension within you or the other person or both of you. The question is, how do we overcome that conflict? How do we overcome that suspicion that the other person isn’t really listening? There’s actually a technique for this, which is known as looping for understanding, and has three steps. Step one is: You should ask a question, preferably a deep question.
11:14
Step two is: After the person has answered that question, repeat back in your own words what you heard them say. What’s important about this is not to mimic them, but to match them. This gets back to what we were saying before, that if I mimic you, if I just repeat back in a rote way what you just said, you’re not going to feel like I’ve listened to you. But if I try and say in my own words what you just said, and maybe even I proved to you that I’ve been processing it… “What I hear you say is that you really hate hot dogs. And it sounds to me like it’s not just hot dogs. It’s all processed meat, is that right?”
11:52
And then the third step, and this is the step we usually forget is: Ask if you got it right. Because one of two things will happen. The first thing is, is that they might say, “No, I don’t think you completely understood me.” Which is useful to know, right? But the second thing, the more likely thing that will happen is, they say, “Yeah. Yeah, I think you heard what I was trying to say. What just happened in that moment is that I asked you for permission to acknowledge that I was listening to you, and you gave me permission to acknowledge that, and as a result, you become more likely to listen to me in return. That’s how we prove that we’re listening in a conflict conversation.
Jon: 12:53
Nice. So I am guessing that we aren’t in a conflict conversation, but I’m going to go through this loop anyway to [inaudible 00:34:38], because that is something that I… You might’ve already picked up. It is something that I try to do on the show anyway, which I think is helpful to our listeners. Especially, you’re an audio-only format, a lot of our listeners, you’re driving in the car, and so a little bit of repetition, me saying things in my own words, especially if I can do that, it might be helpful to understand things, so I try to do that anyway. But, let’s also do it for this loop to avoid a conflict conversation. So, if we feel the conversation is a conflict conversation… Which also, I think we should probably make clear that conflict conversations are not bad inherently.
Charles: 13:17
Right.
Jon: 13:18
That’s going to happen between…
Charles: 13:20
Conflict conversations are actually good. Usually, the most important conversations we have are conversations with a little bit of tension in them. I want to discuss something important with my wife. I want to talk about something that I’m actually scared to talk about. Conflict conversations are really, really good.
Jon: 13:37
Yeah, yeah, yeah. For sure. But to keep those conflict conversations from going off the rails, which they can, you can have this crescendo of emotions, and people get defensive, people start to… I don’t know if this is something about our amygdala being activated or something, but it seems from my anecdotal experience that once people start to get emotional in the experience, they don’t listen as well, in terms of when emotions really go off the rails.
14:18
Realizing that you’re entering into a conflict conversation, which could be important, which could be productive, which doesn’t need to go off the rails, to keep things well contained, we go through this three-step process. I’m confident I wrote down the second and third correctly, the second and third step. The second was to prove that you’re matching in conversation, not just mimicking, by repeating back in your own words. And the third step was to ask if you got it right. Was the first part to ask deep questions? Was that-
Charles: 14:49
Yeah, ask a question, preferably a deep question. Because what you really want to do is, you want to get the other person talking, so that then you can repeat back what you’re hearing and prove to them that you’re listening. That doesn’t mean this is the end of the conversation. A part of a conversation is not just me listening to you, but also me saying things and you listening to me. But it’s very easy to ensure that we’ll actually listen to each other by proving that I’m listening to you.
15:18
You can use this in situations where it’s not totally obvious, like this party thing that I mentioned. You’re at a party, you don’t know how to end the conversation. One of the things that is the go-to technique is, you say to the person that you’re talking to, “I need to go refresh my drink.”, or, “I want to let you play host.” But before I let you go, let me ask you one last thing…” And you ask some question and they will answer it in 15 seconds, right?
15:49
They will not belabor, because they know what you’re doing. You’re basically gracefully saying, I want to give us an opportunity to end this conversation, but I’m so fascinated by you that I want to pay homage to that fascination and show you that I like you. That’s a form of looping for understanding, where I’m asking that question, and I’m putting the “I hear what you are saying” before the question. You can use this in settings where it’s less obvious that you can use them to basically resolve something very gracefully.
Jon: 16:24
Hmm, hopefully I’ve been honing a lot of these ideas while interviewing guests on this show! Active listening is so important and yet so undervalued despite how powerful it is. Moving on to episode 801, I ask Charles Goddard — Chief of Frontier Research at the super-fast-growing AI startup Arcee — I asked him about model merging, a powerful new AI technique that is definitively at the frontier. Model merging involves combining multiple LLMs’ capabilities without increasing the number of model parameters, drastically reducing compute costs and inference time… as Charles explains.
17:00
How model merging represents the next frontier in AI, as well as in transfer learning, and what potential you see in this technology for transforming various industries. So I guess that’s a two-parter. So give us some insight into what model merging is. Explain it because you could do a better job than I can. You’re the expert on that. And then explain to us the potential you see in this model merging, and then I’ve got lots of follow-up questions related to that.
Charles: 17:25
Absolutely. So model merging, it’s a family of techniques that let you take the pre-trained weights of neural networks and combine them into a single network that captures some of the strengths or abilities of all the networks that you threw into that melting pot. And of course, there’s a big variation in the techniques of what performance characteristics they have and what scenarios they excel in, but essentially it’s a natural advancement of the philosophy of transfer learning. So we discovered that if instead of training language models from scratch every time on however many trillion tokens and then throwing in our specific task, if we just do the foundation model training once and then fine-tune many different versions of that on our various different tasks, then we get better performance and it’s infinitely cheaper than training from scratch every time.
18:18
Model merging is very similar in that it’s an even further finer subdivision of the work involved. So whereas with typical language model training setups today, you’ll take some foundation model like Llama 3, a 8 billion parameter, the untrained based model, and then you’ll curate some extensive data set that reflects what you want it to be capable of doing, which includes just general instruction following capabilities. You need a very diverse data set to get a robust instruction following capability in a text model.
18:53
So you get all of that data and curating all of that data is quite difficult. The open-source community has some good ones, but it’s still not to the level of what companies, like Meta or OpenAI for example, have curated. Then on top of that general stuff, you also need to curate a specific data set for the task that you actually care about. So let’s say you’re a finance company and you want a language model to summarize financial statements, you need to curate that data set and then you train the model on top of that entire aggregate pile. And the end result is fantastic, but there’s really no reason to be doing 90% of that work because we have incredible instruction following models available, open weight, just on the internet. If you create just the data set for your specific tasks, then model merging is a tool that lets you take those pre-existing artifacts and incorporate their strengths, essentially for free.
Jon: 19:42
Yeah, so let me try to explain in my own words and you can tell me where I get this wrong. When you have, out there in the world, there’s lots of different kinds of specialized models that are great at different tasks. And right now, if I was without model merging, if I was going to try to make use of that, I might have some kind of separate model that is predicting where to triage requests to and then I’d have to grab… So let’s say there’s five different tasks that I want my LLM to be able to perform really well at. And so I train, probably myself, this model for triaging natural language requests that come in. And I then, based on that I get some probability that okay, this is a great situation for model two out of five, and then I separately call just that one model and send the original natural language request to that, allow to follow through.
20:45
That is going to require me to have a whole bunch of different models in production, which could be all very large, requiring a bunch of GPUs maybe to be in production running, at great expense to me. With model merge, I can have one model running, that is probably smaller than my five in aggregate, and so that means I can reduce my compute costs and probably also deliver results more rapidly in real time to my users.
Charles: 21:16
Absolutely, yeah. So that’s one very important use case of model merging and it is good to highlight, as you did, the size of the models involved. So for most of these model merging techniques, you put in some number of models that are all the same size and you get out a model that’s that same size, just one model. So to give concrete example there because that wording was not great. Say that you’ve trained seven or eight models based on Llama 3, 8 billion parameters, and you put all of those into a merge, t he model that you get out is still just 8 billion parameters. So versus a classic [inaudible 00:09:19], for example, where you’d inference each of these seven models and then combine the results at the end. You’re just inferencing a single model. [inaudible 00:09:29] the same size as each of these individual ones.
Jon: 22:02
Yeah, and so let me make sure that I’m understanding this. So you could actually, so if I was starting with five, 7 billion parameter models, a common size that we see these days. So let’s say I have five different 7 billion parameter models. With model merge, when I merge them, I end up with one 7 billion parameter model.
Charles: 22:19
Exactly, yes.
Jon: 22:20
That is pretty wild. Next, if you’re publishing an academic paper with 441 other people, how do you get your own voice heard? My guest in episode 797, the prolific Google DeepMind researcher Dr. Rosanne Liu, was part of a team that wrote the widely referenced paper, “Beyond the Imitation Game,” and I wanted to know how to get the paper’s 442 authors to agree with each other.
22:44
Another paper of yours is called Beyond the Imitation Game. This was well received, it’s had over 800 citations, but something that’s really interesting about this paper is it has 442 authors across 132 institutions. Tell us about the paper first, the key findings of the Beyond the Imitation Game paper, but also tell us about what it’s like working with such a crazy big team.
Rosanne: 23:13
Yeah. The paper we’re talk about is BIG-bench. It’s hard to say it’s my paper. Of course my name is on it. So are 400 other authors. I’m one of the core organizers of the BIG-bench, but it’s mainly started by a team here at Google before I joined Google. They had this idea that, “We should evaluate large language models better. How do we evaluate them?” We as researchers can each sit down and write a task, say, “Okay, let’s, say, come up with this maths problem. If the model can solve this problem, I’ll consider it capable.” Each of us can write down a task and if we have, I don’t know, 40 researchers we can write 40 tasks, but that doesn’t really scale and doesn’t really cover all the things we want to test about models.
23:58
This is one of the first I think open-sourced benchmark. We’re really asking the society, “What do you think you should test the model with?” I mean, if you have proposed that people can submit tasks via pull requests and we merge them, so there’s a core team in Google where that’s in charge of merging them, evaluating them, running the models against them, and generating plots and the venture paper come out of that. It’s a huge effort that involves a lot of organizing, of course a lot of community members submitting their tasks, and us on our end evaluating models on the tasks, structuring the tasks in a way so that they’re all very standardized and such. I think later there are research like this more and more that are crowdsourced. We’ve seen that, but I think this is one of the first that show that this is possible. It’s possible to organize over 100 institutions and 400 authors.
Jon: 24:56
Yeah, it’s a super cool paper and BIG-bench is one of those papers that you do… Or one of those evaluations that you do hear about a lot for LLMs, so it has certainly been impactful.
Rosanne: 25:08
Mm-hmm. Yeah. I originally read that one downside of BIG-bench is that it’s not that easy to use. I think I wrote… Sorry, I read [inaudible 00:37:46] blog about that and I completely agree. Nowadays, there are so many more evaluations you can run and each of them has its own pros and cons, and eval is an interesting field. It’s almost like the moment you publish an eval, it’s outdated because now you know that models will be looking at that and learning how to do that. You have to figure out the next eval. It’s an ever-changing field. I have opinions on whether… They’re private evals as well, so evals that you don’t tell people what they are, you just tell them your scoring of each kind of models. That prevents, of course, model learning from your existing data, but I think I also have problems with that. I think closed doors eval are also problematic. As long as you’re completely neutral, you can do that, but most evaluation parties out there are not completely neutral.
Jon: 26:09
Right. Exactly. If Google invests a lot of money or another big tech company invests a lot of money in a benchmark, it’s unlikely to be completely neutral.
Rosanne: 26:21
Yeah. Yeah, or if the founders are buddies, which happens a lot in the Bay Area. Small circle.
Jon: 26:28
That’s right. Yeah, so trickiness around evaluating LLMs and really understanding… I mean, kind of a longstanding joke in this space is that every new model released is the state of the art across every benchmark.
Rosanne: 26:45
Right. Yeah. For two minutes before the next one. Yeah.
Jon: 26:47
Yeah, exactly. Yeah. You get this kind of inflation of evaluation metrics. Yeah. It’s a tricky thing that, like you mentioned there, there’s advantages to having things being open because that provides more transparency, but then of course there then you’re opening up those data to being used as training data in the model so of course they perform well on that evaluation because, again, then it’s imitating. It’s the imitation game again as opposed to evaluating something deeper. Yeah. Trickiness abounds. Do you have any insights into how you would, I don’t know, tackle something here in eval?
Rosanne: 27:29
Well, there’s the old Kaggle model where you can fit your model on the training set, but the test set is [inaudible 00:40:05]. I’ll give you the score on test set, but people still overfit. You can just keep submitting until you see your test score come up, that kind of thing. Yeah, so we have to prevent that infrastructure by limiting their numbers of submissions or something. But yeah, no, I think this just gives everyone more things to do. Everyone has a job. We just have to keep inventing new evals. There are people pushing new models. You can be the people pushing new evals.
Jon: 28:05
Which brings me to my last clip from July. In it, we go to episode 799 with Dr. Andrey Kurenkov, who’s an AI Engineering Lead at the super-cool text-to-video-game Generative AI company Astrocade and also host of the extremely popular “Last Week in AI” podcast. I had a great time talking to Andrey about how AI can help support us in our mutual journeys to find great podcast content. We rounded up the capabilities of LLMs like Gemini, Claude and ChatGPT, which have just exploded in such a short time. We also talked about all the various media we can now plug in to LLMs to take our dialogue with the GenAI tools we use so much further.
28:45
Interesting times for sure. I’m sure that’s part of what compels people like you and me to be creating shows that air once or twice a week with updates on this because it moves so rapidly. Actually, it’s interesting even in the four years that I’ve been hosting this podcast when I started in… My episode started airing in January 2021, and at that time, some weeks it didn’t seem like I had… At that time, Tuesday episodes were always with a guest and they were long and Friday episodes were always quote unquote Five-Minute Fridays. They’re pretty much always longer than five minutes, but the idea was to keep them really short. It was just me and my goal with those Five-Minute Fridays was to always have some important piece of news covered, some important paper.
29:34
And in 2021, a lot of weeks I would end up talking about things like habit-tracking spreadsheets of things outside of AI that I found interesting and helpful and I thought listeners might enjoy these kind of productivity tips. I even started doing this series of habits. It was a numbered series that I can’t remember where I got to. I maybe got to number eight or number nine of these are I think the most important productivity habits that you can have, and it was intended to go on for dozens of episodes, but then I think ChatGPT came out, and then since then, it’s just every week I have an endless list effectively of extremely interesting, directly relevant AI stories that it never occurs to me anymore to be like, “We should do the habits episode this week.”
Andrey: 30:29
Yeah, it’s interesting looking back, we’ve been doing the show for almost three and a half, almost four and a half years. We started March 2020 and I remember as a PhD student, as someone working on AI, it already felt like things are moving super fast at the time, at least within research, but now I kind of miss those days. It felt really rapid then, but now it’s moved out of academia and out of this niche circle where we are making a lot of fast advancements on beating benchmarks, on utilizing neural nets to do things you could not do before and now all those things are coming into the real world. Yeah, exactly. It was like it used to be that we didn’t have 30, 40 stories per week that are actually meaningful and interesting and now it is definitely the case.
Jon: 31:39
Yeah, it’s wild. Something that I actually talk about in public talks that I give a fair bit is I open with how this is likely only going to get faster and faster.
Andrey: 31:54
Yeah. I’ve reflected on this recently and I don’t know why it took me… I used to think that even going back two years, one year, my thinking was it’s hard to say whether we will have human level AI anytime soon. It seemed possible, but it didn’t seem definitely likely, which some people did already believe in, and I had particular technical reasons of scaling, of limits of length and memory, these different things. Now, a year later, a lot of those challenges we’ve made impressive progress on. I remember initially ChatGPT had a context limit of 4,000 tokens. Now you have context limits of 250,000, one million.
Jon: 32:49
In Gemini, you have a million. That is also, that’s another great use case for people. Real-time search in Gemini or also if you’re uploading a huge media file, you can upload into the context window of Gemini today 10 or 11 hours of audio, an hour of video, the equivalent of 10 novels, and they’re testing a 10 million token context window version, which would 10X all of those numbers I just gave, so it would be 100 hours of audio, 10 hours of video or 100 novels worth of context.
Andrey: 33:26
Yeah. It’s just I guess since ChatGPT, there was a lot of money going into AI to some extent already, but now it’s impressive to see what humanity is capable of if we take some of the brightest, most ambitious, most hardworking generally, let’s say, I don’t want to say the top or the most talented, but you have a lot of people pushing to progress to make progress and to solve problems, and we are doing that at a faster and faster pace. We got Claude 3.5, Gemini 1.5 Pro that are already better than the larger and more complex systems, and we’re still solving some challenges like making these passive models. We give it one input and you get one output.
34:33
A lot of people working on the agent aspect of it where you give it one input and it can operate independently for some period of time and go off and do things, which I think would be required for human level intelligence to really… You can’t have it as a just passive model. You also need multimodality with vision and with movement ideally and audio, and that’s another thing that we’ve made a rapid progress of. What I want to say with all this is after just a year or two when I used to think, I don’t know exactly when AGI happens, it could be soon, it could be decades, now I feel like with all the money, the people and all the talent, I could easily see it happening in two or three years and that’s crazy, human level AI in just a couple of years, but that’s the world we live in.
Jon: 35:31
I’m in exactly the same boat as you on exactly those timelines. The watershed moment for me was GPT-4 where yeah, GPT-3.5, the ChatGPT Experience, I was like, “This is very impressive. This is very impressive. No question.” I loved it. When GPT-4 came out, I was like I flipped from exactly what you just said, is something that is exactly the way I felt and I’ve said on air a number of times before on different shows including this one where I went from exactly as you said, thinking if it’s possible at all to have human level intelligence across all human tasks, it’ll probably take decades if it’s possible at all. I don’t know if I’ll live to see it. And now it is exactly the same. It could be a few years.
Andrey: 36:21
Yeah. And I guess part of why that is, part of why you believe it is what is happening now is everyone is spending billions of dollars, right? OpenAI is spending billions of dollars on just training. Well, they’re not spending billions of dollars yet. They spent millions to train GPT-4 that we know, maybe 10 million, but Meta, it’s certainly in the order of tens of millions and it seems like people expect GPT-5 or something like that to take something like 100 million or 1 billion or whatever, but they’re doing it. They’re getting the money, they’re getting the capital.
37:07
You have giant companies like Meta and Google that have the money anyway just throwing money and infrastructure and compute at it. And so that’s another aspect of it where if a big part of humanity comes together independently to make progress in a direction. It’s something that I’ve realized that there were these technical challenges, but so many people are working on different aspects of a problem and there’s so much capital and other things made to progress fast that this is what humanity does apparently. We just do amazing sort of miraculous things as we’ve done with computers and smartphones and everything.
Jon: 37:59
All right, that’s it for today’s ICYMI episode. To be sure not to miss any of our exciting upcoming episodes, be sure to subscribe to this podcast if you haven’t already but most importantly, I hope you’ll just keep on listening. Until next time, keep on rockin’ it out there and I’m looking forward to enjoying another round of the SuperDataScience podcast with you very soon.
