Jon Krohn: 00:00 This is episode number 960, our In Case You Missed It in December episode. Welcome back to the SuperDataScience Podcast. I’m your host, Jon Krohn. This is an ICYMI an episode that highlights the best parts of conversations we had on the show over the past month. Let’s start off with something from a widely reported study from last year from MIT’s Nanda Lab, which is somewhat contentious this figure, but the headline figure was that they reported a 95% failure rate for enterprise AI solutions. In episode number 953, I talked to Dell’s global CTO and global chief AI officer, John Roese, about how businesses can make sure they’re in that rare 5%. For people who are unfamiliar with that MIT study, the 95% of GenAI projects fail. I’ve got episode number 924 for you listeners dedicated to that. But even more important than that reference to a back episode is John Roese’s previous episode on the show, which is 887.
01:05 And in 887, he provides tons of information on what he just gave an overview of where you can be basically guaranteed to have successful projects within your enterprise, how to select the projects that are going to have the best ROI. And you just gave some citations there, which I don’t think you did in the previous episode of kind of 10 to one, 30 to one ROI ratios, which is amazing. And as soon as you have a few of these projects, then you start this AI ROI flywheel spinning, which is a term that I learned from you and I have used in countless business meetings since we did that episode 887 together.
John Roese: 01:46 It seemed to work. By the way, it’s been a while, but look, the big impact that we had at the end of that process once we got the four big areas running is in our fiscal results last fiscal year, which we reported in March, that was the first year that we’d kind of got all this stuff up and running. Something happened at Dell that hadn’t happened in 41 years as a company. Our revenue grew at $10 billion and our absolute cost declined by 4%. We have never decoupled revenue growth from cost. Every other time the revenue grew, the cost grew with it. It didn’t happen last year. And it was a combination of people, process and technology, but it was really a positioning and redesigning the company to actually get better productivity as we did work. And if you do that, boy, it’s a pretty powerful tool when your revenue grows and your costs don’t grow with it.
02:29 That results in margin expansion, that results in just a better performing company. And so we now have a lot of empirical data. Those ROI metrics can measure. We measure everything. And so I feel good that, hey, not only was it good theory and not only was it discipline, but it actually seems to work in terms of producing really strong ROI, which, hey, that flywheel works better if the ratios are higher.
Jon Krohn: 02:51 For sure. 100%. You get a lot more attention when you have a 30 to one ROI ratio for sure. And it’s great to hear those kinds of stats around revenue increasing without costs going up or costs even decreasing because I’ll keep this completely … This isn’t a comment on anything related to Google necessarily, but a friend of mine, a good friend of mine, just the other day at the time of us recording, said to me, “Do you really think you’re getting better productivity from this AI stuff?” And yeah, it’s fantastic for you to be able to cite these hard numbers on how if you do it right, you absolutely can be getting a great return on investment in your organization.
John Roese: 03:30 Yeah. I remember one of the first principles was have a hard ROI metric. It’s hard to measure goodwill and happiness. It’s a lot easier to measure things like, I don’t know, cost or margin or revenue or things that you actually can measure and understand. And so by focusing your AI efforts on things that are empirically measurable, you can actually get some level of confidence that one, you’re either achieving success or two, you’re not. If the ROI is nothing, and there’s actually a term that I won’t use the full language, but I think it was a McKinsey report and it described what they call BS work. And they basically said generative AI is a BS work detector because it turns out that if you use a whole bunch of AI tools on a piece of work that you currently do without them, and after you use them, even if you feel good about it, nothing changed.
04:22 Your revenue didn’t go up, your cost structure didn’t go down, nothing really changed. What it’s telling you is maybe that work isn’t worth doing. Maybe it’s BS work. And it turns out that there’s probably a lot of that out there where just doing AI to something doesn’t necessarily result in an impact. And I’m pretty, let’s call it focused on, you want an impact from this stuff. You’re investing heavily in the technology. It’s costly, it’s complex. And if you’re not getting an impact that disrupts your business in some way in a positive sense, then why are you doing it? And trust me, there’s a lot of targets to go after. So spend the time there.
Jon Krohn: 05:00 Right. So what you’re saying is in a lot of cases, the reason why AI projects are unfruitful or don’t appear to move the needle internally is because it’s automating something that was BS in the first place when humans were doing
John Roese: 05:13 It. Yeah, exactly. And like I said, I think it was McKinsey, one of the consulting firms put this article out. You go Google, BS work and AI. And it makes a lot of sense. I mean, look, hey, you can apply AI to anything you want. Great. That might make it better, might not, but you should probably pick the things that if you make them better, your business will improve. Something material will happen. And that level of discipline is something that people have to spend a little time on upfront. And if you do, yeah, seems to result in things that matter.
Jon Krohn: 05:42 To quote John, “As AI professionals, we should always be preparing to make something material happen.” My next guest put that very idea into action. In this clip from episode number 945, the software engineer and podcast host, Joel Beasley, talks about using GenAI as a writing partner, coach, and analyst to help him get his feet wet in the standup comedy scene. And so tell us about the AI tools that you use to support your comedy development.
Joel Beasley: 06:10 Yeah. So I started in December of this past year, so it’s been about 10, 11 months or so. And I realized really quickly, I was like, “What’s the difference between good comedians and great comedians? And why are some people more popular than others?” And comedy is very subjective. What’s funny to me and my group may not be funny to you and your group. So I was looking for the common data point that would transcend groups. And I found that that common data point was laughs per minute. And so you can look at these comedians, the Nate Bargetskis, the John Chris, you can take their, you can do it old school and just watch their special and jot down the laps per minute, or you can run it through Gemini and have it calculate the laughs per minute. And so I found that four to five laps per minute is the standard for a professional full-time comedian.
07:02 And so I said, “Okay, all I have to do is write material that elicits four to five laughs per minute, and I will be on par with a professional comedian.” And I’ve pretty much done that. So what I do is I record every set. I run it through the AI after the set, nothing special, just Gemini prompt. Why Gemini? It was the only one through the browser UI that would consume video. GPT and Groc as of the recording date today will not consume video and it was able to do it. And so I said, “Okay, that’s super easy.” And I just started, I have a spreadsheet. John, I have a spreadsheet like a nerd does. And it lists every performance with a recording of the performance, the laughs per minute, what work didn’t work, notes and how many people were in the audience, how much money was made, all of that stuff.
Jon Krohn: 07:53 I love that. I can’t wait to see if it gets more sophisticated over time. As your comedy career goes on, you get more and more sophisticated metrics. So right now, it’s kind of like laugh velocity. I wonder if we can take the derivative of that and get laugh acceleration.
Joel Beasley: 08:08 Yes. Well, also all laughs are not equal, which is actually a harder thing. Some people build software for this, by the way, called like Laugh Evaluator Pro. But the duration of the laugh is important. The intensity of the laugh in relation to other laughs are important. So it can get pretty nerdy. I’m trying to keep it basic. So use AI and the evaluation of it, but I also use it in the writing process. So for example, what I like to do is I write the joke, like I write it and practice it and I’m like, “That’s a good joke.” And then I put it through AI to ask for different perspectives, how they would see that joke, different potential areas, like directions that joke could go in, maybe words near this joke that have a double meaning, things like that. And I’ve got a saved prompt that I use to act like a writing coach.
09:01 Now, the way I figured this out, John, was I was going to a writer’s group, a comedy writer’s group where there would be like eight comedians in a circle and we’d do like a minute of our jokes for that week and then they would all give feedback. And so I went to like 10 of those, modeled how those professional comedians gave each other feedback, turned that into a prompt, and now I just get to do it with the crock.
Jon Krohn: 09:23 From counting laughs per minute, we turned to the more serious subject of burnout. In episode number 952, I asked the fit data scientist, Penelope LaFeuille, how she used AI to help her recover from work fatigue and put her on the path to a much more rewarding career. Tell us about how you were a burnt out scientist and how you turned things around.
Penelope LaFeui…: 09:46 Yes. I was working in finance in New York right after my master’s degree. And even if I was good at what I was doing, I think that the fact that I was burnt out came from two different aspects. The first one, I was working too much. And the second one, what I was doing was not really aligned with the direction I wanted my career to grow. And so when I was waking up every morning thinking about how do I see myself in five years, there was a huge disconnect between what I was doing and what I wanted to be doing in five years. And that’s when I decided to change careers and to switch from finance. I was still doing data science for finance, but doing data science more in the life science industry because this is the industry I’m genuinely passionate about, just like studying the human body and using this technical knowledge in order to create new drugs, develop clinical trials and so on.
10:46 So that was the first point. And then the second point was about, I’m not going to be successful at being a data scientist working in the life science industry if I am not taking care of my body myself. So that’s when I hired a lifestyle and fitness coach. And when I started working out pretty regularly taking care of my nutrition as well as my recovery with like sleeping, recovery practices and so on. And that’s how I was able to very clearly do a 180 in both my career and my personal life at the same time.
Jon Krohn: 11:23 Fantastic. Were you into fitness before getting this fitness and lifestyle coach or was that something new?
Penelope LaFeui…: 11:30 So I was, but I was not doing the right things, meaning that I was also burnt out in my fitness because outside of my job, I wanted to have something that I was looking forward to. So I was working out nearly every single day and I was very clearly overtrained. I was not fueling my body properly. So everything was completely out of work, to be honest.
Jon Krohn: 11:54 Yeah. This probably happens to a lot of the kinds of people that listen to this podcast who are not only pursuing a career in data science or AI in some way, you’re also listening to a podcast about it in your free time probably. And so you’re probably the kind of person that is trying to like maximize everything all the time. And so I certainly, with my workouts, I very, very often tend to overdo it, training too many days in the week, too hard on a given day, not taking enough rest days and just kind of having the level of intensity too high on a regular basis. And so kind of instead of fitness being something that rejuvenates me, it is often something that just wears me out even more. And then I’m kind of like, on the worst days, it’s like I keep feeling like I just need to nap.
12:47 And it’s like just sitting at my desk feels like it’s too hard. Yeah. So how do you strike that balance? How do you design a program that is going to make you feel rejuvenated? How do you structure that?
Penelope LaFeui…: 13:03 So working with my coach, we’re looking at what’s my work schedule and also what are the activities that I’m doing outside of the gym that also give me energy and that I don’t want to sacrifice. So for me, for instance, I play a lot of pickleballs, of pickleball going to play in tournaments and so on. So I still want to have this background.
Jon Krohn: 13:29 What is that game called in French?
Penelope LaFeui…: 13:31 We don’t have it.
Jon Krohn: 13:35 It’s too low brow, too American. No pickle.
Penelope LaFeui…: 13:40 Yeah, that would be weird in French.
Jon Krohn: 13:42 I don’t know. I actually, I recently saw, I’ve lately I’ve been working out at this lifetime gym in Central Manhattan called, it’s a Penn Station location and they have tons of pickleball courts. And in fact, it’s impossible. Pickleball is so popular and there’s so few nice places that you can do it in Manhattan that there’s a waiting list. I’m already a member and I pay a crazy membership fee already, but I can’t access the pickleball courts if I want to access them. I do want to access them. So I’m on a waiting list to hopefully someday get an invitation. And then if I get that, I have to pay a fee, a huge fee, hundreds of dollars, plus my membership goes even higher just to have access to these pickleball courts. But the whole reason why I’m saying this is that the other day I saw a woman in there who was wearing a T-shirt.
14:33 She was playing pickleball and it said the pickle. So I don’t know. Maybe that’s what French people say. But yeah, so anyway, so pickleball, you do a lot of pickleball. That’s nice. It is pretty rejuvenating.
Penelope LaFeui…: 14:47 Yeah, absolutely. And it’s also a great way to meet people, which is another point about rest and recovery, which is that it’s also a lot about the people that you’re surrounding yourself with because hanging out with people who want to go out every night or every weekend, spoiler alert, there is a very high likelihood that you’re not going to be able to recover and to sleep well. By all means, I do go out and I do drink, but it’s all about striking that balance and surrounding yourself with people and activities that you enjoy doing together beyond just going to the bar or going to the restaurant, even if we are doing it afterwards, after playing pickleball.
Jon Krohn: 15:28 You basically, you set up a schedule, a fitness schedule that’s based on work, that’s based on your social schedule. It kind of sounds like that means that, does that mean that you need to be in a really rigid routine or is there some flexibility as well?
Penelope LaFeui…: 15:42 Both actually, meaning that I still want to be able to go to the gym four times a week, just because I know that it’s good not only for my physical health, if I want to build or even to just maintain the muscle that I have, but also for my mental health. I typically go right after work. I work New York Time and I’m on the West Coast. So it’s typically in the middle of the afternoon, which is good because there are not that many people at the gym. And so it’s a good break between my professional life and my personal life because then when I go out of the gym, my friends are also out of work and so I can go hang out with them or even go play pickleball. So I know that I’m going to be working out four times per week, usually Monday, Tuesday, Thursday, Friday, but it’s not super rigid.
16:36 I just want to get those four times per week in.
Jon Krohn: 16:39 Excellent. And so how does nutrition fit into this? Because you mentioned that that’s part of the change that you made as well. So in addition to kind of having this balanced fitness structure, what are your key tips for refueling for your job?
Penelope LaFeui…: 16:55 Yeah. So I would say the main two tips are around eating enough protein. So I try to have one gram of protein per pound of body weight just because as I’m working out, I want my muscle to have the right amount of protein in order to be able to grow or just like maintain them and also for recovery. And the second one is about timing my calves the right way, meaning that I don’t want to have a huge glucose spike when I’m in the middle of my work session and then I just want to go nap. So what I do is like usually I have most of my calves in the forms of either like rice, potatoes, fruits, sometimes ice cream, mostly around my workout before, during and after, because it’s the prime time for your body to absorb those carbs the right way without feeling sluggish afterwards.
Jon Krohn: 17:51 Got it. So ice cream before, during, and after working out is the key to being a successful data scientist. I love that. And you also, you mentioned to me when I asked you before we started recording about kind of your top tips related to avoiding burnout as a data scientist, you said that rest is the most important thing. Is there anything you want to dive into on that?
Penelope LaFeui…: 18:15 Absolutely. I would say more specifically around sleep. I try to have at least eight hours of sleep, which means actually more than eight hours in bed. It’s more or less, I would say 8:30, 9:00 just because I love reading before bed to downregulate a little bit because if you’re swiping or on your computer, there is no way you’re going to be able to fall asleep. And it’s also about the quality of the sleep that you have. It’s not only about eight hours, but eight hours of quality sleep with deep sleep, REM sleep for your brain. And the best way to have it is also to be able to down regulate and not be stressed during the day. And for me, it comes in the forms of just going on walks outside, even just a five minute walk in between meetings to kind of shift my brain a little bit instead of being always in the go, go, go mode.
19:13 And instead of swiping on Instagram, I just go on walks without my phone. Sometimes I forget my keys and I’m locked outside. And so just going on works outside and helping me not only focus for my meetings afterwards on my work session, but also telling my body that you can downregulate in five minutes, which then is a good way to fall asleep faster and to avoid waking up in the middle of the night if you’re too stressed during the day.
Jon Krohn: 19:45 A healthy bondy is a critical foundation for a healthy mind. And I personally believe that keeping active and well-rested are important and effective ways to avoid burnout. I’m taking my next clip from episode number 947. My conversation with Netflix senior data scientist, Jeff Li was a popular one in that episode, not least because it gave listeners a little insight into getting a job at tech giants like Netflix, Spotify, and DoorDash. Here, Jeff explains the application and interview process behind his success, netting roles at these highly desirable companies. So we’ve talked about some of the approaches that you’ve been applying at top companies like Spotify, Netflix, DoorDash, some of the most competitive companies to get into for any kind of role, whether it’s data science or software engineering or marketing or even probably people working in legal, HR, those are probably some of the companies that people most want to get into.
20:43 What kinds of tricks do you have for our listeners to get hired by these top firms?
Jeff Li: 20:46 Yeah, I have so many. So I would say I’ll kind of talk through all the tricks that I used. So when I was trying to get into DoorDash, there’s a technique I learned back then called the briefcase technique, which I think actually still works. And I’ve talked about this on a number of podcasts, but basically the idea is that the core principle of getting a job, it’s not like interviewing. It’s actually, can you add value and solve that company’s problems? So an interview is just a way to kind of test your skillset to see if you can actually solve their problems, but in reality, you can actually just circumvent this and actually see if you can solve their problems. So an example is actually at DoorDash, what I did was I figured out what their biggest pain points were and then I put together a doc outlining, “Hey, these are your problems.
21:39 This is how I would solve it. ” And it was pretty detailed. I spent at least like a couple days on it and then you send it to the hiring manager and sometimes it’ll hit. It’s not always going to hit, but you have a reasonable shot for it to hit. And if it does hit, then they’ll actually be much more sold on saying, “Hey, okay, this person has a skillset that could solve my problem. I’ll bring them in again to have a conversation with them.” So I found that technique to work pretty well for-
Jon Krohn: 22:08 Do you mind if I interrupt you for one quick second on this one? On that first point there. Something that’s interesting today is, so you started working at DoorDash more than five years ago, I think, if I remember. Yeah,
Jeff Li: 22:19 It was a while ago. Yeah.
Jon Krohn: 22:20 And so back then, obviously we didn’t have generative models that could be creating. So if somebody gets an email from you that’s like, these are your pain points and you clearly spend a lot of time, a couple of days. This is quite an unusual email to get probably for a hiring manager. And that’s part of what makes it such a great tool for getting hired. But I wonder if today a hiring manager might say, “This is definitely GenAI.” And it’s almost like the more effort you put into it, the more crisp and perfect it looks, maybe they’re more likely to say this is something that was just created by GenAI. This person doesn’t … They’ve figured out some relatively simple agentic workflow to be spamming tons of hiring managers with these what look like very complex reports. Because I get that for the podcast, for example, or for my consulting business, for Y Carrot, I end up getting these huge PDFs, like 30 page PDFs with illustrations, tons of detail, but I get them once a week for each business from some different random person who’s just doing a cold outreach.
23:29 And I’m like, “This is definitely some agentic thing going on.
Jeff Li: 23:31 ” Yeah, I see. I see. So it’s an interesting point because back then it definitely would’ve worked. I think today I still don’t think people are doing it as much for these big companies. Definitely for you, you have an audience, so people are going to reach out.
Jon Krohn: 23:51 Well, and it’s a different kind of thing. I’m getting sales pitches.
Jeff Li: 23:55 Yeah, I see. I see.
Jon Krohn: 23:55 And so I think that it could be something very different. Maybe nobody is … I mean, because actually, because I haven’t been getting those at all, people reach out and say, “Are you doing any hiring?” And it’s a relatively simple message. Nobody has been sending me these big like, “These are your pain points. This is how my data science expertise, this relevant experience I have could be useful.” So maybe you’re right, maybe it would stand out anyway.
Jeff Li: 24:16 Yeah. So I do agree that for the GenAI piece, you can easily spam that for sure. I think the hard part is still you want it to hit if you have a gen AI because I think a lot of times, even if I do this, if somebody does it to me, if they didn’t actually understand my problems, then it’s not really going to hit, then I’m going to ignore it. But then if they actually said, “Hey, I listened to all your podcasts and I figured out, hey, your podcast is maybe kind of like you could use this kind of software.” I don’t know, you can improve this aspect of it. And it was very specific and it was clear that they understood what you needed, then it would hit a lot better.
Jon Krohn: 24:59 Use the magic button you press to 10X your audience and revenue.
Jeff Li: 25:03 Exactly. So I do think that was one trick that worked back then. I haven’t tried it recently because I don’t think I need to as much these days because I think as you get more experience, it’s easier to get your foot in the door.
25:19 But I would say my recent job at Netflix, the way I got it actually wasn’t through applying or using any kind of tactic. It was basically because I had been working in ads at Spotify. I was working in forecasting. So I had this unique intersection of skills in ads and forecasting at Spotify. And then when Netflix decided to start doing ads, they needed forecasting is essential to running an ads business. They needed somebody with that exact experience and that exact overlap and skillset. So then I was basically the perfect candidate. And then when I interviewed, it was like pretty … I’ve been thinking about this stuff for the last few years, so it was pretty smooth, but I had been rejected multiple times before at Netflix.
Jon Krohn: 26:12 You had been?
Jeff Li: 26:12 I had been. Oh, there you go. So I think that the key thing is you have to just keep trying even if you get rejected. And then also too, it’s like, do you have a unique intersection of skills that is hard for the person to hire for? Because I think there’s many people that will do forecasting, there’s many people that do ads, but there’s a lot fewer people that do both. So I do think that as people develop their careers, it is good to build some sort of niche expertise and an overlap of say industries and skillsets. And that’s actually what really differentiates yourself from the rest of the market.
Jon Krohn: 26:46 Mixing in some niche expertise with the basic requirements is undoubtedly a sound strategy for getting hired. For this next clip, we look back in time to episode number 949. And it’s Stanford Professor Sandy Pentland talks to me about how history shows that systems need to be designed with reference to their human contexts. Yeah. So it sounds like you might be making a parallel there between the way that the Soviet Union fell apart as a result of the reliance on the kinds of AI systems that they had then. And maybe are you suggesting that this system that we have today could fall apart because of the AI systems that we have today?
Sandy Pentland: 27:22 Oh yeah, absolutely. And it’ll be the same sort of issues, bad data, the fact that it doesn’t take into account the fact that things change quickly, the fact that it doesn’t really pay attention to human nature and how people work. People are not logical. Come on, right? But we do know a lot about how people make decisions. And if you build something that is complimentary to that, you get better system performance, system including the people, right?
Jon Krohn: 27:56 Right. Yeah. So I have a quote from you that humans aren’t rational. Instead, they’re what you call social foragers who learn by watching others. So they’re kind of like picking up social behaviors from cues, seeing other people. I see that in the way that my puppy learns as well.
Sandy Pentland: 28:13 It’s true of almost all animals and certainly all social animals. I mean, just to illustrate it, nobody ever reads manuals. Nobody ever reads the whole newsletter when they get a newsletter. What we do is we watch what other people do. We ask them the one or two questions that we don’t understand, and that’s how we sort of pick things up. Danny Kahneman, who’s a Nobel Prize economist who invented this fast and slow thinking stuff people have heard of, estimated that probably 95% of all the things we do come from other people, from the culture around us. This is the way we do it here. We talk about company culture, we talk about the culture of the country. And what that means is that there’s certain ways you do things and certain ways you think about things. And we’ve all sort of agreed that this is the right way to do it.
29:07 That’s our wisdom doesn’t mean it’s right. Don’t be suckered by that. And most of what we do is that, and then we add a little bit of our own circumstances to it for that other 5%. Other 5% is really important because it’s what makes you, you, but it’s not the majority. And so what you need to do when you design systems is you need to think about the context, the human context of somebody who’s making the decision. And this is where things like the way social media is designed thinks of people as just individual. And then it says, “Oh, well, we’ll just use machine learning to get people that have similar feature vectors.” They don’t think about the community aspect of it. So you get echo chambers.
29:57 Same thing as we’ve seen that you In social media, people decided you could make a lot more money if you allowed followers and a little bit of advertising. And so that’s what we got. And the consequence is that the only people you really hear on social media are these sort of very loud voices that have done everything they can to be heard. They’re trying to be violent and crazy and this and that. And we’ve been able to show very, very definitely that’s where polarization comes from. You get rid of these big voices and you discover that people actually pretty much agree with each other. And so if you’re building a decision system within a company or wherever, you can’t have these super loud voices. Otherwise, you’re just going to get craziness going on. Most people are pretty sensible if you sort of focus them on that and prevent the big voices.
30:59 We find you get much better decisions.
31:04 Another thing that sort of comes from this is people think, oh, well, human decision making, that’s only approximate. And really what we want is a logical framework and all that sort of thing. Well, in the experiments that we do, and we do this with people who trade $10 million a day. So these are people who are super quant, super expert. This is what they’re paid to do. If they pay attention to the other traders, they do better, reliably better. So all those equations, I mean, trading currencies, for instance, is a real traditional … You can write down the equations, but you have to pay attention to other people to avoid edge risks, unusual things, things that are changing. And actually, we probably all remember 2008 when the financial system almost fell apart. The core thing was they had an equation and they didn’t notice that that equation had tail risk, so unusual events.
32:14 And when an unusual event happened, it almost took down the whole system. The people who listened to other people who were also looking at actual human behavior in their equations came through it okay. The people who are just following the logic got creeped.
32:36 We are not bad at making decisions, particularly in small groups. It’s good for us to listen to the numbers. Absolutely. But if you just follow the numbers, you lose the context, you lose the changeability of the situation, you lose the possibility of rare events. And you tend to come up with systems that don’t perform as well as if you do take the sort of human context and thinking into account. That’s what the book is about incidentally. It’s like, how do we make systems, AI systems that are really robust to weird stuff? Because that’s where they fall apart. You get something that’s weird or an edge case. Things can blow up in very unexpected ways. We hopefully all know that. And that’s why some of this human reluctance to do new things, it’s attraction to wear unusual stories. Some of these traits we have are actually extremely valuable in making sure that systems don’t blow up.
Jon Krohn: 33:45 Finally, Dropbox VP of engineering, Josh Clemm talks us through one way to make sure that at least business systems don’t blow up, high level guidance. Here’s a clip from episode number 951 in which Josh explains what has to happen for AI adoption to be successful. Picking up on another piece that we found of something that you said online. In a LinkedIn post, you commented on a HBR, a Harvard Business Review report on how AI-generated workslop is destroying productivity. And this is something that we have talked about on the show a fair bit, but I’d love to hear your take on it. What are your two cents on how this problem hit home for you and what are your tips to address Workslope?
Josh Clemm: 34:27 Yeah. First of all, I love that term. I don’t know who coined it, but it absolutely is spot on. Workslop is the very plausible and somewhat impressive looking content that you might see at work that you maybe create yourself or somebody sends you. And then you start to look more at the substance and wait a second. This feels pretty generic at best or at worst. Frankly, there’s just hallucinations in there. And there’s a bit of a paradox because I think the more you work in AI, like I am and a lot of your listeners are, the more you can start to spot the patterns. Everybody kind of jokes about the MDash, but there’s other sort of markers that, “Hey, this is AI generated.” So I think the big question is, well, why is this happening? Why is so much workslop getting created out there? And you see different reports.
35:29 A lot of CEOs, almost three-fourths of CEOs feel there’s a lot of competitive pressure just to adopt AI. You hear that, “Oh, we need AI. I don’t know what it is, but we need it here.” And so you get these really early deployments or quick deployments, employees start using it and it isn’t really doing what you’re expecting. And we’re hearing that from some of our customers, “Hey, we need AI.” And the first question I ask is, “Well, what exactly are you hoping to accomplish? What are those use cases? What are the goals that you’re trying to do? ” And if you don’t really get crisp on that, you’re going to unfortunately get workslap. The way I kind of think about it, I don’t know about you, but back in, let’s say college, let’s say you’re writing an essay, you’re going to write, maybe you’re working all night, maybe the last minute you’re putting together all this stuff for your essay and the next day- That’s definitely me.
36:21 Yeah, exactly. The next day, you don’t just turn in that first draft. You don’t sort of do a first draft and you’re like, “All right, I’m done. Did my work, send it off. It’s going to be an A. ” No, absolutely not. You read it over, you update, you make things stronger, maybe you go to the source and you start switching in some words, you add in a lot of extra research. That’s really how we should be treating a lot of these AI tools. They can be phenomenal partners. They can be phenomenal at generating a lot of that first draft, if you will, but it still requires that human touch. It still requires really ensuring that it reads correctly. It’s high quality. The signal to noise ratio is very high and it has very much a verifiable facts. Very, very kind of important to get right.
37:11 So how do you kind of fix that other than more of this guidance, high level guidance? Kind of goes back to the stuff we were talking about before. Your work context just matters a ton here. A lot of companies will maybe superficially go off and add third party connectors. There’s a very popular approach is to use MCP tools in agents, and they are a phenomenal … That’s a phenomenal protocol to kind of get up and running and build some really impressive agents right off the bat, but they’re very slow. They can’t really get access to all the types of content you may want, and they use a ton of tokens, a lot of cases. So while that’s a good solution, you still really need to kind of think about where you want to get your work context overall. On our side with Dash, like I mentioned before, we do bring everything in, we ingest it, we do understanding, and we then index it.
38:12 Right now we use both. We’re building both Elexical and a vector index. On the electrical side, we use BM25. It is still the workhorse. This thing has been around for a few decades and it is very, very good and more like keyword type searches. And this is important. If your customers need part numbers, you need to do more of a keyword search. If your customers are creatives and they’re looking for vintage cars, okay, great. Symantec works there. A lot of customers will want both. And so you really want to have that kind of hybrid retrieval. And so that’s something we’re doing here to just ensure the context that we’re providing these LLMs, these agents are of the highest quality. Other things to look at are the evals. I mentioned before, when we talk to customers, you want to understand what are their goals? What are they trying to accomplish?
39:06 What metric may they want to move? A lot of that you can actually bundle and create a bunch of successful test sets like, “Okay, this is what good looks like. This is effectively my benchmark, my internal benchmark.” And then once you have that, you can compare it with some of these AI deployments and it’ll be much more clear. “Is this working or am I just going to get more workslock? “And the last kind of tip I’d say here, there’s still a lot of pressure out there to adopt AI. A lot of CEOs, CTOs, CIOs, got to use AI, got to use AI. And so the pressure’s there, but about 55% of employees, they don’t even know how to use AI. So I do think you should look at training. It’s essential to kind of do share outs, let people do demos of what they’re doing and your most likely candidates to do that training are probably your highest performer.
40:07 So even if you just went there, they’re likely already using and adopting these AI products. They’re likely doing it in a way where it’s much higher quality. Let them help train up the rest of the force.
Jon Krohn: 40:19 All right that’s it for today’s. In Case You Missed It an episode, be sure not to miss any of our exciting upcoming episodes. Subscribe to this podcast if you haven’t already, but most importantly, I hope you’ll just keep on listening. Until next time, keep on rocking it out there and I’m looking forward to enjoying another round of the SuperDataScience Podcast with you very soon.
