Jon Krohn: 00:00:00
This is episode number 877 with Shirish Gupta, director of AI product management at Dell. Today’s episode is brought to you by ODSC, the Open Data Science Conference.
00:00:17
Welcome to the SuperDataScience Podcast, the most listened to podcast in the data science industry. Each week we bring you fun and inspiring people and ideas, exploring the cutting edge of machine learning, AI, and related technologies that are transforming our world for the better. I’m your host, Jon Krohn. Thanks for joining me today. And now let’s make the complex simple.
00:00:51
Welcome back to the SuperDataScience Podcast. Today we’ve got a fan of the show joining us as my guest for an episode about efficiently designing and deploying AI applications that run on the edge. So like on local laptops, workstations, that kind of thing. That guest, who’s also a fan of the show is named Shirish Gupta. He has spent more than two decades working for the global technology juggernaut Dell in their Austin, Texas headquarters. He’s held senior systems engineering, quality engineering, and field engineering roles at the firm. For the past three years, he has been director of AI product management for Dell’s PC group. He holds a master’s in mechanical engineering from the University of Maryland.
00:01:32
Today’s episode should appeal to anyone who’s involved with or interested in real world AI applications, which I’m assuming is just about every listener to this podcast. In this episode, Shirish details what neural processing units NPUs are and why they’re transforming AI on edge devices. He provides four clear compelling reasons to consider moving AI workloads from the cloud to your local device. He talks about the AI PC revolution that’s bringing AI acceleration to everyday laptops and workstations. What kinds of large language models specifically are best suited to local inference on AI PCs? How Dell’s pro AI studio toolkit will drastically reduce enterprise AI deployment time. And he provides plenty of real life AI PC examples, including how a healthcare provider achieved physician level accuracy with a custom vision model. All right. You ready for this illuminating episode? Let’s go.
00:02:24
Shirish, welcome to the SuperDataScience Podcast. Where are you calling in from?
Shirish Gupta: 00:02:34
I am calling in from sunny Austin, Texas, Jon.
Jon Krohn: 00:02:38 Excellent. So you’ve been a long time listener to the show. How long have you been listening to it, Shirish?
Shirish Gupta: 00:02:44
I’d say about a year and a half.
Jon Krohn: 00:02:46
Nice. I’m putting you on the spot here a bit, so no worries if this is too much pressure, but any standout episodes for you that-
Shirish Gupta: 00:02:54
Oh, yeah. I have one. I don’t remember the number, but it is the one in which you go deep into what is a transformer.
Jon Krohn: 00:03:04 Oh, Yeah.
Shirish Gupta: 00:03:05
I really enjoyed that one. I’ve listened to it multiple times.
Jon Krohn: 00:03:08
Yeah. That’s 747.
Shirish Gupta: 00:03:10
747.
Jon Krohn: 00:03:11
Yeah. It’s easy to remember because of that. And people bring that up. It’s interesting. I think that’s a lot of people’s favorite episode. It had the most listens, if I remember correctly, in 2024, and a fair number of people came in as listeners to the show because of that. Cool. Well, it’s great to have you on the show to have a fan of the show on the show. So we’re going to talk in this episode a lot about neural processing units, NPUs, which I don’t know that much about. I know that they are an AI accelerator alternative to a graphics processing unit. So we have had episodes on the show in the past that have talked about alternatives to GPUs as an AI accelerator. But this episode we’re going to talk a lot about NPUs, specifically neural processing units, and that is not something that we’ve talked about on the show specifically before. So fill us in on what neural processing units are.
Shirish Gupta: 00:04:07
Be my pleasure to Jon. So they’re fairly new. NPUs stand for, as you said, neural processing units. They’re maybe a year old in terms of being out on the market. The very first ones that came into the market were with Intel’s Meteor Lake Chipset, which launched sometime around this time last year. So I’d say fairly new kid on the block. Why they’re exciting is because if you think about the devices that they are incorporated into, which is your average laptop or a desktop even. But a PC that is used by your everyday knowledge worker, it’s used by individuals for their personal use, but just your typical PC. For the longest time, we’ve had CPUs in these PCs. For many, many years we’ve had GPUs or graphics processing units and those are integrated and discrete. Integrated graphics processing units or IGPUs are more common. Have been more common for many, many years.
00:05:24
The NPU is the newest kid on the block. So what they do is it’s a very, I’d say, purpose-built architecture that is designed to do one thing really, really well, which is matrix math. And just because it is almost hard-coded to that extent, it is extremely efficient in terms of power consumption for those multiplications and additions in a matrix, which is essentially the building blocks, as you know for AI and ML workloads. So is this super important for your average PC? Because yes, you can run AI and ML on the box, on the CPU and the GPU, but your battery isn’t going to last very long if you keep that up.
Jon Krohn: 00:06:16
Gotcha. So it’s more efficient. So an NPU, it would be a replacement for a GPU for training or inference time running of a machine learning model, probably particularly a deep learning-based model, like a large language model. Most foundation models out there, those are all based on a deep learning architecture. Those have tons of matrix, multiplication and addition as you say. And because the NPU is designed for that specifically, instead of being general purpose like a GPU is originally designed and still today is used for all kinds of uses, like rendering graphics, a graphics processing unit, or mining Bitcoin. An NPU wouldn’t necessarily be great for rendering graphics or mining bitcoin, but if you’re training or running an AI model, an NPU will be more efficient. And so on your PC for example, there, it’ll save you battery power. It might be more efficient as well perhaps in terms of time.
Shirish Gupta: 00:07:19
The NPUs, at least for now, I think the future holds a lot more possibilities. But for the time being, for the foreseeable future, I’d say the NPUs are most suitable for inference workloads. I think you still need GPUs for training, fine-tuning. NPUs are going to be amazing for AI in production or consumption by people. So your data scientists, AI engineers, app developers, they’re going to use a workstation to build these AI capabilities. Guess what? They’re building it for someone. Someone’s going to use them. Think about the average knowledge worker. They’re going to start using on the box capabilities that allow them to do anything from use AI to really accelerate productivity for anything from just asking questions to an assistant that can quickly give them the answer from a vast knowledge base to even embed it into a workflow, which is where you get into a little bit more into agentic and multi-agent workflows, which again, I foresee that in the future.
00:08:43
But the point I’m getting to is you’re really going to use an NPU for inferencing, and that’s where your efficiency matters. That’s where your average knowledge worker is going to use something. You don’t want to start offloading AI features to a device only to tank its battery life. That’s going to lead to a pretty bad experience in the long term. So it’s pretty important to keep that in mind that while you can still need GPUs and high-powered workstations to build your solutions, NPUs are perfect for those being consumed by the average person.
Jon Krohn: 00:09:32
I gotcha. So we’re talking about taking capabilities that today might require you to have an internet connection and depend upon some cloud service in order to get some like say large language model or other foundation model capability. But instead with an NPU, you could potentially have the operations, the inference time calls. Instead of going out over the internet and using cloud compute. You can have it running locally on device, so you’re also probably going to get lower latency, you have fewer dependencies. Yeah. Talk us through some of the other advantages of being able to now do things on edge instead of having cloud reliance.
Shirish Gupta: 00:10:16
Yeah. I think this is a perfect segue. In fact, this is a mnemonic that I came up with myself. The term that is being thrown around for these devices with NPUs is an AI PC. Sure you’ve heard of it, So to think about the benefits of an AI PC, I’ve created a mnemonic with those four letters. So A is accelerated. It’s basically you have now a local hardware accelerator that gives you that low latency real-time performance for things like translation, transcription, captioning, and other use cases where latency is super important for persistent workloads. So that’s A. I is individualized. Again, this is great because if you have an AI that is on your box, it has the ability to learn your styles. Let’s say if you’re creating emails, if you’re using it to generate emails. It’s learning your style. It starts writing in your style.
00:11:14
It’s great for … We had a healthcare customer that we’ve been working with on a use case. There’s two parts to it. I’ll talk about the second part. The first part is even more interesting, but I think it’s related to a different example that we’ll come back to later. But the second part of the AI solution is that they were taking information from a physician’s diagnosis of a patient in the ER and they were using that information to auto-generate the physician’s report. Mundane stuff. Physicians don’t like spending time on that. They’d rather go to the next patient, have that interaction, increase their ability to spend time with patients. But the feedback they gave was with this solution, now that it started seeing the way that I’m changing and editing its initial draft, it’s starting to take on my style, and now it just sounds like me. And I love it because I don’t have to do this report generation, it does it for me and I’ve got more time for my patients. That’s the individualized value.
00:12:27
The third is P, it’s private. Like you said, the data doesn’t have to leave your device and its immediate ecosystem. You don’t have to send it back and forth to a public tenant or even a private tenant for that matter. You may have confidential information with PII that you have access to, but you don’t want to merge with even a private tenant. There is sensitive information like that or unclassified information depending on your vantage point. So that inherent privacy of data and the inherent security of running the model locally on your device gives you that assurance that this is more private than it would be. That’s P.
00:13:12
And C … This is really important because I hear this from customers. It is an important cost paradigm shift. And I’m starting to hear this from some of our maybe earliest adopters of on-device AI. Which by the way is not ubiquitous today. In terms of enterprises building out their own AI capabilities and using on-device accelerators for offloading them, we’re at the tip of the spear with Dell Pro AI studio, and we’ll come back to that later. But the early adopters, what they say … And I had a finserv or financial services customer tell me, Shirish, my developers are using CodeGen and they’re using our data center compute. 15% of my data center compute is going to these developers that are using it for CodeGen or code completion or writing test cases, unit tests, what have you. They all have PCs. I want to get them to an AI PC with a performant NPU so I can offload that compute from my data center because they don’t need H100s hundreds to do that code completion. I think I can do that with the NPUs on your Dell devices.
00:14:34
So that’s a real opportunity as well, is just because you have the compute doesn’t mean you should use it. It’s like the right compute or the right engine for the right workload at the right time. So there’s plenty of use cases where offloading from even your private data center to a on-device capability makes a ton of sense. And then if you’re actually using the cloud, you’re paying for every inference. It’s tokens and an API access. So now that you’ve got an AI PC, it’s no cost to you. You built your solution, you’re using it on the device. That’s it. So cost is a big factor. Now, you’ll argue that the cost of inferencing in the cloud is coming down. It’s scaling very fast. But again, I get back to the point that it’s the right engine for the right use case at the right time. Just because you have it doesn’t mean you should.
Jon Krohn: 00:15:35
Excited to announce, my friends, that the 10th annual ODSC East, the one conference you don’t want to miss in 2025, is returning to Boston from May 13-15! And I’ll be there leading a four-hour, hands-on workshop on designing and deploying AI Agents in Python. ODSC East is three days packed with hands-on sessions and deep dives into cutting-edge AI topics, all taught by world-class AI experts. Plus, there are many great networking opportunities. ODSC East is seriously my favorite conference in the world. No matter your skill level, ODSC East will help you gain the AI expertise to take your career to the next level. Don’t miss – Online Special Discount ends soon! Learn more at odsc.com/boston.
00:16:22
Something that comes to mind for me when I think about this initially as you started describing this, I was thinking about myself sitting at a desk with a laptop, and I’m sure that a lot of users of this AI PC paradigm with an NPU in it are doing that. But when you talk about cost and lots of inferencing, is it the case also that these get used for commercial or industrial applications where you have an AI PC that could be sitting there in a factory or in some commercial setting where it could be basically continuously analyzing images or audio. And so if you were sending that high bandwidth information, images, video, audio, if you were trying to send that over cloud, there’s huge bandwidth costs. If you had a bunch of machines doing it, then you’d need to make sure that you had a network that could support all that, and then you’d also have much bigger costs on the cloud side as well. So is that use case also relevant here?
Shirish Gupta: 00:17:15
Very relevant. In fact, there are vision models like YOLO, just for an example where you’re totally right. It makes a lot of sense to do that at the edge. And it’s real time and it’s much cheaper. So those are use cases we’re looking at in manufacturing. We even have a customer who’s into insurance company. They want to use a capability like this to take pictures of damage. Use a model like that is good for image classification. To be able to go back, refer to that database and look at, okay, for this damage, what category am I looking at? Is this going to cost me T-shirt size, small, medium, large just to get the adjuster maybe 70% of the way there before they have to make their judgment? So that’s one use case we’re looking at with another customer. Manufacturing is an absolutely very valuable use case for customers to offload to the device right there in the factory so that they can do real-time anomaly detection of defects.
Jon Krohn: 00:18:31
Right. Right. Right. Right. Right. Yeah. Real-time anomaly detection. Perfect application example there. Cool. So thank you for giving us that rundown of your AI PC mnemonic. So accelerated, individualized, private, and cost.
Shirish Gupta: 00:18:42
Effective.
Jon Krohn: 00:18:43
Cost-effective. Yes. That makes more sense. I didn’t quite finish my note there. So very cool. Makes it easy to remember some of the advantages. Are there also compatibility issues that are resolved in this framework where if you have different chips from different manufacturers like Intel, Qualcomm, AMD, are also handled in this paradigm?
Shirish Gupta: 00:19:10
Great question. So you’re actually touching into one of the reasons why we built Dell Pro AI studio. So without going into that, I would say this because the NPU is such a new architecture, we are working very closely with our silicon partners to get these array of models that support this variety of use cases ready and compatible with the DSOCs. So that itself is a really important point for us to touch on later, which democratizes access for developers to use a variety of silicon targets without having to start over every time they are faced with a new silicon architecture. That’s pretty important.
Jon Krohn: 00:20:03
Nice. Okay. So now let’s talk about the mechanics from the perspective of the data scientist or the software developer who’s using these tools, or maybe even from the perspective of a click and point user, like you say, a knowledge worker who is not necessarily coding, but they’re taking advantage of some application that was built on an AI PC. In these kinds of scenarios, you’ve said a number of times how it’s a PC, obviously, so I assume it’s Windows. Some of our listeners will primarily have been doing their work on Macs or other UNIX-based machines like Ubuntu. Why should somebody be considering using a Windows computer? And as a follow-on question to that … If this isn’t too much that I’m putting in there. I’d love to understand what the mechanics are like. If I’m a data scientist or a developer using AI models or developing an application with an AI model, what kinds of tools am I using on my AI PC? This question came to me as you were answering the compatibility, because you’re talking about how you only get set up once and then you can work across all these different silicon providers, and then I was like, “Oh, what does that look like?” What does it look like when I’m doing the work on a PC?
Shirish Gupta: 00:21:13
Yeah. And it’s a great question. Again, keep in mind that we’re talking about inference time here. It’s all about outcomes that your AI engineers and developers are driving for. And so you think about who’s going to consume the applications that are going to run on these NPUs and AI PCs. It comes down to a Windows environment, that’s where they’re most likely going to get consumed. So if you start with that paradigm, if you’ve got to actually build a solution that has to run on an AI PC NPU that’s running Windows, you’ve got to make sure that your model and runtime and your ability to call the model and run the local host on the AI PC … You can do some of your development work on a UNIX-based machine. Absolutely. Because ultimately, what are you doing? You’re going to take all of these AI bits, you’re going to take the model, you’re going to combine that with the ability to run it on a local host on an AI PC. But you still have to build your app and then you’ve got to integrate these bits into it.
00:22:21
So you have the option of using your IDE of choice to do all of that piece. But when it’s time for you to actually integrate your model into the app and set up the local host and run it and make sure that it’s working, you’re going to have to do that piece on a Windows-based device because that’s where your app’s going to run. Does that make sense?
Jon Krohn: 00:22:49
It makes perfect sense. And you’re absolutely right. Windows is still in the corporate world in a lot of enterprise applications, in a ton of industrial applications, windows is the default operating system, and so it makes a huge amount of sense to me.
Shirish Gupta: 00:23:07
I will add one thing, Jon. Sorry
Jon Krohn: 00:23:09
Go ahead.
Shirish Gupta: 00:23:11
I think I’m getting ahead of myself here, so we should come back to this when we talk about Dell Pro AI Studio. I’ll let you ask your follow-up.
Jon Krohn: 00:23:20
Okay. Okay. I would still love to have the question answered for me, if you may. There must also be, because I don’t have experience developing applications. For this episode’s purpose, we’re talking about inference. So a large language model has already been trained, or maybe I’m taking some open source model weights, some DeepSeek model weights, or some Llama model weights, and I’ve got them on my local PC. If I want to build an application as a data scientist or a software developer on that PC, what do you recommend? What are the best tools for a data scientist or a software developer on a PC in that scenario?
Shirish Gupta: 00:24:04
There’s a variety. I think the choice of the developers. This is where I think our intention is to arm developers with the tools so that they can actually integrate into their apps and their own choice of IDEs the tools and the framework and the blueprints to be able to get workloads to run on the IPC NPUs.
Jon Krohn: 00:24:30
Okay. Fantastic. All right. So when you apologized for interrupting me, but you didn’t really interrupt me, it’s just a conversation and you said we can come back to some point that you wanted to make when we start talking about Dell Pro AI Studio. So let’s talk about that now. The Dell Pro AI Studio. Tell us about what that is. It’s a specific implementation as far as I know of an AI PC that incorporates neural processing units, NPUs. But tell us more about it. Tell us why they were developed. Tell us why they could be helpful to our listeners.
Shirish Gupta: 00:25:04
So very important to note, Dell Pro AI Studio is a toolkit. So think of it as an SDK. But it’s got an array of tools, models, a framework, actually frameworks and recipes. So it’s the tools and how to use them. So I get asked this question very commonly. It’s not an app that’s going to be shipped on the Dell AI PC from the factory. It is targeted towards developers and IT pros that want to build, deploy, and manage apps with AI features on their fleet or subset of their AI PC fleet, So that’s what it’s for. So when I say toolkit, what is it? So it’s got three parts to it if I really boil it down at a high level. At least our initial release will have three parts. And each of them solves a problem or a pain point that we heard from our customers as well as experienced ourselves as we tried to bring workloads to the AI PC for inference time.
00:26:22
So one, we have Dell validated models. So we have curated or hand-selected a range of open permissive models that cover a variety of use cases from language, speech, and vision to enable a variety of use cases. Well, you can get those from Hugging Face, you’d argue, and the answer is yes, you’re right. They’re actually going to be available on Dell Enterprise hub, which is on Hugging Face. We already have that capability today and models for enterprise today that are targeted to run in containers for training, fine-tuning on servers, on Dell servers. So we already have that. We’re going to add the models that run on the IPCs right alongside those server models, if you will.
00:27:20
These would be smaller models. Think for a language model, I’d say today’s NPUs is perhaps the most you can fit. And I say fit, it’s a loosely used term. I’d say get the performance that you think would be acceptable for the average user would be around 15 to 20 tokens per second. So to get that target LLM output, I’d say you would probably not be able to go beyond an eight or nine billion parameter model with quantization. But that’s still pretty good. Today those models are extremely capable. And so this inflection of NPUs is getting more performant with 40 plus tops on the NPU and these smaller LMS getting way more performant and accurate, that inflection point has really enabled local inferencing on the device, So that’s what we wanted to democratize for customers.
Jon Krohn: 00:28:23
Did you know that the number one thing hiring managers look at are the projects you’ve completed? That’s why building a strong portfolio in machine learning and AI is crucial to your success. At Super Data Science, you’ll learn how to start your portfolio on platforms like Hugging Face and GitHub, filling it with diverse projects. In expert-led live labs, you’ll complete an exciting new project every week. Plus, through community-driven projects, you’ll tackle real-world, multi-week assignments while working in a team. Get hands-on experience with projects like retail demand forecasting, building an AI model from scratch, deploying your own LLM in the cloud and many more. Start your 14 day free trial today and build your portfolio with www.superdatascience.com.
00:29:04
Nice, so I’m going to repeat that back because it seems like a really important point. That this paradigm that we’re talking about today of having NPUs running locally or on the edge … Same idea. On an AI PC, at the time of recording in early 2025, that’s ideally suited to a seven, eight billion parameter model. Okay. Perfect. And so seven, eight billion parameter model. Yeah, absolutely. If you’re using a Llama-7B model, they are hugely capable today. And you’re saying that it will pump out 15 to 20 tokens per second, which corresponds to about 10, 15 words per second. Much faster than somebody can read. And so yes, absolutely. That sounds like a sweet spot. Perfect.
Shirish Gupta: 00:29:51
Yep. Yep. You got it. So I went on a tangent there, but I was talking about the elements of Dell Pro AI Studio. So back to the Dell validated models. We realized one of the key pain points was for anyone trying to build AI features that run locally on the device they either have a model that they’ve picked out and they don’t know what system it can run on, or they have a system in their fleet and now they want to use it and they will figure out, okay, what model can I run on this? That’s typically the way that I’m seeing customers approach this. And we’ll get to like custom and fine-tuning later. Let’s keep it simple for now.
00:30:34
But let’s just say you’re using a base open permissive model. And if you’re a developer, you’re going to have to iterate. You’re going to take a model and a system and you’re going to figure out whether they can actually coexist. Can it run? Can it give the performance I need? Is it going to give me that 15, 20 tokens per second performance? How big does the model need to be? Let me try a 14 billion parameter model. Let me try 30 billion parameter model. You quickly realize, oh, I don’t have enough memory. Let me go to 32 gigs. Let’s try again. Oh, it runs now, but oh God, it’s like two tokens per second. That’s not going to work. But believe it or not, every iteration takes a lot of work for them to get to that point. And that’s insane. Anyone faint of the heart is going to say, “Okay. This is too much work. I don’t think I’m ready to build for the NPU quite yet.” So it’s a huge deterrent today.
00:31:37
So by solving that problem and having models and systems paired up like, “Hey, here’s the performance you’re going to get. It’s been fully tested by Dell. It’s fully compatible with silicon and a variety of silicon. It doesn’t matter what SOC you’ve got. If it’s a Dell system with this specification, you are good to go.” And all of these use cases are enabled by this curated set of models. It’s a big pain point. It seems trivial, but it’s a big pain point that we’ve solved with that set of models on Dell Enterprise hub.
00:32:12
The second element of Dell Pro AI Studio is its enterprise readiness, which is massive, Today you have plenty of developer POCs. There are apps out there, I won’t name them, but there are some really good apps and people who want to test and see what running a model locally on the device and chatting with it looks like. I encourage people to go find those apps. And those apps are great for bringing a model, running it locally and chatting with it. But that’s it. It’s a POC. What do you want if you’re an enterprise? You want to be able to turn that into value for your end users. You want to drive productivity. You want outcomes. So you’ve got to figure out how you’re going to use it in an actual example that is going to make your users productive, like the ones we talked about, like the manufacturing anomaly detection, and there’s several others I can give you for the average knowledge worker.
00:33:18
If you really want to now take that capability and deploy it across a fleet and control it completely end to end, including the data that goes in and out of those apps, that app has to be enterprise ready, which means your model’s got to be enterprise ready. It has to be testable so that you’re not injecting risk into your enterprise. And then you have to be able to control it through its entire lifecycle. That’s what enterprise ready means. And then you’ve got to, as an IT pro, you want to be able to manage every aspect of it. Who has permission? What models get deprecated? Updated? How is it being used? How are the apps being used? What’s the telemetry? None of that exists today. So that’s the other piece. Via Dell management portal, you have that enterprise ready capability where every element of your solution that uses the Dell Pro AI studio is ready for the enterprise. It’s ready to be embedded into your workflows and your data is completely in your control. That’s the part two.
00:34:28
Part three is our middleware. It’s the framework that runs on every device that will be in the fleet where these AI apps are deployed. And it does all of the cool stuff. Model operations are automated through this. System and model discovery is automated through it. So it can detect the silicon on the device. It knows the model that the developer intended for that silicon. It pulls that from the registry and it slots it in. It can slot in, slot out, load, unload, secure it by verifying attestation. All that stuff, which is if you’re a developer, you would have to build that into your app. And that’s not trivial. You’re talking about multiple libraries of code with hundreds and hundreds of lines of code that you’ve got to write, debug, test, and manage. And at the end of it, if you have a change in your features, if you want a new model, you’ve got to spin on a whole new version of that app. Quickly becomes very unsustainable if you’re an enterprise-grade developer or IT pro, you just can’t sustain that over the long term. So that enterprise-grade capability that is enabled through Dell management portal and that AI framework on the box is pretty special.
Jon Krohn: 00:35:57
Nice. So what I’m hearing is that with the Dell Pro AI studio, you’re allowing your users, developers, data scientists of AI applications to be able to dramatically accelerate their timeline from proof of concept to enterprise application.
Shirish Gupta: 00:36:16
Yes. Accelerate and simplify. In fact, for lifecycle management, it’s a big enable because it doesn’t exist today for the PC.
Jon Krohn: 00:36:26
Nice. Yeah. That sounds like actually really game-changing. That’s really exciting. Okay. So if we have this accelerated capabilities, what kinds of use cases are you seeing for people using Dell Pro AI Studios, using your software development kit, your SDK? What are the kinds of practical applications that you’re seeing happen?
Shirish Gupta: 00:36:46
So we already covered CodeGen and code completion. We covered manufacturing anomaly detection. Let’s talk about a couple that are really cool. I talked about the insurance agent or adjuster using image capture to get started on their claim for damage. We’ve been talking to a shipbuilding company or ship inspection company. They could use the same for the field .have the ability to go take this app, scan the parts of the ship and automatically check for damage so that you’re not completely reliant on the human. But do a quick scan through, document the damage during inspection, and again, auto-generate the report. So those are all really cool use cases.
00:37:40
Another one that we’re looking at is first responders. If you have EMS or police that’s responding to a distress call, they go onsite and they realize the person or the victim doesn’t speak their language. Now you have real-time translation on the app. No latency or low latency, because that’s super important without having to be relaying that info to the cloud. And then on top of that, if you couple that with … You take the transcript of that entire conversation and you convert that into an auto-case generation. That’s massive for the officer or the EMS prep for the personnel. Saves them a ton of time and paperwork that they have to usually do after hours or between calls. Improves accuracy of all the information captured. And guess what? The raw transcript can be discarded once the report’s created. So you have privacy and you’re not necessarily sending information that you shouldn’t send to a cloud somewhere.
00:38:45
I can go on and on. And you talk about general use cases within the enterprise. There’s one that we’re looking at within Dell, actually. I’ll give you two. You have a young engineer that comes across a defect while they’re developing a product and they want to check, have we seen anything like this before? And what did we do? What were the possible causes and how did we solve it? They can quickly go into the VA, the virtual assistant, and say, “Here’s a summary of my issue. Tell me if there’s anything like it in our knowledge base.” Thinks for a moment it comes back with five Jira tickets with the links and says, “These are the owners of these tickets. Go talk to them and go to the tickets and figure it out.” That maybe it’s a good starting point. It’s just the possibilities are endless. And you always go back to why does it make sense to do them on the device? The truth is not every use case is meant for the device, but if it meets that AI PC mnemonic, you should consider it.
Jon Krohn: 00:39:51
Accelerated, individualized, private, and cost-effective.
Shirish Gupta: 00:39:57
Cost-effective.
Jon Krohn: 00:39:57
Yeah. So that’s really cool. All of these applications that you just ran through. And of course, something that I say on the show pretty frequently, especially on Friday episodes recently, is if you are listening to this episode, you could be a hands-on developer, data scientist or not, you could be a more commercially oriented individual listening to the show, but you’re interested in having somebody … Like you just listed a whole bunch of examples. You can go and talk to your favorite large language model. So for me right now, that would be Claude 3.7 Sonnet from Anthropic. When I’m looking to brainstorm on ideas for a particular application or use case, I would put in context. So if I was you listener, I’d be putting in the specific context about the situation that you’re in, what business you’re in or what business interests you have, or what academic interests you have, what background you have. What is your special niche or what are special data that you have access to that other people might not, that can form a bit of a moat. And then just talk to an LLM about what kinds of applications you could be building.
00:41:02
And then something like Dell Depro AI Studio could allow you to go from idea to proof of concept to enterprise application very rapidly in I guess a matter of weeks or months. And while you’re doing that, it sounds like based on these neural processing units that would be used in an AI PC, you mentioned earlier Shirish, this idea of a seven billion, eight billion parameter model being ideal. I highlighted already when you were speaking earlier that Llama models could be great for meta, but some other options that are small that could be great for this scenario that have a lot of capability are the series of PHI models from Microsoft as well as the Gemma models from Google. And I’ll have links to all three of those. Llama PHI and Gemma in the show notes as great options that people can be using. And potentially, I haven’t gone and looked to see exactly which models are in the pre-validated Dell Enterprise hub that’s available in the Hugging face, but I suspect that those are the kinds of models that would be.
Shirish Gupta: 00:42:11
Yes. And I should point out that Dell Pro AI Studio is currently in development, so you are going to see it maybe mid-year. We did announce it earlier this year, but we’re building it. We’re in early access right now. So I would say this, that if you’re listening and if you’re interested in early access and doing a beta program with us on it, please do reach out. And I’ll share the information with Jon to add in the show notes.
Jon Krohn: 00:42:44
Eager to learn about large language models and generative AI but don’t know where to start? Check out my comprehensive two-hour training, which is available in its entirety on YouTube. Yep. That means not only is it totally free, but it’s ad-free as well. It’s a pure educational resource. In the training, we introduced deep learning transformer architectures and how these enable the extraordinary capabilities of state-of-the-art LLMs. And it isn’t just theory; my hands-on code demos, which feature the hugging face and PyTorch Lightning Python libraries, guide you through the entire life cycle of LLM development, from training to real-world deployment. Check out my generative AI with large language models, hands-on training today on YouTube. We’ve got a link for you in the show notes.
00:43:28
Yeah. Fantastic. Appreciate that. At the end of the episode, we’ll get … You’re a regular listener, so you know that at the end of the episode, my final question is always how people should get in touch with you, so you may get bombarded through that as well. Nice. All right. So given these large language models that we could be using, given the applications that you’ve discussed and that people can get from their own discussions with a large language model of their choice, I’ve alluded here, I’ve made the assumption that with something like the Pro AI Studio SDK that is in early access now and will be fully available mid-year 2025, I’ve assumed that we’re talking about weeks or months timeline to developing an enterprise AI application. Fill me in on whether I was right with that assumption. What are the real timelines here? If I am trying to do things on my own and trying to build all the glue and security that I would need for an enterprise application on my own, as opposed to relying on something like the pro AI studio to do that, what’s the difference in timeline?
Shirish Gupta: 00:44:36
Great question, Jon. So in our estimation, a typical to take a typical app from POC to in production for running on the AI PC NPU could take you about six months. And this includes all of the discovery and iteration with identifying the model and the device that you would run on for your use case. It would involve all of the development to enable the runtime compatibility and all the model operations on the device. And then finally, you come to deployment, which will again be a very manual process of deploying it onto a fleet of apps. So up to … And I call that point time to initial value, So from build or discover to build to deploy. That’s about six months in our estimation for a typical app, nothing too complex like a RAG chatbot, for instance.
00:45:37
With Dell Pro AI Studio, you can shrink that down to under six weeks in our estimation. So that’s about a 75% reduction in time. The time that you’re going to still spend on is the time that you should spend on, which is picking the use case, making sure the solution output’s tuned for your users, is delivering the accuracy that you need and validating the outputs. So that’s where we want developers to focus their time, which is really what they own. In all the parts that are painful that they just have to do to get to the point from point A to point B, but they don’t need to. That’s what we’re automating for them with Dell Pro AI Studio. So it’s all of the stuff that they shouldn’t have to do. We’re working to solve that. And so again, I repeat for that typical app to go from discovery to deployment, time to initial value, we can reduce that from six months to six weeks.
Jon Krohn: 00:46:38
That’s wild.
Shirish Gupta: 00:46:38
With Dell AI Studio.
Jon Krohn: 00:46:39
I believe that. With the right tooling, it’s definitely possible and it’s nice to think how that could make processes repeatable. It just makes it so much easier for an organization or for an individual to pick up an SDK like this and be able to iterate, be able to develop more applications. It’s really exciting. That 75% number is huge. All right. So we understand the benefits today, I think now of a framework like this. What do you think these kinds of tools like Dell Pro AI Studio that let you rapidly go from prototype to enterprise AI application as well as technologies like NPUs running on AI PCs, which again, with Windows being the defacto standard across most of the commercial and industrial world, what does that mean for the future? How are AI development workflows and applications going to change in the coming years?
Shirish Gupta: 00:47:37
Great question, Jon. I think if I look to the future just from a standpoint, I’ll touch on what do I see the AI workloads of the future on device. I think you’ll see a lot more agenetic behavior on the device. And what I mean by that is you’re going to have the AI have some agency to take action on your behalf, and it really comes down to the parameters you or your enterprise define. But I think we’re not there yet, but that is the next wave where I see AI assistants working in unison or in collaboration with others to get things done for you and come back to you to ask for your input based on the parameters that you’ve set.
00:48:27
So again, to me, that’s what’s going to enable true productivity gains when it’s automation within your workflow for the average end user. Whether it’s personal uses … You might have your own travel agent that you set the parameters and say, “Hey, go buy my tickets for this concert,” or something like that. And you’ve given it enough parameters so that it can take action on your part. You tell it, you want the best deals, you want it on these dates, and it goes and scours everything, finds it for you and comes back to you and say, “Hey, this is what I found. Is this good? Do you want to pay?” And then if you’ve authorized it to pay, just say yes. And it goes and does it.
00:49:16
Now, that’s not available today. I think models have to evolve further. You need to get to reasoning models that get small enough that you can run them locally. That might take some time. But I do think that is the future. The other thing I see is hybrid compute. Where you may not be completely black and white. You may not just work only on your device or only on your private cloud tenant. You may seamlessly use compute locally where it makes sense. And then the cloud steps in as soon as something that requires more compute than your device can offer, than your PC can offer. That’s going to be more ubiquitous in the future. So I see a lot of that hybrid workload orchestration happening in the future as well.
00:50:15
I also think the Dell Pro AI studio is going to keep going to keep abreast of all of these developments. So not only will we continue to expand the model set that we support to keep enabling more use cases, as models become more performant and NPUs become more capable, we’ll expand models, we’ll expand silicon-supported architectures. Our support will widen for architectures, and then we’re going to have more automation in the future. Whatever today starts with a manual process, you can think anything that’s manual, if it can be automated, it will be automated, So that’s what I definitely see happening in the future, and we’ll keep evolving Dell Pro AI Studio to keep up with that evolution.
Jon Krohn: 00:51:09
Fantastic. That is a really cool vision for the future. So before we wrap, I’m broadly aware of a broader AI ecosystem at Dell, which is something called the Dell AI Factory. So how does the Dell Pro AI studio, this SDK that we’ve been focused on most of this episode, how does that fit into the broader AI ecosystem? And particularly for our listeners, what are the most valuable takeaways for them from this broader Dell AI Factory ecosystem?
Shirish Gupta: 00:51:45
Great question. So this absolutely fits into the Dell AI Factory. Let me touch on the Dell AI Factory construct itself for a moment here, for those that may not be aware. So the Dell AI Factory came to life about a year ago. It was announced at Dell Tech World in 2024. And it was a really important announcement and capability for our customers to take Dell infrastructure, whether it’s client devices, edge devices, data center, compute, compute storage, networking fabric, what have you, but take all of that infrastructure, combine that with an open ecosystem of tools, models, frameworks, essentially the software layer, and combine that with services. Because not everyone wants to just take the hardware and the software and build it themselves. Now you can .but for those who don’t want to DIY, you have Dell Professional Services to help you along your journey. Either do it build completely for you or consult with you on the parts that you need help with.
00:53:08
So these three components of the Dell AI factory, the infrastructure, the open ecosystem, and the services together enable customers who have data ideas and they want to come to outcomes and use cases and productivity or efficiency for their company or their employees. To go from that left side, which is ideas and data to the right side, which is outcomes and value. You use these three components of the Dell AI Factory to go from there, from A to B. So it’s everything. How do you manage your data? How do you store your data? How do you organize your data? How do you use the best set of models and tools and tool chains and frameworks on the right set of hardware coupled with services as needed to get to those outcomes? That’s the Dell AI Factory construct. Dell Pro AI studio fits into this because it is truly our client story of the Dell AI Factory. So you couple client devices, which are the AI PCs with the open ecosystem, which is the Dell Pro AI Studio itself, which has open permissive models, the tools, the frameworks, and other recipes. And then we will also be standing up Dell Professional services to enable customers to get there where they need help.
Jon Krohn: 00:54:52
Fantastic.
Shirish Gupta: 00:54:52
That’s how Dell Pro AI Studio fits.
Jon Krohn: 00:54:54
Yeah. And I think a key thing here to note too is before I ever started … Because it’s now been about a year since I started doing television ads actually for Dell AI Factory. I didn’t know that Dell offered services. And so I think of Dell as a hardware company is making PCs, as making servers, and so this professional services angle that the Dell AI factory encapsulates, yeah, I think it’s something important to highlight for people who might not be aware. Dell is an absolutely enormous company and that it has this professional services arm. Yeah. It’s a big business as I’ve discovered in the past year. Yeah. Cool that the Dell Pro AI studio, which will particularly appeal to our listeners out there who are developing AI applications, but it’s nice to know for them as well as all of our listeners, I guess, that if they want to be churning out AI applications out of the factory, you have all of the services needs covered for them as well. Yeah. So something that I didn’t know.
Shirish Gupta: 00:56:06
Absolutely. You made me remember one more thought that I would share with the listeners. If you juxtapose our Dell AI Factory and overlay it along the journey of an enterprise developer or IT pro, the flow always starts with your data scientist. That’s where they do all of the training model fine-tuning and take the base PyTorch, the base versions of the model. The AI engineers come in and apply two chains to go from the general base to maybe more of targeted model for the specific run times. And then you have the software developers that come into the picture where they now take those targeted models, combine or integrate into apps and create those solutions that can be consumed by end users. That’s a very simplified version of a flow for AI.
00:57:12
The way I see it in every customer conversation, this becomes pretty relevant. How do I start? We’ll always tell them, take your smartest people. It maybe your data scientists. Put them in the room with the people where you need that improvement. So let’s say if you’re trying to improve healthcare delivery in an ER, take your data scientists, let them sit and coexist with your ER staff for a month or however long you can afford it, and before they start killing each other. But the idea is let them coexist, let them really study the problem, and then you put the most powerful workstations in the hands of those data scientists and tell them, okay, go figure out a solution to this, Go do your POCs. You now know what the problems are, you know how they work, what the flow is, go figure out the solution. And so that’s what I tell everyone is to take the Dell Pro Max workstations, put them in the hands of your smartest people, let them study the problem, do the POCs. Once they understand what their use case needs, you can go scale that on server infrastructure, like you need to do some serious training or fine-tuning. You want to really prove out your solution. You may need infrastructure for that.
00:58:44
And then once you have a solution ready and deployed, then you have the opportunity to assess, okay, now that it’s in production, do I really need my H100s B200s or B300s to really run that? Or can I actually use a smaller model, use AI PCs, and if that’s the case, let’s offload that workload to my fleet of AI PCs and let all the users consume them right there. This is the flow that we are encouraging our customers to take is start with the high-powered workstations, scale to the servers and deploy to the endpoints the AI PCs. So it’s almost like the client devices bookend our infrastructure on either side as part of the Dell AI Factory.
00:59:44
And I do want to share that example, which I told you from healthcare earlier, which I said was really cool, but the first part I said, I’ll hold back for later. This is one customer that has actually done a pretty impressive job in doing exactly what I just said. They took that data scientists, they put workstations in their hands, they went to the ER and they studied the problem. They created a custom vision transformer model using their own radiology images from the ground up. They trained it on their own radiology images and now they were able to take new patients, get those radiology images, compare them to their training dataset and get an initial diagnosis for the physician to look at. And they started seeing that they’re getting equal to physician level accuracy in the model predicted diagnosis, which is pretty impressive if you think about it.
01:00:41
And then this is the part which I said was super cool, right, that they use this custom vision transformer. And they’re very keen to bring this down to the AI PC. Because today all of this lives in a HIPAA-compliant data center that they can replicate across the globe, and they want their entire hospital system and doctors in mobile areas that are all over the globe to have this capability. And the best way to do this is if I can do this on a PC. This is the part where I had said earlier, they took that diagnosis and converted it into an auto-generated report as well, so that’s the two-step process that I talked about.
01:01:20
Yeah. So just to wrap up, I think making it real with Dell Pro AI Studio is really what we’re trying to do for our customers. And as you saw in that example, it’s all about taking your use case, how you are enabling value for your business. Again, I hark back to the four reasons why you should consider potentially bringing the workload and running it on an AI PC. Accelerated, individualized, private, and cost-effective. If any of those apply to your use case, once you’ve stood it up and it’s in production, you should consider whether it’s right for you to offload it to the PC using Dell Pro AI Studio.
Jon Krohn: 01:02:14
Yeah. Makes sense. Fantastic. All right. So you’ve also got another great thing for our listeners in addition to that consideration is that you have the beta program for this AI studio that our listeners can get into or can apply to get into. So I’ll have that link for them in the show notes. Before I let my guests go as a regular listener, you’ll know that I always ask for a book recommendation. So do you have something for us?
Shirish Gupta: 01:02:46
It’s not data science related, but it is a book that I am reading right now. I’m terrible at author names, but I think you’ll find this. It’s called Manifest and it’s all about manifesting the future that you envision for your life. And actually there’s another one I’m reading concurrently. I’ll give you two book recommendations. Another one that I really like is 5 Types of Wealth by Sahil Bloom. They’re very similar. Reading them concurrently. But it’s all about manifesting the future you want for yourself. And Sahil Bloom’s book is all about … It’s not just about one type of wealth, there are actually five types of wealth in your life. So if either of those books interest you, they’re both highly recommended.
Jon Krohn: 01:03:42
Nice. Fantastic. Shirish, thank you. And as I mentioned would happen earlier in the episode, how can people follow you after this episode?
Shirish Gupta: 01:03:52
Yeah. I am on LinkedIn. So if you do want to follow me, I’m on LinkedIn. It’s linkedin.com/shirish29 is my handle, but you can search me up. I don’t think there’s another Shirish Gupta at Dell right now.
Jon Krohn: 01:04:08
No. We’ll have a direct link in the show notes as well.
Shirish Gupta: 01:04:11
Perfect.
Jon Krohn: 01:04:12
Yeah. Fantastic. All right, Shirish, thank you for coming on the show and opening our minds to the possibilities with having AI PCs, something that is actually really important because of how widespread Windows operating system is across the world. And so when you’re thinking about deploying into that very common environment, it makes a lot of sense to me to be using a tool like Dell Pro AI Studio that allows you to accelerate and have lots of compatibility and scalability, enterprise readiness. Very cool. Thank you for filling us in today, Shirish, and maybe we’ll catch up with you again in the future.
Shirish Gupta: 01:04:49
Sounds good. Thank you very much for having me today. It’s been a pleasure talking to you.
Jon Krohn: 01:04:58
Well, I learned a ton in that episode and had my attention drawn to the importance of considering deployments of AI applications to edge devices running the globally ubiquitous Windows operating system. In today’s episode, Shirish covered how neural processing units, NPUs are specialized chips designed specifically for matrix math operations, making them highly efficient for AI inference workloads. He also talked about how NPUs are optimized for running inference rather than training ideal for deploying AI to end user devices with better battery life. He talked about the AI PC advantage and how it can be remembered with his mnemonic, A accelerated low latency, I individualized learns to your style, P private data stays local, and C cost-effective, no cloud fees.
01:05:44
He talked about how current NPUs can effectively run seven to eight billion parameter LLMs at 15 to 20 tokens per second, making local inference practical for many applications. How the Dell Pro AI Studio may reduce AI application development and deployment time from six months to six weeks, a 75% reduction by automating model discovery, compatibility, and lifecycle management. And he provided lots of real world PC AI applications including manufacturing defect detection, insurance damage assessment, real-time translation for first responders and medical image diagnostics.
01:06:17
As always, you can get all the show notes including the transcript for this episode, the video recording, any materials mentioned on the show, the URLs for Shirish’s social media profiles, as well as my own at www.superdatascience.com/877. And if you’d like to engage with me in person as opposed to just through social media next month, you can meet me in real life at the Open Data Science Conference, ODSC East, which is running from May 13th to 15th in Boston. I’ll be hosting the keynote sessions and with the extraordinary instructor, Ed Donner, who’s also a longtime friend and colleague of mine. We’ll be delivering a four-hour hands-on training in Python to demonstrate how you can design, train, and deploy cutting-edge multi-agent AI systems for real-life applications. It should be exciting indeed.
01:07:03
All right. Thanks of course, everyone on the Super Data Science podcast team, our podcast manager, Sonja Brajovic, media editor, Mario Pombo, our partnerships manager, Natalie Ziajski, researcher Serg Masís, writer Dr. Zara Karschay, and our founder Kirill Eremenko. We can never forget him. Thanks to all of them for producing another illuminating episode for us today. For enabling that super team to create this free podcast for you we are deeply grateful to our sponsors. You can support the show by checking out our sponsors links, which are in the show notes. And if you want to sponsor the show, you can find out how. Just head to jonkrohn.com/podcast. Otherwise, help us out by sharing the show with people who would love to learn about edge AI applications. Review the show on your favorite podcasting app or on YouTube. Subscribe, obviously. Edit videos into shorts if you want to. But most importantly, just keep on tuning in. I’m so grateful to have you listening and hope I can continue to make episodes you love for years and years to come. Until next time, keep on rocking it out there, and I’m looking forward to enjoying another round of the SuperDataScience Podcast with you very soon.
