Jon Krohn: 00:05
This is episode number 736 with Jan Zawadzki CTO of Certif.AI.
00:19
Welcome back to the Super Data Science podcast. Today’s episode was filmed live in person at the Merantix AI campus in Berlin with Jan Zawadzki, who’s spearheading the development of safer, more reliable AI systems through formal independent certification and testing. Jan is CTO and co-managing director of Certif.AI, spelt Certif.AI, Certif.AI, a startup that is an early mover in the fast developing AI certification ecosystem. He was previously the Head of AI at CARIAD, the software development subsidiary of Volkswagen, where he grew the team from scratch to over 50 engineers. Today’s episode is accessible to anyone. In it, Jan details how certifying your AI models could give you an advantage over your competitors and why in the EU at least it may be essential for you to do very soon. All right, let’s jump right into our conversation.
01:13
Jan, welcome to the Super Data Science podcast. You’re the Co-MD, Co-managing director and CTO of Certif.AI, C-E-R-T-I-F dot A-I. And it’s right there in your name. You guys certify AI companies. Explain to me what that means.
Jan Zawadzki: 01:32
For sure. So first of all, thank you for having me. It’s a pleasure being here at the Super Data Science podcast. What do we do? So what we do is we test AI based applications and there’s something called the EU AI Act coming in, and it is guiding the development of high risk AI applications in the future. It hasn’t been passed yet, but once it will be passed, it will actually require self-auditing or auditing from an external application for your high risk application. So anything in HR tech, for instance, if you use AI to automatically screen applicants and also determine which applicant would be admitted as the next process step or not, that could contain some sort of bias, could violate some human rights and those systems will have to be checked going forward.
Jon Krohn: 02:19
You’re definitely speaking to my Nebula.io application from my own startup there. Yeah.
Jan Zawadzki: 02:24
Where did I get that from? For sure.
Jon Krohn: 02:26
So yeah, so the interesting idea here is you’re not… So if somebody described this to me that you’re the certification body, I might think of that as a government organization, but you’re not. You’re a private company.
Jan Zawadzki: 02:39
Absolutely. So how this works is first a law gets passed and then the law basically creates a request for standardization, and then standards are developed. So in the automotive industry, we have functional safety, the ISO 26262, and other standards. And then against these standards, certain products have to be tested and certified. Now, there is not a single standard yet that implements the EU AI Act requirements and no standard against we can certify yet, but it will be basically developed in the future. And what we do in the meantime, we do evaluations, we provide evaluation seals, we help companies with improving and testing the AI-based applications, making them more reliable in the end, helping them, making them do more of what they’re supposed to do.
Jon Krohn: 03:26
Nice. Yeah. So let’s dig into a concrete example. I think you have a lot of experience actually yourself from before even this company-
Jan Zawadzki: 03:33
That’s right.
Jon Krohn: 03:33
… with self-driving cars. Maybe we can talk about that application area.
Jan Zawadzki: 03:36
Yeah, absolutely. So what I always say is in the automotive industry, we were already way ahead in terms of safety critical AI development to I would say many other industries. And in the automotive industry, you don’t need a government agency to tell you, hey, you have to be sure that your AI system does what it’s supposed to do. We know that we do not want… Anything that can go wrong in autonomously driving cars, it can kill people, literally. And that is an absolute nightmare. So you want to do everything in your best interests and powers to actually prevent that. So I’ve been working for five and a half years in VW Group the last three years as the head of AI at CARIAD. And what we’ve developed there are processes, methods, and tools for implementing automated driving functions in a safe and automotive compliant way.
04:24
So everything from creating a process that is a mix between a waterfall process model, how we have it in automotive to the more agile iterative data science development process to additional requirements that you have in automotive like traceability and operational design domain. And those things I think are really applicable also to other industries. And that’s what I’m trying to do now with Certif.AI, bring what we did to kind of make automated driving safer to healthcare, manufacturing and any other industry.
Jon Krohn: 04:51
Nice. Really quickly, what was that you were saying, at Volkswagen Group, which obviously everyone knows Volkswagen Group, but you said something like CARIAD.
Jan Zawadzki: 04:57
Exactly. That’s the Central Software Development Company of Volkswagen. So it’s about 6,000 people and they developed software for VW Group vehicles, all of the software. And they’re also partnering with Bosch together in an Automated Driving Alliance for advanced driver assistance functions. Exactly.
Jon Krohn: 05:19
Very cool. All right, so. Yeah, so nice. So you’re taking this know-how that you developed at CARIAD-
Jan Zawadzki: 05:24
Exactly.
Jon Krohn: 05:25
… and now applying it. And so I guess in any industry, obviously it isn’t just self-driving cars, you gave lots of examples there, healthcare, manufacturing, even someone like me in automating recruiting might want to take advantage of your kinds of certifications. Why would we want to do that? What are the advantages to us?
Jan Zawadzki: 05:43
For sure. So first, once the EU AI Act is up, is actually ratified, then it will help with conformity assessments, just with complying with legal laws. So whoever wants to deploy an AI-based system that falls into the high-risk category, has to get it certified. You can do self-certification, but you can also do an external audit from companies like us and it helps you with two things then. One, it helps you with consumer trust. So I mean, next to the regulatory requirements, consumer trust, it’s a signal into the market like, hey, this is organic food signal, or this is a nutrient score or something we have here in Europe to kind of give feedback of how healthy certain food items are. It simply helps you with saying, “Hey, this AI application was developed according to state-of-the-art, and you do not need to be too worried about this.” It’s a qualitative AI-based product.
06:35
And the second thing it helps you with, it helps you with liability because an example of automated driving, some person in any OEM has to sign eventually a paper saying this automated driving function was developed according to state-of-the-art. And then if something actually goes wrong, a judge could actually sue that person or ask that person, “Hey, show me, give me proof that you’ve done everything in your powers to safeguard this application.” And then an external assessment, external audit from us helps.
Jon Krohn: 07:04
Nice. And you used a word there are OEM, that’s original equipment manufacturer.
Jan Zawadzki: 07:08
Exactly. Any vehicle manufacturer like Volkswagen, Porsche, Audi, Renault, Citroen.
Jon Krohn: 07:13
Yeah. Or I mean, it could be in other industries like Apple is an OEM for a computer hardware.
Jan Zawadzki: 07:18
Absolutely. Absolutely. Samsung, for sure.
Jon Krohn: 07:20
Maybe self-driving cars from Apple someday too. Awesome. So going forward, what are the kinds of places that you see real opportunity with certification, with making AI maybe more broadly accessible and how certification can play a role in that? Prior to us recording, you were talking a bit about the OpenAI DevDay, which happened not recently at the time of recording.
Jan Zawadzki: 07:49
For sure.
Jon Krohn: 07:49
Which happened very recently at time of recording.
Jan Zawadzki: 07:51
Absolutely, absolutely. So let me actually take a step back. So what we sell here or what we’re trying to do or what motivates really me. I’m motivated by improving the engineering quality of AI-based systems. So I’ve been in the AI community for I think six or seven years now. I’ve been saying for years, “Oh wow, the AI breakthrough is going to come and we’ll have so many more AI-based applications.” And then just six or seven years later I look around and I don’t think we are as an AI community as far or we are there where I thought we would be by now, whether it be automated driving or just the application of AI in different industries. And I wonder what is the cause of this? And I believe not being able to trust the AI system or suboptimal AI engineering is actually one of the reasons.
08:39
So 20 years ago when the principle of test-driven design started to take place, it dramatically improved the quality of general software applications. I don’t think we really are there yet in AI community. I think we still have many POCs, many Jupyter Notebooks, much hacking where you can easily go from zero to one, but actually making it a production-ready system is still very hard. So what we’re trying to do is improve the overall quality of AI-based systems. I don’t want to test your AI-based applications if I know it’s a good engineering quality. If it does what it’s supposed to do, nobody really needs to test it. But truth is, these tests can help improve the quality of your product. And with a good qualitative AI product, you can create even better customer experiences, hopefully driving adoption of your AI-based system in the market. So this is kind of what motivates me. And then-
Jon Krohn: 09:28
I just want to dig into that for a second a little bit more because it’s something that isn’t automatically intuitive to me and so maybe isn’t for some of our listeners as well. Why is getting certified going to improve… If my company, Nebula.io, this automated recruitment platform, we come to you and we say we want to be certified, we want people to know that they’re going to get high-quality matches in our system, that there’s not going to be bias in there, and so is it because it’s kind of like an iterative process?
Jan Zawadzki: 09:59
We sell a methodology and a tool basically. What we help? We have a methodology of how you evaluate the risk. So you just mentioned the fairness, [inaudible 00:10:09] mentioned. Neural networks are prone to discriminate basically on data that they’ve seen in the past. And this is something that we do not want. So what we do is then we have the methodology that we look at this risk, fairness, we look at what could go wrong. Hey, we could discriminate against certain groups based on their skin color, ethnicity, religion, sexual orientation, whatever. And then you define what are KPIs that I have to meet. So my KPI would be that I do not discriminate based on any gender. And then you define measures of how can you mitigate the risk that you created here. And by the way, this methodology is just well established across industries anywhere. In the automotive industry, you always have a risk-based assessment.
10:50
There will never be a 0% risk, no system will ever be without risk. But by this approach, by defining measures to mitigate those predefined risks, we can create an acceptable risk that we have at the end. And then when we come to the measures, we always look at the data. What data was the trained on? Was it just trained on white men or is it actually a mixed data set? What model was used? Do you use decision tree that’s more transparent to use on your network? What can you do to estimate uncertainty or make the model more robust?
11:20
Then you look and how is it integrated into the overall product? Could something go wrong by entering the data or could something go wrong when you exert the data? And then lastly, we look at the deployment. How does the model’s behavior change over time? How do you monitor output and how do you make sure that there’s no concept drift or no data drift involved? And then if you have these different mitigation measures, you make a bottom-up argument again and then this is how we could help you. We can help you, all right, these are potentially things that you’re not doing yet, but this would significantly mitigate the risks of deploying an unfair system.
Jon Krohn: 11:55
Nice. Okay, now I understand. So it’s because you have methodologies, you have standards that I will have to meet, and so it will be a consultative process. We come to you in the beginning and we say, “I’d like to be certified by you.” And you say, “Well, that’s great. What are you doing?” And I explain that to you. Then you say, “Okay, you’re probably going to want to have all of these different kinds of criteria that you meet.” And then so I go off and I either prove to you that I do meet those criteria or I work on my AI system, my engineering solution to ensure that I do tick that box.
Jan Zawadzki: 12:31
Yeah, absolutely. So we have to do a small distinction. We are not allowed to do consulting because you can’t consult and test all at the same time because then you would on the one hand just tell people what to do and then you certify your own solution. That’s not really, we have to have a separation of concerns, but what we can do is we definitely can give you a status. This is where you currently are in the audit process. These are best practices. And then it’s up to you to implement those best practices to improve the quality of the AI system. And then in the end, what I think also really motivates me, if we can get this AI made in Europe quality seal, if we can establish that worldwide, I think that’s the way for us to go in Europe.
Jon Krohn: 13:09
Nice. And yeah. So we are recording today in Berlin at the Merantix AI campus, but so obviously our listeners in Europe know that they can come to you. Do you work with partners outside of Europe as well?
Jan Zawadzki: 13:22
We are a global startup, so we are open to discussing with anyone also internationally. And like I said, any US company that wants to deploy their product in the European Union, once the EU AI Act is live and it’s a high-risk category, we’ll have to go through some sort of testing. So yeah.
Jon Krohn: 13:38
Awesome. Thank you so much, Jan. This has been super informative. It has opened my eyes to an entire different part of this AI industry. I guess I vaguely kind of was aware that certification was possible, but in terms of any of the details of how that works, I had no idea. And now I know. Thank you, Jan.
Jan Zawadzki: 13:55
For sure. Absolutely. It was a pleasure to be here.
Jon Krohn: 13:58
Hope you learned a lot from today’s informative episode with Jan Zawadzki. In it, Jan covered how the forthcoming EU AI Act will require AI models in high-risk categories such as self-driving cars to be certified. And how whether you are required by law to be certified or not getting certified could provide advantages such as increased quality and trust as well as decreased liability.
14:20
All right. That’s it for today’s episode. If you enjoyed it, consider supporting this show by sharing, reviewing, or subscribing, but most importantly, just keep on listening. And until next time, keep on rocking it out there. And I’m looking forward to enjoying another round of the Super Data Science podcast with you very soon.
