Summary
Software systems power much of the modern world. For applications that impact the safety and well-being of people there is an extra set of precautions that need to be addressed before deploying to production. If machine learning and AI are part of that application then there is a greater need to validate the proper functionality of the models. In this episode Erez Kaminski shares the work that he is doing at Ketryx to make that validation easier to implement and incorporate into the ongoing maintenance of software and machine learning products.
Announcements
Parting Question
Software systems power much of the modern world. For applications that impact the safety and well-being of people there is an extra set of precautions that need to be addressed before deploying to production. If machine learning and AI are part of that application then there is a greater need to validate the proper functionality of the models. In this episode Erez Kaminski shares the work that he is doing at Ketryx to make that validation easier to implement and incorporate into the ongoing maintenance of software and machine learning products.
Announcements
- Hello and welcome to the Machine Learning Podcast, the podcast about machine learning and how to bring it from idea to delivery.
- Your host is Tobias Macey and today I'm interviewing Erez Kaminski about using machine learning in safety critical and highly regulated medical applications
- Introduction
- How did you get involved in machine learning?
- Can you start by describing some of the regulatory burdens placed on ML teams who are building solutions for medical applications?
- How do these requirements impact the development and validation processes of model design and development?
- What are some examples of the procedural and record-keeping aspects of the machine learning workflow that are required for FDA compliance?
- What are the opportunities for automating pieces of that overhead?
- Can you describe what you are doing at Ketryx to streamline the development/training/deployment of ML/AI applications for medical use cases?
- What are the ideas/assumptions that you had at the start of Ketryx that have been challenged/updated as you work with customers?
- What are the most interesting, innovative, or unexpected ways that you have seen ML used in medical applications?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on Ketryx?
- When is Ketryx the wrong choice?
- What do you have planned for the future of Ketryx?
Parting Question
- From your perspective, what is the biggest barrier to adoption of machine learning today?
- Thank you for listening! Don't forget to check out our other shows. The Data Engineering Podcast covers the latest on modern data management. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email hosts@themachinelearningpodcast.com) with your story.
- To help other people find the show please leave a review on iTunes and tell your friends and co-workers.
- Ketryx
- Wolfram Alpha
- Mathematica
- Tensorflow
- SBOM == Software Bill Of Materials
- Air-gapped Systems
- AlexNet
- Shapley Values
- SHAP
- Bayesian Statistics
- Causal Modeling
- Prophet
- FDA Principles Of Software Validation
[00:00:10]
Unknown:
Hello, and welcome to the Machine Learning podcast. The podcast about going from idea to delivery with machine learning.
[00:00:20] Unknown:
Your host is Tobias Massey. And today I'm interviewing Erez Kaminsky about using machine learning and safety critical and highly regulated medical applications. So, Erez, can you start by introducing yourself?
[00:00:30] Unknown:
Yeah. Thanks for having me, Tobias. It's a real pleasure. So my name is Erez. From Boston, I, I'm the CEO of a company called KETRIX that helps regulated software teams perform CICD and just develop fast and stay compliant and make safe and reliable applications. I spent, basically most of my adult life working on high reliability applications and the nuclear engineering domain and medical devices and pharmaceuticals. And kind of most recently before doing this, I I was leading AI for Amgen, which is the largest biotech company in the world, as a medical device division. So I've been thinking for a long time on how to do this and then kind of classical developer.
I worked at a big company developing this very complicated thing and said, you know, what if I had better tools to do this with? Are there other people who feel the same way and left to to build the tooling company, really? And Ketrix is a software development tooling company
[00:01:29] Unknown:
for highly regulated industries, including kind of regulated AI and so on. Before we get too much into the bulk of what we wanna talk about today, I'm also interested in how you first got started working in the machine learning space.
[00:01:42] Unknown:
Yeah. So really my background is I'm I'm a I'm a physicist, and and I worked in in physics and helped to build numerical simulation applications. That's kind of where I always say that I'm not a software engineer. I'm just a person who knows how to build fancy calculators, in low level languages or high level languages. And so that started an interest, right, because numerical simulation, you get really into computing, into creation. The time is, like, 2014, 2015, 2016. So AlexNet made a bit a big impact. You hear a lot of it. And even in the research community I was in, there was starting to be conversations of can we accelerate certain numerical simulations with, neural networks. But the real jump for me is, I went to work for a company called Wolfram Research that develops Mathematica and Wolfram Alpha, So a lot of both kind of more classical expert system work, but also more kind of modern machine learning work. And there, I got exposed really to everything that was going on there as part of their machine learning group and and helping to support them developing products and, you know, we do a lot of different things even even back in the day, which is not too long ago, but in machine learning time, you know, even 5, 6 years ago is is eons ago. So that's how I first got involved, and then I've been around the Boston machine learning community for a long time and helping different government private establishments do machine learning, get trained on machine learning, and then did a lot of work at MIT around machine learning specifically for health care and biology.
[00:03:12] Unknown:
And the idea of safety critical, highly regulated systems is a massive area of active research and development in the software community, and I can only imagine at the additional levels of complexity that are involved when bringing ML into the equation. But before we get too much into the specifics, I'm interested in getting a description of what some of the regulatory burdens are that are that exist for ML teams who are building solutions for medical applications.
[00:03:43] Unknown:
Yeah. And and I think, you know, I'll start more in general than that even, but we'll get into medical. I think that there's basically 2 type of things in the world. Right? There's things that you could just build and no 1, focuses on, no 1 cares about, there's no public pressure about them, they're not society critical, and there are things that are. And generally speaking, society critical or or just critical things are regulated. There's a a wide range of what it means to be critical. Are you mission critical? Are you safety critical? And with that, kind of, the burden of regulation and the burden of evidence goes up.
But, generally, everybody's asking for the same type of thing, which is if you are claiming that this system or application you built can help people in a moment of need with something they need to rely on to make sure their finances are okay or make sure that they are safe from harm, there's usually a regulator in the world that regulates that for every jurisdiction, and we see that's just growing, both kind of the amount of regulation but also the scope of what is being regulated, and a lot of the discussions over, I think, regulated AI touch that. But fundamentally, all regulations that deal with product, there's regulations that deal with your company, and that's more, you know, are you compliant with SOC 2? Do you have proper privacy controls overall?
But then there's regulations that deal with your product, and and that's our focus is basically how do you know that a system is doing what you said it did or what you said it was going to do. And at the end of the day, all the regulator is asking people to do is have a long document that describes for a less educated observer, right, someone who's just coming to audit you or inspect your facility, that explains what your system does, in a very robust way, and the robustness really depends on how dangerous your system is, like very dangerous systems more, less dangerous systems less. And then have evidence, that those things are being are being met, Those requirements are met by the system for every version of the system.
And while that sounds very easy, just kind of share with us what you do and then show us that you do those things. If you've ever built any web system or any complicated software system, you realize that's really, really challenging to do and very, very difficult to do and takes an immense amount of time. Now machine learning introduces into that problem the question of what does it mean for something to do what I said it does. Right? Like, it's a statistical model. It will always have certain parameters in which it operates correctly in and certain parameters in which it doesn't operate correctly, and then you need to try and assess. And and in the same way that you validate your software system to meet certain requirements, validate your machine learning system or subsystem, as the FDA calls it, to meet certain performance parameters and be consistently meeting them. And I think that's a very challenging environment for people who grew up building mobile apps, ad tech, even Fintech, to try and do. Like, it's so hard to do that. It's so hard to make something reliable and meet certain requirements, and machine learning just adds some complexity. And I think a lot of that complexity is about the subject matter expertise more than the regulation, frankly.
It's just like, what does that even mean to show that a neural network does what you said it does? What is what do you say it does? What do you think it does? You know? Absolutely. Yeah. Model validation is a hard enough problem on its own without having to do it in a provably correct manner.
[00:07:18] Unknown:
And before we get too much even into that model development, model validation, model design question, in terms of security and safety in the software realm. There's also the whole question of the software supply chain and being able to manage and validate the dependencies that go into the the system. And then with ML, there's also the questions of the accuracy and reliability and the awareness of and correction for bias within that source dataset. And I'm wondering what are some of the ways that the combination of software supply chain along with data supply chain come together to compound the complexity that you're trying to address?
[00:07:59] Unknown:
That's such a good question, Tobias, and and it's actually the the first place where we started to work on on our company and and the product is how do we help people integrate open source software and, you know, kind of third party software into the regulated applications because it's it's complicated and people didn't use to develop with that much open source in the healthcare community, medical device community until kind of recently, and that's caused a lot of issues, right, because if you developed a medical device like 20 years ago, even a very advanced medical device, it would mostly have embedded software and a a very small amount of dependencies.
And now, you know, you wanna spin up, some image classifier with TensorFlow. There's already a lot of dependencies going on in there. Could be, you know, in the many thousands easily. And I think there was it's it's it's not an impossible task. It's a task of of understanding what your system does and how these different dependencies affect the safety of the system and the connection to harm, and then being able in a sophisticated manner to triage that based on both the security aspect aspect of that dependency, but also how the dependency works inside your validated application. And I think, you know, that's very challenging for the men mentality of software teams where, you know, you just kinda put whatever dependency you want in, whatever makes it work, and let's move forward.
But here, we really need to understand what we're doing, so that's why we we sometimes need to stop and think about it and make sure that, you know, we're not introducing a dependency that, would hit our performance or would get our end users exposed to unnecessary harm. And we have a big offering that helps do that, manage that softer bill of material and that softer supply chain because it's a common attack vector into all kinds of products. Right? And then there's recent study that says that kind of the average cost of a cybersecurity hack in health care is about $11, 000, 000 for a company, and that's just gonna go up and up and up.
So it's something that I think people don't think about enough, what I'm introducing into my application, and they need to do it more. And we've developed a lot of both kind of best practices and software to think of how to do that and how to connect back from the things your system can do poorly to harm another person to what you're using to develop it. And most of it, like you can imagine, is just saying, you know, have we thought about this? Do we know where we're using this? And is the thing we're using it in high risk or low risk? If it's low risk, maybe we need less attention to it. If it's high risk, we need more, and just to have some kind of procedure and thought process. And and I think the new guidance from both FDA and Congress, which is part of the patch act that passed in December of last year, has a huge focus on that. Right? If you build a medical application, particular medical device in the United States by October 1st this year, that device has to have a very robust software supply chain methodology.
[00:10:57] Unknown:
Another aspect of the security question before even touching on the complexity of ML is also the operating environment that you're dealing with, which the software bill of materials is 1 aspect, but then there are also questions around system hardening. Does it need to run-in an air gapped environment? Who has access to these systems? And I'm curious, again, how that factors in with these regulated and safety critical systems.
[00:11:23] Unknown:
It's a huge factor. Right? Because, again, it's just another vector that introduces risk. Like, let me give you an example. It's a pretty interesting recall by a large medical device company recently. They developed insulin pumps. That insulin pump, had a vulnerability exposed to it that would allow a user to take control of certain aspects of the pump. As far as I recall, it wasn't actually something super dangerous, but, you know, if I if if, you know, my child had an insulin pump, the very idea that someone can connect to it makes me extremely uncomfortable and and makes me not wanna do it at all or change your brand. But because they had good risk controls, and this is where you do see big companies doing a good job with these processes, the risk was actually quite small because they mitigated it enough from an architecture perspective that you couldn't really hack into the device from the Internet.
You could maybe hack into the device maybe if you were can connect to it via Bluetooth and you were close enough. But I think, you know, as kind of a white hat type person, that's a totally different paradigm. Right? Like, can anyone in the world access it now or just people that are in my vicinity? And they announced it and they got back to all the users. And To me, that was an example of, look, this stuff works, right? Planning everything works, but yeah, you need to have the context of where your device is used, what other devices it's connected to, and then have an understanding of what could go wrong. And if it's a pacemaker, I think it needs to have a very different architecture than if it's an image classifier for radiology images, right, just because of how immediate the risk is.
And I think being sophisticated about that is just part of developing serious systems. And if you wanna build serious software that's deployed in serious application that people depend on, there's just an amount of work that goes into that. We often hear it just to spin this around in in the side of like, should I run this highly safety critical process on the cloud and the cloud is out of my control, right? And AWS or Google or IBM, they do a bunch of different things in the cloud while you're working in this kind of server farm, and maybe some of those things can cause a latency issue or it can cause a performance degradation and you know, our response is, well, you know, maybe you shouldn't put something that can kill someone on a cloud. Maybe that should only be on the device and the way it's sequenced with commands, should be much more hardened, as they say in the DOD space.
[00:13:48] Unknown:
In terms of that risk evaluation framework, you mentioned that the level of control and the level of validation is dependent on whether it is seen as as a high risk or medium or low risk. I'm wondering what are some of the factors that come to play to determine what that level of risk happens to be and how much, education and awareness you have had to do with developers to understand what are those heuristics to identify. This is high risk. I really need to be sure of x, y, and z, or this is low risk, so I can ignore b, c, and d.
[00:14:25] Unknown:
Yeah. And isn't that the question of the century, I think, is what does it even mean to be high risk and low risk? And I think we're gonna see a lot of regulation roll out in the AI space around that that, yeah, you could still build little widgets and apps that don't do anything, but if you're making decisions about loans that are gonna substantially impact people's lives, well, it better be not biased as many models are today, and we already know this. And I think that's gonna be a good question. How do you triage? What does it mean to be high risk? I think it's gonna change away from a paradigm around harm to a paradigm around harm that's not physical, harm to people in the abstract and I think we're already seeing that with social media companies, right? Like I know a lot of people who have kids who are like, my kids aren't getting on Facebook and Instagram until, you know, they beg me to be on Facebook and Instagram, and I can't deny them anymore because we don't we don't feel it's a safe place for people because the companies are not worried about this. But fundamentally, it's all about risk triage, and the way you do that is by saying there's an element that's high risk and what does that mean to have risk? Risk is a probability, is a calculation that is the result of the severity of something happening, usually it's called a harm, not getting drug infused to you or falling off a staircase, that's a harm, right, and then people put risk controls to prevent that harm like a rail or something like that, or empty slip kind of little patches on the floor.
And then the risk is really a result of the severity, how, you know, could it kill me? Would it cause a severe injury that I would need to get hospitalized for? Would it cause a light injury or would it cause no harm? That's usually kind of, the different aspects of the adverse event that can happen. And then there for physical devices, this is where it gets a little complicated, there's also a probability, the probability of harm and the probability of that harm actually happening, right? So there's like the probability that I will get in front of the stairs and there'll be water on them and then I'll slip, So the probability that there's water, let's say, you know, the stairs are outside, it rains so many days a year, so 1, you know, 30% of all days in Boston, there's probability of water.
Then if there is water, what's the probability of me actually slipping? Let's say it's 1 in 5. And if the risk is quite low, maybe overall, we say now that the first probability, p 1 times the second probability times the severity, we can do a little matrix and calculate it and say in our risk management, it's either too much or or fine. Right? The issue is in software, you don't have that aspect. Because you don't really know when harm will appear in software, people assume that the probability is usually 1. And so you're only trying to mitigate the severity with most software things because, who knows will my child be exposed to bad content on Instagram? I don't think they can produce a probability of that. And so it's mostly about the severity and then what risk controls could you put in place. And I think the future is really about teams understanding what the product does, the possible risks it would have, and then ways to mitigate that risk, which is, by the way, a little bit different than how a medical device thinks about it because a medical device says you don't just need to mitigate risk. You also need to have evidence called validation that the system performs what you wanted it to perform. But I think in some cases, it's gonna be less about proving it works. It's mostly about proving it doesn't do bad things for most common applications. But does that make sense of how you would think about it? Starts from the end. Absolutely.
[00:17:50] Unknown:
And another layer in the case of ML systems also is how much authority does it have to take an action, where is the model able to take direct action on some other system that will have an impact? Or is this purely a an informative output where a human is the 1 that's going to be interacting with and interpreting it? But even in that situation, there's also the question of ensuring that you're surfacing the level of confidence and, addressing any potential for bias where in the radiology example that you gave, what is my level of confidence that the, scan of this X-ray actually includes a tumor versus just some other obstruction or abnormality in the physiology of the patient. And if the, if the level of confidence isn't surfaced and the practitioner just assumes that it is going to be accurate, then maybe they're taking an action that is not properly informed, and they don't do enough of their own due diligence or rely on their own expertise to actually critically evaluate what the model is telling them.
[00:18:58] Unknown:
Yeah. And I think that's the most common risk control for machine learning systems is that there's a human in the loop that can prevent any action, and that's why most ML systems today in medicine are really not taking any actions. They're the vast majority of them are suggesting certain things to focus on for a physician. Vast majority of applications in medicine, no surprise, are image processing, image classification applications. That's happened kind of post AlexNet. Now there's gonna be a big boom in in language models for sure. But the biggest way they, you know, mitigate the risk is by the severity. Just if there's you know, you have to it's just suggesting to a person and just helping them focus, but the person makes the decision, it's a lot less risky.
But it it'll be impossible for us to keep doing that forever. Like, eventually, the whole power of AI and machine learning and software, because because I think it all part of that is to automate, things in our lives, and and we will, have to do that. And we just need to figure out how we do it in a safe but innovative way. Right? Because in most things, innovation wins, but in medicine, you need to have innovation, but needs to be safe.
[00:20:04] Unknown:
And in that ML space, bringing it back around to the design, development, and validation of the models, I'm wondering what are the ways that the regulatory requirements impact the approach that teams take to developing and testing their ML models or even their willingness to invest in machine learning given the additional burden versus other ways that they might use their skills in the ML space?
[00:20:32] Unknown:
So I think a lot of what's needed in the market is folks who really understand the aspect of validation of the machine learning models. And I think there that's gonna be a big need in a lot of time spent by folks, trying to find people like that. They can help the regulatory folks in their organization, the quality folks understand how do you even test if this model works. Right? You need a subject matter expert, and I think that's where there's gonna be a big gap. Not that many people have approved the medical device that's based on AI, and so the amount of people who know how to do that is not very high, but that's gonna grow and grow. So I think that's, like, a major focus for us is, like, how do we help companies reduce the the need for so many SMEs that can't be found in order to accelerate their innovation?
[00:21:21] Unknown:
For the development process, another aspect that you mentioned is the question of auditability. There's record keeping requirements. I'm wondering if you can talk to the ways that the development teams can structure their work to reduce the number of procedural tasks that they need to add on to their existing development work. Just being able to make that something that happens in tandem in some of the ways that, your product at Ketrix is working to, further reduce that, kind of managerial burden of the work.
[00:21:57] Unknown:
Yeah. I'm trying to to think you even have to answer that because it's a very complicated question. Right? The whole issue is that the regulator in general, both FDA and others, they regulate the total product life cycle. They wanna regulate everything from the early design, the early risk management, the production, the testing, deployment, and then, you know, there's a huge area called post market surveillance, which is what happens after something's in the market, and all of that is a big cross functional team, and we believe that that team wants to do that work through commonly used development tools and we just enable them to do that instead of using some system, some legacy system that, you know, you can't even find a developer that knows.
And so the way we see it is is actually a collaboration between people and computers. We have a lot of AI in our system and that AI basically guides people onto how to develop things based on their existing processes. It could be configured to them and can suggest stuff and can help them author certain documents, and then through that, we just reduce the amount of of communication and amount of lack of knowing what needs to happen. A big challenge in developing something that has so much process control is that what if you don't know the process. Right? What if there's not that many people in your organization that know anything about the standard operating procedure to develop this type of thing and you need to wait on that SME to to keep working? So we're trying to have that SME partially in the computer because I think it's an iterative thing because if you wanna develop fast moving, large, complicated web systems that are learning, you probably need to have something similar to that helping you build them.
And that's what Ketrix is really, it's this mind that sits behind all the different tools and and helps guide the overall process to meet some existing standard.
[00:23:52] Unknown:
Given the need to integrate with that suite of tools for the, overall life cycle of the model, I'm curious how you're approaching that from the design and development of your platform, given the fact that the ML space is in a state of high volatility because of all of the activity that's happening, particularly around LLMs, but just in the overall advancement of machine learning as a discipline and the fact that there are still a lot of, as yet to be decided best practices?
[00:24:23] Unknown:
So I think we you know, our approach has been to start from the things that we know matter. Like before you even wanna work on AI, first you need to know how to build software, right, and we use Kedriks to develop Kedriks. It's recursively developed in the same system, which allowed us to do that. So we started by first, can we help people build large complicated software systems And then we use that ability to build our own system and to integrate it with a lot of different systems. And now, we kind of go to the tried and true things. In some ways, being in a regulated industry helps you with that because it's not like everybody's gonna switch on doing the latest cutting edge thing in medical devices.
There was some kind of time till that echo of the innovation reaches the regulated industries. And in that time, we we can kind of see how things are developing and be that gate that, ensures that the tools that are kind of well suited and the best in class tools and also the best in class processes can be used for that. And so basically, we integrate with what our customers want and what people want the most and also what makes sense for us, and then we're slowly building it up as we connect to more and more things. But I think, you know, there's a lot of needs before we even talk about large language models that are not met yet. Like, the vast majority of companies can't even deploy in a monthly or weekly fashion, which is very outdated compared to how most software is deployed today.
And and we're very much focused on enabling that core technology that would let everything else happen and also build the tools that would allow us to do this. And for the
[00:26:00] Unknown:
validation aspect, there are a number of different approaches that are being developed and explored, things like Shapley values for being able to understand what is actually happening at each of the phases of the model, the idea of causal modeling using Bayesian statistics as opposed to stochastic methods. I'm wondering how you have seen teams try to tackle that question of how do I either understand what my model is doing or explain why a model has made a certain decision.
[00:26:32] Unknown:
And I think that's the the, in some ways, the hardest part of this entire question. Because even if I solve it for you to some specific case, it would always depend on your system and not just on the technology you're using. Right? That's kind of the developer's perspective. It really depends on the intended use. I think there's every 1 of those aspects of the error generation and forecasting of what, the way the failure mode could happen and if it emerges, right, like if the loss goes down, maybe it's starting to get intelligence and all that discussion. I think it would always be kicked back to the manufacturer and that's actually very clear in the regulation that the manufacturers are just gonna take more and more responsibility and liability about that.
And, basically, whatever is right for your use case and your technology, the regulators would expect you to have an SME in house that is able to understand what that is and what's the appropriate controls to put in place and that you've put those controls in place. And I think it could be that as those controls, I think, even, it's gonna be a combination of things mostly tracked over time because I think that any any person who's built a lot of validated or high reliability systems will say, in some ways, it's not about what I choose to measure. It's about if I accepted that measurement in the first go around, how does that measurement change over time?
Right? I'm kind of, in some ways, more interested in that because I know that it's iterative, and I know that things change in in medicine, let alone the software can be wrong. What about the clinical model? What if the way I'm treating the disease is totally wrong? I don't understand the biology. So I have to kind of always monitor it, and I think it's to me, it's less of a question of the particular error. Like in forecasting, I did a lot of work in supply chain forecasting forecasting of all kinds of things. There's a lot of different arrows you could use. I think the most important thing is just be consistent to make sure it's appropriate for your intended use. Like, if your intended use, it's not a good idea to have an absolute value because you need the negatives, maybe it's a bad idea to have an absolute value in the way you do it, maybe it's fine.
But as long as you pick something that's appropriate for your intended use of Facebook profit or whatever you're using for forecasting, then it's basically more of a of a question of has my model become worse or better over time. And I think that's that's where the best practices will will emerge of, like, how do you technically do that? But I I in my view, is I just less focus on a particular methodology and more of a way to develop products, in general. And then every organization, every product
[00:29:12] Unknown:
is so unique, and they need to do whatever fits for them. Yeah. Also, that question of change management is interesting because up till this point, the we've largely been talking as if we're doing an initial builds and validation and deployment, and that's the end of the story. But for anybody who's ever worked on software systems over any period of time knows software is constantly evolving. Machine learning models are constantly evolving both from the software perspective, but also from the data being fed to it. Your example of a change in the clinical methodology for treating a particular symptom is also very interesting because that introduces another potential avenue of drift. It's not just drift in the source data that I have, but also drift in our understanding of the world from a perspective that is completely outside of the software and the data that I'm working on. And I'm wondering how that question of change management and model evolution factors into the regulatory requirements around that life cycle management.
[00:30:10] Unknown:
I could I could answer this question for an entire lifetime. But I I that's the core of it. Right? And that's why that's like my thought is when I got into a fairly regulated company coming as a developer, I felt that as I went to industry conferences and saw the industry works, I was like, well, this doesn't really align with how anyone does it today. Like, there you can't just not change the software for such long periods of time. Like, it's not safe even if you can't change it very fast, which is why the FDA came out or the congress with a patch act to say you have to have a way to have hotfixes. We it's not legal in the US right now to have a medical device that you don't have a robust hotfix patch process for. It's not just frowned upon. It is no longer legal to do that. There's an, you know, a law about the matter called the patch act. We meet a lot of teams, especially innovative young teams that are they think that, like, creating the first validated version is their challenge and how do you create the submission, but that's never the problem. The true problem is how do you create consistent, validated versions that are referenceable, that are well recorded.
And we built an entire system to deal with change because you make it once, but you change it forever, and it grows and you add stuff for it. And I'll also say that our developers over time, the people who work for us, kind of feel that this is an approach that's generally good for all software. Like, just to have a little more structure and think about what you're gonna change instead of just changing it. But, basically, the the requirement is you need to know what you're changing. You need to know how that change affects your system. You need to make sure that it's not adding risk. And if it is, you understand why it's worth it. Right? You can always have a benefit risk analysis that maybe the the benefit is greater than the risk. But I also think regulation is changing to allow that more, but, again, not in the sense of allowing you to do changes without any record keeping. That's very unlikely to me, that that's gonna be any future state in any high risk industry, that you could just change stuff all the time without any reference. The the main thing is that, changes are gonna be allowed more frequently by the manufacturer, pushing more liability to the people building the software.
And then, basically, there's now this new regulation that came out of the FDA. It's in draft right now. It's the AIML draft guidance. And in there, it says it was just released a few months ago. You know, basically, if you can segment your system well enough, for example, as a microservice and have the machine learning model as a separate piece, maybe you can preapprove with us the way you're gonna change it as long as there's a plan, as long as you can explain to us the decision points in general, it doesn't even need to be that specific that you're gonna make to change things. And the fact that you have a way to change that's gonna regenerate proper documentation and evidence over time, I think that's how the FDA sees it that you can change as long as you know what you're doing. And I think that's really, really hard for folks because, you know, you usually think of change in in, like, the Git perspective. You only need to change the code.
But frankly, in validated systems, code is just 1 of many, many, many different design artifacts that need to be produced to show that the system works as you said it does, and we kind of have this innate trust that health care and medical device and pharmaceutical companies are doing this because we would never trust them with their children's lives otherwise. And so change is kind of a core part and just maybe a finishing note, why is is change such a big part? It's because in the nineties, the FDA did research and showed that 20% of all software recalls in the United States are software related. And out of that 80 percent 20%, 80% of them. So it's like 16% of all recalled in the United States are a result of changes to the original system that caused harm or caused something that required a recall.
And so change is kind of inherent to this, change is inherent to software, and I think the kind of best in class assumption right now and especially the highly regulated industries, the change is not gonna happen very often, and it should be avoided. And I I I I just don't think that's true. I think change is a blessing, and we just need to do it in a controlled manner.
[00:34:17] Unknown:
Yeah. And that is an interesting parallel as well to the whole advent of the DevOps movement where software as an industry was largely stuck in this mode of we are going to release large blocks of change infrequently because that gives us a lot of time to verify the things that we're changing. But it was shown that that's actually the least effective way of making sure that you're doing things safely because you're never going to be able to, cover the entire service area of what you're doing and that it's actually much better and much safer and easier to manage to have more frequent small change sets because every time something breaks, you have a much smaller delta of what to evaluate versus, oh, I'm only gonna release once every 6 months. So now here's, you know, 4 months worth of software development, 2 months worth of testing, and then your software engineers are sitting on their hands for 2 months. And so there's a lot of wasted cycles and still a lack of any trust in the system.
[00:35:18] Unknown:
Right. And you wanna shift left everything in general. Right? That's, like, the approach we're talking about. And it's shift left because you're shifting it left on the v model, which all this regulation, you know, is all about, the v model. How do you validate something and you validate it based on the v model? Yeah, I think it's that's true, but with that said, I also think the changes you have to, in these type of systems, have a lot of rigorous checks before you even change something because you're never gonna catch everything. But just the core things you know it should do and the core things that you're mitigating need to be tested. That's a legal obligation. All risk controls for any product that is released, as a medical device in the United States need to be mitigated for every single release. I'm just thinking that, like, as we were talking about it, right, it's like the the challenge is that that's it's just complicated. It's not black or white. It's not like release fast and and see what happens or don't do that at all. I think it's just it requires some knowledge and understanding and sophistication in how we deploy software. Absolutely. It's very hard to make things that work well. Yeah. Definitely.
As somebody who works in the software and technology industry, I'm constantly amazed that anything works. Yeah. And that's the the, you know, especially the more you do it, the more you're like, how when I started this journey, I was a developer and I was very frustrated. Like, how can anyone do what what these regulations are asking to do? Like, no 1 will make anything if that's the case. And frankly, over time, you realize that the things that bother you the most in this regulation, which is mainly design verification, the act of saying that this thing, you know, my original requirements are met by the thing I built eventually and any change, I need to redo that. That aspect, which is the most burdensome and most difficult and most annoying, is also the most obviously a must have. Like, if I'm building something that people rely on, yeah, I have to sit and say, yeah, I think we did everything we said we did and we didn't forget anything.
It just people want it to be black and white, right? You know how people are and how developers are, especially. Developers, they're very flexible, but they also want it to be very clear and, you know, develop faster, die.
[00:37:25] Unknown:
And and that also goes to the airline regulations of having to have checklists for every procedure in flight because if you don't have the checklists, then you start to lean on your, you know, memory or intuition. You say, oh, yeah. I already did that. I don't need to worry about it, and you invariably slip up and miss something. And so this is just another manifestation of that of we need to have these checklists to make sure that we're actually looking at all the things we're supposed to look at and not just assuming that it's already been done.
[00:37:54] Unknown:
Yeah. And you can't assume when someone's life's on the on the line. Right? You can't just say, we'll see if in production, if we get a bug, we'll fix the bug. And a lot of developers tell me 2 things. 1 is if you do this this way, you'll make less features, and the response is, yeah, of course, that makes sense. Maybe you need less features. You don't need all the features in the world in certain systems. And I think there's also, like, a fear of it slowing down things and adding more documentation, but there is a point to that and I think I'm very much against having documentation as a sec of documentation. I think you need to have it in a way that makes sense for your organization, for your product, for the risk level, but I also think that without that, without the ability for someone to step in and see what you're doing and audit you, there are just bad incentives. And and everything here, we're just, you know, in 2023, we forget how it used to be before this regulation, but people are selling literal snake oil, telling people that their kid will be cured of of polio if they just drink olive oil. We've gone a long way from there, but it's not easy to do that. Right? There is I heard someone say it very well.
Said that the best companies in tech might have better quality than the best companies in medical devices, but the average company in tech has way worse quality. So and that's what the regular is concerned about. Overall, am I improving the quality of things?
[00:39:16] Unknown:
From your experience of entering this space, building the product around KetchX, what are some of the ideas or assumptions that you had about the potential space for solutions that have been changed or updated as you worked through with customers and came to understand more about the overall, problem space?
[00:39:36] Unknown:
I think 1 is just how hard it really is to build, regulated web based applications. Machine learning just adds more complexity to it that people often run kind of against the wall and say no one's ever gonna regulate machine learning models and high risk applications because of the statistical nature. Fundamentally, I disagree with that assumption because we regulate, biopharmaceuticals and biopharmaceuticals are produced in a statistical manner. They're grown. They're not made. And if we can regulate that, we could really regulate everything. And this goes into cancer patients, you know, direct IV. And so I think that the 1 assumption I had is it can't be that complicated. Well, it is. Some assumptions I had were where can we reduce it, and there are places to reduce it, but there's also places that you just totally understand why they're asking for it.
And the third is that there would be some limit to regulating AI systems, but I don't think that's true. I think there just needs to be a lot of subject matter expertise and knowledge that there's not enough people who know. And that's what software tooling is made for. Right? It's made to automate subject matter expertise and make it easier for people for less people to do more.
[00:40:44] Unknown:
And in your work of helping customers gain control of their overall development cycles, reduce some of the operational burden of developing these systems and help with some of the automation aspects and just understanding what their requirements are? What are some of the most interesting or innovative or unexpected ways that you've seen ML used in the medical space?
[00:41:08] Unknown:
There's so much of it going on right now. I think, most innovative. Most innovative is hard. I think some of what people are doing with images is amazing. I think the way it's been used, even like linear models or multi linear models for a long time is pretty pretty cool and useful. Right? People think about regulated AI. There have been approved machine learning applications in medicine since the late eighties, early nineties, just not what we do now. I think what it's gonna do for home care, for people that live in places that have less access to health care is gonna be transformational. And I think that's to me is why I'm I'm so excited about machine learning is we're gonna take a lot of things that people know and no 1 else knows, and we're gonna make it more accessible to people to increase the level of care. Because at the end of the day, the growth of population, is outpacing the growth in providers, physician, nurses, and so on.
And we need to find a way to reduce the cognitive load it takes to perform certain medical interventions and then the amount of time physicians spend on it. And I think there's a lot of amazing stuff going on in surgical robotics as well regarding this. How do you make a robot that's just easier to do surgeries with To just make it less burdensome to learn how to make how to do surgeries and and I'm pretty excited about that. And I think it all ties together to, we want systems and software to automate tasks in our life and reduce the challenge of doing that, and machine learning is just kind of the 1 of the most advanced stages of doing that.
[00:42:42] Unknown:
And as you have been working in this space and building the KETRIX product and business, what are some of the most interesting or unexpected or challenging lessons that you've learned?
[00:42:53] Unknown:
I've learned that there's amazing people out there in the world that are spending their life building really important medical applications, and they're yearning for a way to do it better and faster and right. And I think that was really exciting to me is to meet a lot of dev teams that are not just trying to do this, but are trying to do this really, really, really well and are thinking of patients all the time and how to make good companies to do that with and and are excited about what we're building. Right? It's always exciting where you have this idea. You're sitting in some company while other people care. And and so things that seeing that they do care and also the overall acceleration and the way the regulator in the US and the EU has taken on software over the last 24 months, I think, is very exciting. There have been more movement in the last 24 months or 12 months in software than there has probably in the last 30 years.
The real big set of regulations came out in the early 2000, but even they were just less robust than what's going on now. And we're seeing it all over. We're seeing the EU start to really enforce a lot of things around software, whether that's GDPR or or many other things or the EU AI Act or MDR, medical price regulations, or IVDR, as well as a lot of clinical trial regulations for data. And we're also seeing the same thing in the US, and and it excites me to see that everybody's seeing the impact software can have on everybody's lives and want to make sure that that's gonna happen in a reasonable, efficient, and safe way. And it really excited me to just see how many people are thinking of that desire to help other people, save lives. And I just think it's limitless what's gonna happen in medicine. We're gonna see a transformation over the next 20 years, that's gonna reduce the the cost of things and also give access to treatment that's extremely advanced to all kinds of places in the world.
[00:44:45] Unknown:
And for people who are building or thinking about building products in this space of medical devices, medical applications, trying to address all of the regulatory aspects of it, what are the cases where Ketrix is the wrong choice?
[00:44:58] Unknown:
If you're building software in medicine and you're not using Ketrix, you've already made the wrong choice. And I think every person that works for us will say, this is the only way to do it. I think if you're building certain embedded systems that take a long while to develop, it could be in some ways less attractive or more attractive. But I think if you're building complex systems and and we even have people designing hardware on top of KETRIX now just because the system is so intuitive and the AI in it is so helpful. But I I I really you know, that's my belief in we're building a company to just help people that build software and medicine do it faster, you know, at 10% of the cost and 10% of the time.
And I think that in 5 years from now, we're gonna see that it's way over that. I think that the amount of the amount of money people are spending to build medical software is outrageous, is outrageous and it is a shame for us as a society And it is a shame for the millions of patients out there in the world that need treatment and need funding for devices that are not getting it. And we're gonna reduce that by so much that every entrepreneur and every innovator that wants to start sees something in the clinic and wants to start a company never thinks again about it, but, I don't really wanna do medical devices because it's hard. And I think that's part of a broader story of automating society that we're gonna we're gonna be part of on how do you do it in other spaces. But I think if I was for my first time trying to develop this, I would go and read FDA's good principles of software validation and understand what's even been asked here and then figure out how to go from there because most people just wanna build and they don't think about the regulatory aspect of things and regulatory strategy and you can save a lot of time doing it right early on and not fixing it later.
Yeah. And I think our best users are leaders because it's, just over time, what happens is if you don't do it well, if your company doesn't go anywhere, it doesn't matter. But if it goes, the company is really successful, it'll matter a lot. This might be the only thing that matters ever in your company at the end of the day.
[00:46:53] Unknown:
And as you continue to build and iterate on the Ketrix product, what are some of the things you have planned for the near to medium term or any particular problem areas or projects you're excited to dig into?
[00:47:05] Unknown:
More platforms, more ecosystems, more community around it, a bigger way to collaborate. I think that in coming months years, you're gonna see a lot of collaboration features that help in a wide variety of ways people, work together to make products, in a totally distributed fashion and across companies and across the supply chain. That's something I'm very excited about because there's a very robust world of people that help you make biologics or pharmaceutical agents or hardware medical devices, but it doesn't exist in software and it's kind of sad because software is the easiest way to do that. Right? I can just ship someone code. And so we're gonna have a big focus on that and how people work together to make stuff.
And,
[00:47:48] Unknown:
yeah, it excites me. It's all about people collaborating together and building ideas together and then making sure those ideas are are working really well. Right? That's all the FDA really asks for. Are there any other aspects of the work that you're doing at Ketrix, the product and platform that you're building, or the overall space of building software and machine learning applications for the medical space that we didn't discuss yet that you'd like to cover before we close out the show?
[00:48:14] Unknown:
Not really, except maybe the fact that people need to focus on the overall product, and there's a tendency to focus on machine learning as a product. I don't think the vast majority of models are not products. They're building blocks for other products, and people need to start thinking about it that way. Machine learning as a model generally would be a feature. LLMs are more complicated than that, but they're generally a feature unless you're chat GPT and there's not gonna be a lot of other chat GPTs. You're a feature within a broader context, and you need to understand how that broader context looks and works.
[00:48:45] Unknown:
Well, for anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest barrier to adoption for machine learning today.
[00:49:00] Unknown:
Fundamentally, it's subject matter expertise as it has been for any new technology ever. Some of it is resources, but I think there's also a lot of great resources, technical resources, like compute and the United States has decided now to create a central governing resource for AI research and and share that across communities and university. I think it's very exciting, but I think it's all about knowledge, and software is also about knowledge, but things are progressing very fast. And if it's hard to find someone to build a model, how about someone who understands those models well enough to explain that to a regulator?
Right? And then Albert Einstein always said, if you can't explain it to someone who doesn't know what you're talking about, you probably don't know well enough. That's all the FDA is asking for, show us that you know it well enough that you can explain it to us. And I think that's going to be the challenge in machine learning overall and the challenge in machine learning in medicine because it just kind of over the machine learning complexity. You get another layer of regulatory complexity, another layer of software development and DevOps and all those together. You know, how many people know how to develop web based machine learning products that are validated to medical regulations?
Not
[00:50:12] Unknown:
many. But how many of those do we need? A lot of people. Well, thank you very much for taking the time today to join me and share the work that you and your team are doing to simplify and, reduce the overhead of working in this space. Definitely appreciate the time and energy you're putting into that, and I hope you enjoy the rest of your day. Yeah. Thank you, Tobias. It's been a real pleasure. I enjoyed it.
[00:50:37] Unknown:
Thank you for listening, and don't forget to check out our other shows, the Data Engineering podcast, which covers the latest in modern data management, and podcast dot in it, which covers the Python language, its community, and the innovative ways it is being used. You can visit the site at the machine learning podcast.com to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email hosts at the machine learning podcast.com with your story. To help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.
Hello, and welcome to the Machine Learning podcast. The podcast about going from idea to delivery with machine learning.
[00:00:20] Unknown:
Your host is Tobias Massey. And today I'm interviewing Erez Kaminsky about using machine learning and safety critical and highly regulated medical applications. So, Erez, can you start by introducing yourself?
[00:00:30] Unknown:
Yeah. Thanks for having me, Tobias. It's a real pleasure. So my name is Erez. From Boston, I, I'm the CEO of a company called KETRIX that helps regulated software teams perform CICD and just develop fast and stay compliant and make safe and reliable applications. I spent, basically most of my adult life working on high reliability applications and the nuclear engineering domain and medical devices and pharmaceuticals. And kind of most recently before doing this, I I was leading AI for Amgen, which is the largest biotech company in the world, as a medical device division. So I've been thinking for a long time on how to do this and then kind of classical developer.
I worked at a big company developing this very complicated thing and said, you know, what if I had better tools to do this with? Are there other people who feel the same way and left to to build the tooling company, really? And Ketrix is a software development tooling company
[00:01:29] Unknown:
for highly regulated industries, including kind of regulated AI and so on. Before we get too much into the bulk of what we wanna talk about today, I'm also interested in how you first got started working in the machine learning space.
[00:01:42] Unknown:
Yeah. So really my background is I'm I'm a I'm a physicist, and and I worked in in physics and helped to build numerical simulation applications. That's kind of where I always say that I'm not a software engineer. I'm just a person who knows how to build fancy calculators, in low level languages or high level languages. And so that started an interest, right, because numerical simulation, you get really into computing, into creation. The time is, like, 2014, 2015, 2016. So AlexNet made a bit a big impact. You hear a lot of it. And even in the research community I was in, there was starting to be conversations of can we accelerate certain numerical simulations with, neural networks. But the real jump for me is, I went to work for a company called Wolfram Research that develops Mathematica and Wolfram Alpha, So a lot of both kind of more classical expert system work, but also more kind of modern machine learning work. And there, I got exposed really to everything that was going on there as part of their machine learning group and and helping to support them developing products and, you know, we do a lot of different things even even back in the day, which is not too long ago, but in machine learning time, you know, even 5, 6 years ago is is eons ago. So that's how I first got involved, and then I've been around the Boston machine learning community for a long time and helping different government private establishments do machine learning, get trained on machine learning, and then did a lot of work at MIT around machine learning specifically for health care and biology.
[00:03:12] Unknown:
And the idea of safety critical, highly regulated systems is a massive area of active research and development in the software community, and I can only imagine at the additional levels of complexity that are involved when bringing ML into the equation. But before we get too much into the specifics, I'm interested in getting a description of what some of the regulatory burdens are that are that exist for ML teams who are building solutions for medical applications.
[00:03:43] Unknown:
Yeah. And and I think, you know, I'll start more in general than that even, but we'll get into medical. I think that there's basically 2 type of things in the world. Right? There's things that you could just build and no 1, focuses on, no 1 cares about, there's no public pressure about them, they're not society critical, and there are things that are. And generally speaking, society critical or or just critical things are regulated. There's a a wide range of what it means to be critical. Are you mission critical? Are you safety critical? And with that, kind of, the burden of regulation and the burden of evidence goes up.
But, generally, everybody's asking for the same type of thing, which is if you are claiming that this system or application you built can help people in a moment of need with something they need to rely on to make sure their finances are okay or make sure that they are safe from harm, there's usually a regulator in the world that regulates that for every jurisdiction, and we see that's just growing, both kind of the amount of regulation but also the scope of what is being regulated, and a lot of the discussions over, I think, regulated AI touch that. But fundamentally, all regulations that deal with product, there's regulations that deal with your company, and that's more, you know, are you compliant with SOC 2? Do you have proper privacy controls overall?
But then there's regulations that deal with your product, and and that's our focus is basically how do you know that a system is doing what you said it did or what you said it was going to do. And at the end of the day, all the regulator is asking people to do is have a long document that describes for a less educated observer, right, someone who's just coming to audit you or inspect your facility, that explains what your system does, in a very robust way, and the robustness really depends on how dangerous your system is, like very dangerous systems more, less dangerous systems less. And then have evidence, that those things are being are being met, Those requirements are met by the system for every version of the system.
And while that sounds very easy, just kind of share with us what you do and then show us that you do those things. If you've ever built any web system or any complicated software system, you realize that's really, really challenging to do and very, very difficult to do and takes an immense amount of time. Now machine learning introduces into that problem the question of what does it mean for something to do what I said it does. Right? Like, it's a statistical model. It will always have certain parameters in which it operates correctly in and certain parameters in which it doesn't operate correctly, and then you need to try and assess. And and in the same way that you validate your software system to meet certain requirements, validate your machine learning system or subsystem, as the FDA calls it, to meet certain performance parameters and be consistently meeting them. And I think that's a very challenging environment for people who grew up building mobile apps, ad tech, even Fintech, to try and do. Like, it's so hard to do that. It's so hard to make something reliable and meet certain requirements, and machine learning just adds some complexity. And I think a lot of that complexity is about the subject matter expertise more than the regulation, frankly.
It's just like, what does that even mean to show that a neural network does what you said it does? What is what do you say it does? What do you think it does? You know? Absolutely. Yeah. Model validation is a hard enough problem on its own without having to do it in a provably correct manner.
[00:07:18] Unknown:
And before we get too much even into that model development, model validation, model design question, in terms of security and safety in the software realm. There's also the whole question of the software supply chain and being able to manage and validate the dependencies that go into the the system. And then with ML, there's also the questions of the accuracy and reliability and the awareness of and correction for bias within that source dataset. And I'm wondering what are some of the ways that the combination of software supply chain along with data supply chain come together to compound the complexity that you're trying to address?
[00:07:59] Unknown:
That's such a good question, Tobias, and and it's actually the the first place where we started to work on on our company and and the product is how do we help people integrate open source software and, you know, kind of third party software into the regulated applications because it's it's complicated and people didn't use to develop with that much open source in the healthcare community, medical device community until kind of recently, and that's caused a lot of issues, right, because if you developed a medical device like 20 years ago, even a very advanced medical device, it would mostly have embedded software and a a very small amount of dependencies.
And now, you know, you wanna spin up, some image classifier with TensorFlow. There's already a lot of dependencies going on in there. Could be, you know, in the many thousands easily. And I think there was it's it's it's not an impossible task. It's a task of of understanding what your system does and how these different dependencies affect the safety of the system and the connection to harm, and then being able in a sophisticated manner to triage that based on both the security aspect aspect of that dependency, but also how the dependency works inside your validated application. And I think, you know, that's very challenging for the men mentality of software teams where, you know, you just kinda put whatever dependency you want in, whatever makes it work, and let's move forward.
But here, we really need to understand what we're doing, so that's why we we sometimes need to stop and think about it and make sure that, you know, we're not introducing a dependency that, would hit our performance or would get our end users exposed to unnecessary harm. And we have a big offering that helps do that, manage that softer bill of material and that softer supply chain because it's a common attack vector into all kinds of products. Right? And then there's recent study that says that kind of the average cost of a cybersecurity hack in health care is about $11, 000, 000 for a company, and that's just gonna go up and up and up.
So it's something that I think people don't think about enough, what I'm introducing into my application, and they need to do it more. And we've developed a lot of both kind of best practices and software to think of how to do that and how to connect back from the things your system can do poorly to harm another person to what you're using to develop it. And most of it, like you can imagine, is just saying, you know, have we thought about this? Do we know where we're using this? And is the thing we're using it in high risk or low risk? If it's low risk, maybe we need less attention to it. If it's high risk, we need more, and just to have some kind of procedure and thought process. And and I think the new guidance from both FDA and Congress, which is part of the patch act that passed in December of last year, has a huge focus on that. Right? If you build a medical application, particular medical device in the United States by October 1st this year, that device has to have a very robust software supply chain methodology.
[00:10:57] Unknown:
Another aspect of the security question before even touching on the complexity of ML is also the operating environment that you're dealing with, which the software bill of materials is 1 aspect, but then there are also questions around system hardening. Does it need to run-in an air gapped environment? Who has access to these systems? And I'm curious, again, how that factors in with these regulated and safety critical systems.
[00:11:23] Unknown:
It's a huge factor. Right? Because, again, it's just another vector that introduces risk. Like, let me give you an example. It's a pretty interesting recall by a large medical device company recently. They developed insulin pumps. That insulin pump, had a vulnerability exposed to it that would allow a user to take control of certain aspects of the pump. As far as I recall, it wasn't actually something super dangerous, but, you know, if I if if, you know, my child had an insulin pump, the very idea that someone can connect to it makes me extremely uncomfortable and and makes me not wanna do it at all or change your brand. But because they had good risk controls, and this is where you do see big companies doing a good job with these processes, the risk was actually quite small because they mitigated it enough from an architecture perspective that you couldn't really hack into the device from the Internet.
You could maybe hack into the device maybe if you were can connect to it via Bluetooth and you were close enough. But I think, you know, as kind of a white hat type person, that's a totally different paradigm. Right? Like, can anyone in the world access it now or just people that are in my vicinity? And they announced it and they got back to all the users. And To me, that was an example of, look, this stuff works, right? Planning everything works, but yeah, you need to have the context of where your device is used, what other devices it's connected to, and then have an understanding of what could go wrong. And if it's a pacemaker, I think it needs to have a very different architecture than if it's an image classifier for radiology images, right, just because of how immediate the risk is.
And I think being sophisticated about that is just part of developing serious systems. And if you wanna build serious software that's deployed in serious application that people depend on, there's just an amount of work that goes into that. We often hear it just to spin this around in in the side of like, should I run this highly safety critical process on the cloud and the cloud is out of my control, right? And AWS or Google or IBM, they do a bunch of different things in the cloud while you're working in this kind of server farm, and maybe some of those things can cause a latency issue or it can cause a performance degradation and you know, our response is, well, you know, maybe you shouldn't put something that can kill someone on a cloud. Maybe that should only be on the device and the way it's sequenced with commands, should be much more hardened, as they say in the DOD space.
[00:13:48] Unknown:
In terms of that risk evaluation framework, you mentioned that the level of control and the level of validation is dependent on whether it is seen as as a high risk or medium or low risk. I'm wondering what are some of the factors that come to play to determine what that level of risk happens to be and how much, education and awareness you have had to do with developers to understand what are those heuristics to identify. This is high risk. I really need to be sure of x, y, and z, or this is low risk, so I can ignore b, c, and d.
[00:14:25] Unknown:
Yeah. And isn't that the question of the century, I think, is what does it even mean to be high risk and low risk? And I think we're gonna see a lot of regulation roll out in the AI space around that that, yeah, you could still build little widgets and apps that don't do anything, but if you're making decisions about loans that are gonna substantially impact people's lives, well, it better be not biased as many models are today, and we already know this. And I think that's gonna be a good question. How do you triage? What does it mean to be high risk? I think it's gonna change away from a paradigm around harm to a paradigm around harm that's not physical, harm to people in the abstract and I think we're already seeing that with social media companies, right? Like I know a lot of people who have kids who are like, my kids aren't getting on Facebook and Instagram until, you know, they beg me to be on Facebook and Instagram, and I can't deny them anymore because we don't we don't feel it's a safe place for people because the companies are not worried about this. But fundamentally, it's all about risk triage, and the way you do that is by saying there's an element that's high risk and what does that mean to have risk? Risk is a probability, is a calculation that is the result of the severity of something happening, usually it's called a harm, not getting drug infused to you or falling off a staircase, that's a harm, right, and then people put risk controls to prevent that harm like a rail or something like that, or empty slip kind of little patches on the floor.
And then the risk is really a result of the severity, how, you know, could it kill me? Would it cause a severe injury that I would need to get hospitalized for? Would it cause a light injury or would it cause no harm? That's usually kind of, the different aspects of the adverse event that can happen. And then there for physical devices, this is where it gets a little complicated, there's also a probability, the probability of harm and the probability of that harm actually happening, right? So there's like the probability that I will get in front of the stairs and there'll be water on them and then I'll slip, So the probability that there's water, let's say, you know, the stairs are outside, it rains so many days a year, so 1, you know, 30% of all days in Boston, there's probability of water.
Then if there is water, what's the probability of me actually slipping? Let's say it's 1 in 5. And if the risk is quite low, maybe overall, we say now that the first probability, p 1 times the second probability times the severity, we can do a little matrix and calculate it and say in our risk management, it's either too much or or fine. Right? The issue is in software, you don't have that aspect. Because you don't really know when harm will appear in software, people assume that the probability is usually 1. And so you're only trying to mitigate the severity with most software things because, who knows will my child be exposed to bad content on Instagram? I don't think they can produce a probability of that. And so it's mostly about the severity and then what risk controls could you put in place. And I think the future is really about teams understanding what the product does, the possible risks it would have, and then ways to mitigate that risk, which is, by the way, a little bit different than how a medical device thinks about it because a medical device says you don't just need to mitigate risk. You also need to have evidence called validation that the system performs what you wanted it to perform. But I think in some cases, it's gonna be less about proving it works. It's mostly about proving it doesn't do bad things for most common applications. But does that make sense of how you would think about it? Starts from the end. Absolutely.
[00:17:50] Unknown:
And another layer in the case of ML systems also is how much authority does it have to take an action, where is the model able to take direct action on some other system that will have an impact? Or is this purely a an informative output where a human is the 1 that's going to be interacting with and interpreting it? But even in that situation, there's also the question of ensuring that you're surfacing the level of confidence and, addressing any potential for bias where in the radiology example that you gave, what is my level of confidence that the, scan of this X-ray actually includes a tumor versus just some other obstruction or abnormality in the physiology of the patient. And if the, if the level of confidence isn't surfaced and the practitioner just assumes that it is going to be accurate, then maybe they're taking an action that is not properly informed, and they don't do enough of their own due diligence or rely on their own expertise to actually critically evaluate what the model is telling them.
[00:18:58] Unknown:
Yeah. And I think that's the most common risk control for machine learning systems is that there's a human in the loop that can prevent any action, and that's why most ML systems today in medicine are really not taking any actions. They're the vast majority of them are suggesting certain things to focus on for a physician. Vast majority of applications in medicine, no surprise, are image processing, image classification applications. That's happened kind of post AlexNet. Now there's gonna be a big boom in in language models for sure. But the biggest way they, you know, mitigate the risk is by the severity. Just if there's you know, you have to it's just suggesting to a person and just helping them focus, but the person makes the decision, it's a lot less risky.
But it it'll be impossible for us to keep doing that forever. Like, eventually, the whole power of AI and machine learning and software, because because I think it all part of that is to automate, things in our lives, and and we will, have to do that. And we just need to figure out how we do it in a safe but innovative way. Right? Because in most things, innovation wins, but in medicine, you need to have innovation, but needs to be safe.
[00:20:04] Unknown:
And in that ML space, bringing it back around to the design, development, and validation of the models, I'm wondering what are the ways that the regulatory requirements impact the approach that teams take to developing and testing their ML models or even their willingness to invest in machine learning given the additional burden versus other ways that they might use their skills in the ML space?
[00:20:32] Unknown:
So I think a lot of what's needed in the market is folks who really understand the aspect of validation of the machine learning models. And I think there that's gonna be a big need in a lot of time spent by folks, trying to find people like that. They can help the regulatory folks in their organization, the quality folks understand how do you even test if this model works. Right? You need a subject matter expert, and I think that's where there's gonna be a big gap. Not that many people have approved the medical device that's based on AI, and so the amount of people who know how to do that is not very high, but that's gonna grow and grow. So I think that's, like, a major focus for us is, like, how do we help companies reduce the the need for so many SMEs that can't be found in order to accelerate their innovation?
[00:21:21] Unknown:
For the development process, another aspect that you mentioned is the question of auditability. There's record keeping requirements. I'm wondering if you can talk to the ways that the development teams can structure their work to reduce the number of procedural tasks that they need to add on to their existing development work. Just being able to make that something that happens in tandem in some of the ways that, your product at Ketrix is working to, further reduce that, kind of managerial burden of the work.
[00:21:57] Unknown:
Yeah. I'm trying to to think you even have to answer that because it's a very complicated question. Right? The whole issue is that the regulator in general, both FDA and others, they regulate the total product life cycle. They wanna regulate everything from the early design, the early risk management, the production, the testing, deployment, and then, you know, there's a huge area called post market surveillance, which is what happens after something's in the market, and all of that is a big cross functional team, and we believe that that team wants to do that work through commonly used development tools and we just enable them to do that instead of using some system, some legacy system that, you know, you can't even find a developer that knows.
And so the way we see it is is actually a collaboration between people and computers. We have a lot of AI in our system and that AI basically guides people onto how to develop things based on their existing processes. It could be configured to them and can suggest stuff and can help them author certain documents, and then through that, we just reduce the amount of of communication and amount of lack of knowing what needs to happen. A big challenge in developing something that has so much process control is that what if you don't know the process. Right? What if there's not that many people in your organization that know anything about the standard operating procedure to develop this type of thing and you need to wait on that SME to to keep working? So we're trying to have that SME partially in the computer because I think it's an iterative thing because if you wanna develop fast moving, large, complicated web systems that are learning, you probably need to have something similar to that helping you build them.
And that's what Ketrix is really, it's this mind that sits behind all the different tools and and helps guide the overall process to meet some existing standard.
[00:23:52] Unknown:
Given the need to integrate with that suite of tools for the, overall life cycle of the model, I'm curious how you're approaching that from the design and development of your platform, given the fact that the ML space is in a state of high volatility because of all of the activity that's happening, particularly around LLMs, but just in the overall advancement of machine learning as a discipline and the fact that there are still a lot of, as yet to be decided best practices?
[00:24:23] Unknown:
So I think we you know, our approach has been to start from the things that we know matter. Like before you even wanna work on AI, first you need to know how to build software, right, and we use Kedriks to develop Kedriks. It's recursively developed in the same system, which allowed us to do that. So we started by first, can we help people build large complicated software systems And then we use that ability to build our own system and to integrate it with a lot of different systems. And now, we kind of go to the tried and true things. In some ways, being in a regulated industry helps you with that because it's not like everybody's gonna switch on doing the latest cutting edge thing in medical devices.
There was some kind of time till that echo of the innovation reaches the regulated industries. And in that time, we we can kind of see how things are developing and be that gate that, ensures that the tools that are kind of well suited and the best in class tools and also the best in class processes can be used for that. And so basically, we integrate with what our customers want and what people want the most and also what makes sense for us, and then we're slowly building it up as we connect to more and more things. But I think, you know, there's a lot of needs before we even talk about large language models that are not met yet. Like, the vast majority of companies can't even deploy in a monthly or weekly fashion, which is very outdated compared to how most software is deployed today.
And and we're very much focused on enabling that core technology that would let everything else happen and also build the tools that would allow us to do this. And for the
[00:26:00] Unknown:
validation aspect, there are a number of different approaches that are being developed and explored, things like Shapley values for being able to understand what is actually happening at each of the phases of the model, the idea of causal modeling using Bayesian statistics as opposed to stochastic methods. I'm wondering how you have seen teams try to tackle that question of how do I either understand what my model is doing or explain why a model has made a certain decision.
[00:26:32] Unknown:
And I think that's the the, in some ways, the hardest part of this entire question. Because even if I solve it for you to some specific case, it would always depend on your system and not just on the technology you're using. Right? That's kind of the developer's perspective. It really depends on the intended use. I think there's every 1 of those aspects of the error generation and forecasting of what, the way the failure mode could happen and if it emerges, right, like if the loss goes down, maybe it's starting to get intelligence and all that discussion. I think it would always be kicked back to the manufacturer and that's actually very clear in the regulation that the manufacturers are just gonna take more and more responsibility and liability about that.
And, basically, whatever is right for your use case and your technology, the regulators would expect you to have an SME in house that is able to understand what that is and what's the appropriate controls to put in place and that you've put those controls in place. And I think it could be that as those controls, I think, even, it's gonna be a combination of things mostly tracked over time because I think that any any person who's built a lot of validated or high reliability systems will say, in some ways, it's not about what I choose to measure. It's about if I accepted that measurement in the first go around, how does that measurement change over time?
Right? I'm kind of, in some ways, more interested in that because I know that it's iterative, and I know that things change in in medicine, let alone the software can be wrong. What about the clinical model? What if the way I'm treating the disease is totally wrong? I don't understand the biology. So I have to kind of always monitor it, and I think it's to me, it's less of a question of the particular error. Like in forecasting, I did a lot of work in supply chain forecasting forecasting of all kinds of things. There's a lot of different arrows you could use. I think the most important thing is just be consistent to make sure it's appropriate for your intended use. Like, if your intended use, it's not a good idea to have an absolute value because you need the negatives, maybe it's a bad idea to have an absolute value in the way you do it, maybe it's fine.
But as long as you pick something that's appropriate for your intended use of Facebook profit or whatever you're using for forecasting, then it's basically more of a of a question of has my model become worse or better over time. And I think that's that's where the best practices will will emerge of, like, how do you technically do that? But I I in my view, is I just less focus on a particular methodology and more of a way to develop products, in general. And then every organization, every product
[00:29:12] Unknown:
is so unique, and they need to do whatever fits for them. Yeah. Also, that question of change management is interesting because up till this point, the we've largely been talking as if we're doing an initial builds and validation and deployment, and that's the end of the story. But for anybody who's ever worked on software systems over any period of time knows software is constantly evolving. Machine learning models are constantly evolving both from the software perspective, but also from the data being fed to it. Your example of a change in the clinical methodology for treating a particular symptom is also very interesting because that introduces another potential avenue of drift. It's not just drift in the source data that I have, but also drift in our understanding of the world from a perspective that is completely outside of the software and the data that I'm working on. And I'm wondering how that question of change management and model evolution factors into the regulatory requirements around that life cycle management.
[00:30:10] Unknown:
I could I could answer this question for an entire lifetime. But I I that's the core of it. Right? And that's why that's like my thought is when I got into a fairly regulated company coming as a developer, I felt that as I went to industry conferences and saw the industry works, I was like, well, this doesn't really align with how anyone does it today. Like, there you can't just not change the software for such long periods of time. Like, it's not safe even if you can't change it very fast, which is why the FDA came out or the congress with a patch act to say you have to have a way to have hotfixes. We it's not legal in the US right now to have a medical device that you don't have a robust hotfix patch process for. It's not just frowned upon. It is no longer legal to do that. There's an, you know, a law about the matter called the patch act. We meet a lot of teams, especially innovative young teams that are they think that, like, creating the first validated version is their challenge and how do you create the submission, but that's never the problem. The true problem is how do you create consistent, validated versions that are referenceable, that are well recorded.
And we built an entire system to deal with change because you make it once, but you change it forever, and it grows and you add stuff for it. And I'll also say that our developers over time, the people who work for us, kind of feel that this is an approach that's generally good for all software. Like, just to have a little more structure and think about what you're gonna change instead of just changing it. But, basically, the the requirement is you need to know what you're changing. You need to know how that change affects your system. You need to make sure that it's not adding risk. And if it is, you understand why it's worth it. Right? You can always have a benefit risk analysis that maybe the the benefit is greater than the risk. But I also think regulation is changing to allow that more, but, again, not in the sense of allowing you to do changes without any record keeping. That's very unlikely to me, that that's gonna be any future state in any high risk industry, that you could just change stuff all the time without any reference. The the main thing is that, changes are gonna be allowed more frequently by the manufacturer, pushing more liability to the people building the software.
And then, basically, there's now this new regulation that came out of the FDA. It's in draft right now. It's the AIML draft guidance. And in there, it says it was just released a few months ago. You know, basically, if you can segment your system well enough, for example, as a microservice and have the machine learning model as a separate piece, maybe you can preapprove with us the way you're gonna change it as long as there's a plan, as long as you can explain to us the decision points in general, it doesn't even need to be that specific that you're gonna make to change things. And the fact that you have a way to change that's gonna regenerate proper documentation and evidence over time, I think that's how the FDA sees it that you can change as long as you know what you're doing. And I think that's really, really hard for folks because, you know, you usually think of change in in, like, the Git perspective. You only need to change the code.
But frankly, in validated systems, code is just 1 of many, many, many different design artifacts that need to be produced to show that the system works as you said it does, and we kind of have this innate trust that health care and medical device and pharmaceutical companies are doing this because we would never trust them with their children's lives otherwise. And so change is kind of a core part and just maybe a finishing note, why is is change such a big part? It's because in the nineties, the FDA did research and showed that 20% of all software recalls in the United States are software related. And out of that 80 percent 20%, 80% of them. So it's like 16% of all recalled in the United States are a result of changes to the original system that caused harm or caused something that required a recall.
And so change is kind of inherent to this, change is inherent to software, and I think the kind of best in class assumption right now and especially the highly regulated industries, the change is not gonna happen very often, and it should be avoided. And I I I I just don't think that's true. I think change is a blessing, and we just need to do it in a controlled manner.
[00:34:17] Unknown:
Yeah. And that is an interesting parallel as well to the whole advent of the DevOps movement where software as an industry was largely stuck in this mode of we are going to release large blocks of change infrequently because that gives us a lot of time to verify the things that we're changing. But it was shown that that's actually the least effective way of making sure that you're doing things safely because you're never going to be able to, cover the entire service area of what you're doing and that it's actually much better and much safer and easier to manage to have more frequent small change sets because every time something breaks, you have a much smaller delta of what to evaluate versus, oh, I'm only gonna release once every 6 months. So now here's, you know, 4 months worth of software development, 2 months worth of testing, and then your software engineers are sitting on their hands for 2 months. And so there's a lot of wasted cycles and still a lack of any trust in the system.
[00:35:18] Unknown:
Right. And you wanna shift left everything in general. Right? That's, like, the approach we're talking about. And it's shift left because you're shifting it left on the v model, which all this regulation, you know, is all about, the v model. How do you validate something and you validate it based on the v model? Yeah, I think it's that's true, but with that said, I also think the changes you have to, in these type of systems, have a lot of rigorous checks before you even change something because you're never gonna catch everything. But just the core things you know it should do and the core things that you're mitigating need to be tested. That's a legal obligation. All risk controls for any product that is released, as a medical device in the United States need to be mitigated for every single release. I'm just thinking that, like, as we were talking about it, right, it's like the the challenge is that that's it's just complicated. It's not black or white. It's not like release fast and and see what happens or don't do that at all. I think it's just it requires some knowledge and understanding and sophistication in how we deploy software. Absolutely. It's very hard to make things that work well. Yeah. Definitely.
As somebody who works in the software and technology industry, I'm constantly amazed that anything works. Yeah. And that's the the, you know, especially the more you do it, the more you're like, how when I started this journey, I was a developer and I was very frustrated. Like, how can anyone do what what these regulations are asking to do? Like, no 1 will make anything if that's the case. And frankly, over time, you realize that the things that bother you the most in this regulation, which is mainly design verification, the act of saying that this thing, you know, my original requirements are met by the thing I built eventually and any change, I need to redo that. That aspect, which is the most burdensome and most difficult and most annoying, is also the most obviously a must have. Like, if I'm building something that people rely on, yeah, I have to sit and say, yeah, I think we did everything we said we did and we didn't forget anything.
It just people want it to be black and white, right? You know how people are and how developers are, especially. Developers, they're very flexible, but they also want it to be very clear and, you know, develop faster, die.
[00:37:25] Unknown:
And and that also goes to the airline regulations of having to have checklists for every procedure in flight because if you don't have the checklists, then you start to lean on your, you know, memory or intuition. You say, oh, yeah. I already did that. I don't need to worry about it, and you invariably slip up and miss something. And so this is just another manifestation of that of we need to have these checklists to make sure that we're actually looking at all the things we're supposed to look at and not just assuming that it's already been done.
[00:37:54] Unknown:
Yeah. And you can't assume when someone's life's on the on the line. Right? You can't just say, we'll see if in production, if we get a bug, we'll fix the bug. And a lot of developers tell me 2 things. 1 is if you do this this way, you'll make less features, and the response is, yeah, of course, that makes sense. Maybe you need less features. You don't need all the features in the world in certain systems. And I think there's also, like, a fear of it slowing down things and adding more documentation, but there is a point to that and I think I'm very much against having documentation as a sec of documentation. I think you need to have it in a way that makes sense for your organization, for your product, for the risk level, but I also think that without that, without the ability for someone to step in and see what you're doing and audit you, there are just bad incentives. And and everything here, we're just, you know, in 2023, we forget how it used to be before this regulation, but people are selling literal snake oil, telling people that their kid will be cured of of polio if they just drink olive oil. We've gone a long way from there, but it's not easy to do that. Right? There is I heard someone say it very well.
Said that the best companies in tech might have better quality than the best companies in medical devices, but the average company in tech has way worse quality. So and that's what the regular is concerned about. Overall, am I improving the quality of things?
[00:39:16] Unknown:
From your experience of entering this space, building the product around KetchX, what are some of the ideas or assumptions that you had about the potential space for solutions that have been changed or updated as you worked through with customers and came to understand more about the overall, problem space?
[00:39:36] Unknown:
I think 1 is just how hard it really is to build, regulated web based applications. Machine learning just adds more complexity to it that people often run kind of against the wall and say no one's ever gonna regulate machine learning models and high risk applications because of the statistical nature. Fundamentally, I disagree with that assumption because we regulate, biopharmaceuticals and biopharmaceuticals are produced in a statistical manner. They're grown. They're not made. And if we can regulate that, we could really regulate everything. And this goes into cancer patients, you know, direct IV. And so I think that the 1 assumption I had is it can't be that complicated. Well, it is. Some assumptions I had were where can we reduce it, and there are places to reduce it, but there's also places that you just totally understand why they're asking for it.
And the third is that there would be some limit to regulating AI systems, but I don't think that's true. I think there just needs to be a lot of subject matter expertise and knowledge that there's not enough people who know. And that's what software tooling is made for. Right? It's made to automate subject matter expertise and make it easier for people for less people to do more.
[00:40:44] Unknown:
And in your work of helping customers gain control of their overall development cycles, reduce some of the operational burden of developing these systems and help with some of the automation aspects and just understanding what their requirements are? What are some of the most interesting or innovative or unexpected ways that you've seen ML used in the medical space?
[00:41:08] Unknown:
There's so much of it going on right now. I think, most innovative. Most innovative is hard. I think some of what people are doing with images is amazing. I think the way it's been used, even like linear models or multi linear models for a long time is pretty pretty cool and useful. Right? People think about regulated AI. There have been approved machine learning applications in medicine since the late eighties, early nineties, just not what we do now. I think what it's gonna do for home care, for people that live in places that have less access to health care is gonna be transformational. And I think that's to me is why I'm I'm so excited about machine learning is we're gonna take a lot of things that people know and no 1 else knows, and we're gonna make it more accessible to people to increase the level of care. Because at the end of the day, the growth of population, is outpacing the growth in providers, physician, nurses, and so on.
And we need to find a way to reduce the cognitive load it takes to perform certain medical interventions and then the amount of time physicians spend on it. And I think there's a lot of amazing stuff going on in surgical robotics as well regarding this. How do you make a robot that's just easier to do surgeries with To just make it less burdensome to learn how to make how to do surgeries and and I'm pretty excited about that. And I think it all ties together to, we want systems and software to automate tasks in our life and reduce the challenge of doing that, and machine learning is just kind of the 1 of the most advanced stages of doing that.
[00:42:42] Unknown:
And as you have been working in this space and building the KETRIX product and business, what are some of the most interesting or unexpected or challenging lessons that you've learned?
[00:42:53] Unknown:
I've learned that there's amazing people out there in the world that are spending their life building really important medical applications, and they're yearning for a way to do it better and faster and right. And I think that was really exciting to me is to meet a lot of dev teams that are not just trying to do this, but are trying to do this really, really, really well and are thinking of patients all the time and how to make good companies to do that with and and are excited about what we're building. Right? It's always exciting where you have this idea. You're sitting in some company while other people care. And and so things that seeing that they do care and also the overall acceleration and the way the regulator in the US and the EU has taken on software over the last 24 months, I think, is very exciting. There have been more movement in the last 24 months or 12 months in software than there has probably in the last 30 years.
The real big set of regulations came out in the early 2000, but even they were just less robust than what's going on now. And we're seeing it all over. We're seeing the EU start to really enforce a lot of things around software, whether that's GDPR or or many other things or the EU AI Act or MDR, medical price regulations, or IVDR, as well as a lot of clinical trial regulations for data. And we're also seeing the same thing in the US, and and it excites me to see that everybody's seeing the impact software can have on everybody's lives and want to make sure that that's gonna happen in a reasonable, efficient, and safe way. And it really excited me to just see how many people are thinking of that desire to help other people, save lives. And I just think it's limitless what's gonna happen in medicine. We're gonna see a transformation over the next 20 years, that's gonna reduce the the cost of things and also give access to treatment that's extremely advanced to all kinds of places in the world.
[00:44:45] Unknown:
And for people who are building or thinking about building products in this space of medical devices, medical applications, trying to address all of the regulatory aspects of it, what are the cases where Ketrix is the wrong choice?
[00:44:58] Unknown:
If you're building software in medicine and you're not using Ketrix, you've already made the wrong choice. And I think every person that works for us will say, this is the only way to do it. I think if you're building certain embedded systems that take a long while to develop, it could be in some ways less attractive or more attractive. But I think if you're building complex systems and and we even have people designing hardware on top of KETRIX now just because the system is so intuitive and the AI in it is so helpful. But I I I really you know, that's my belief in we're building a company to just help people that build software and medicine do it faster, you know, at 10% of the cost and 10% of the time.
And I think that in 5 years from now, we're gonna see that it's way over that. I think that the amount of the amount of money people are spending to build medical software is outrageous, is outrageous and it is a shame for us as a society And it is a shame for the millions of patients out there in the world that need treatment and need funding for devices that are not getting it. And we're gonna reduce that by so much that every entrepreneur and every innovator that wants to start sees something in the clinic and wants to start a company never thinks again about it, but, I don't really wanna do medical devices because it's hard. And I think that's part of a broader story of automating society that we're gonna we're gonna be part of on how do you do it in other spaces. But I think if I was for my first time trying to develop this, I would go and read FDA's good principles of software validation and understand what's even been asked here and then figure out how to go from there because most people just wanna build and they don't think about the regulatory aspect of things and regulatory strategy and you can save a lot of time doing it right early on and not fixing it later.
Yeah. And I think our best users are leaders because it's, just over time, what happens is if you don't do it well, if your company doesn't go anywhere, it doesn't matter. But if it goes, the company is really successful, it'll matter a lot. This might be the only thing that matters ever in your company at the end of the day.
[00:46:53] Unknown:
And as you continue to build and iterate on the Ketrix product, what are some of the things you have planned for the near to medium term or any particular problem areas or projects you're excited to dig into?
[00:47:05] Unknown:
More platforms, more ecosystems, more community around it, a bigger way to collaborate. I think that in coming months years, you're gonna see a lot of collaboration features that help in a wide variety of ways people, work together to make products, in a totally distributed fashion and across companies and across the supply chain. That's something I'm very excited about because there's a very robust world of people that help you make biologics or pharmaceutical agents or hardware medical devices, but it doesn't exist in software and it's kind of sad because software is the easiest way to do that. Right? I can just ship someone code. And so we're gonna have a big focus on that and how people work together to make stuff.
And,
[00:47:48] Unknown:
yeah, it excites me. It's all about people collaborating together and building ideas together and then making sure those ideas are are working really well. Right? That's all the FDA really asks for. Are there any other aspects of the work that you're doing at Ketrix, the product and platform that you're building, or the overall space of building software and machine learning applications for the medical space that we didn't discuss yet that you'd like to cover before we close out the show?
[00:48:14] Unknown:
Not really, except maybe the fact that people need to focus on the overall product, and there's a tendency to focus on machine learning as a product. I don't think the vast majority of models are not products. They're building blocks for other products, and people need to start thinking about it that way. Machine learning as a model generally would be a feature. LLMs are more complicated than that, but they're generally a feature unless you're chat GPT and there's not gonna be a lot of other chat GPTs. You're a feature within a broader context, and you need to understand how that broader context looks and works.
[00:48:45] Unknown:
Well, for anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest barrier to adoption for machine learning today.
[00:49:00] Unknown:
Fundamentally, it's subject matter expertise as it has been for any new technology ever. Some of it is resources, but I think there's also a lot of great resources, technical resources, like compute and the United States has decided now to create a central governing resource for AI research and and share that across communities and university. I think it's very exciting, but I think it's all about knowledge, and software is also about knowledge, but things are progressing very fast. And if it's hard to find someone to build a model, how about someone who understands those models well enough to explain that to a regulator?
Right? And then Albert Einstein always said, if you can't explain it to someone who doesn't know what you're talking about, you probably don't know well enough. That's all the FDA is asking for, show us that you know it well enough that you can explain it to us. And I think that's going to be the challenge in machine learning overall and the challenge in machine learning in medicine because it just kind of over the machine learning complexity. You get another layer of regulatory complexity, another layer of software development and DevOps and all those together. You know, how many people know how to develop web based machine learning products that are validated to medical regulations?
Not
[00:50:12] Unknown:
many. But how many of those do we need? A lot of people. Well, thank you very much for taking the time today to join me and share the work that you and your team are doing to simplify and, reduce the overhead of working in this space. Definitely appreciate the time and energy you're putting into that, and I hope you enjoy the rest of your day. Yeah. Thank you, Tobias. It's been a real pleasure. I enjoyed it.
[00:50:37] Unknown:
Thank you for listening, and don't forget to check out our other shows, the Data Engineering podcast, which covers the latest in modern data management, and podcast dot in it, which covers the Python language, its community, and the innovative ways it is being used. You can visit the site at the machine learning podcast.com to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email hosts at the machine learning podcast.com with your story. To help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.
Introduction and Guest Introduction
Erez Kaminsky's Background and Journey into Machine Learning
Regulatory Burdens for ML in Medical Applications
Software and Data Supply Chain Complexities
Operating Environment and System Hardening
Risk Evaluation Frameworks
Human in the Loop and Model Confidence
Regulatory Impact on ML Model Development
Design and Development of Ketrix Platform
Change Management and Model Evolution
Challenges of Building Regulated Software
Lessons Learned and Assumptions Challenged
Innovative Uses of ML in Medical Space
Interesting Lessons from Building Ketrix
When Ketrix is the Wrong Choice
Future Plans for Ketrix
Final Thoughts and Barriers to ML Adoption
