ML/AI In Medical Devices
Introduction
The integration of Artificial Intelligence and Machine Learning (AI/ML) into medical devices represents a stimulating frontier in MedTech innovation. AI/ML offers a broad spectrum of applications, including, but not limited to, the diagnosis of medical conditions utilizing medical images, automated digital pathology, and continuous health monitoring. While AI/ML has had a path for FDA approval for many years now, many are asking whether Agentic AI is next.
There are many applications where LLMs are already providing value in a medical context, outside of medical devices. These include conversational interfaces for patients, voice transcription and summarization of clinical visits, automation of documentation, etc.
As with any medical device, development must occur under a robust software engineering process with documentation that demonstrates safety, efficacy, and cybersecurity. This article explores the process required for FDA approval for traditional ML in medical devices and examines how that might apply to Agentic AI.
Approach Overview
The regulatory landscape for AI/ML in medical devices consists of one clear regulatory pathway built around traditional ML. AI/ML in a medical device must be developed with good machine learning practices (GMLP) as defined by the IMDRF. This guidance applies to the FDA and Health Canada. There are clear expectations from the FDA for demonstrating safety and efficacy. The FDA has gone a step further and released a draft guidance document listing their expectations for AI-enabled devices.
Agentic AI and LLMs, however, present unique challenges due to their general-purpose training, complex multi-component architectures, and potential for autonomous changes over time, requiring new regulatory considerations that are still being developed.
Traditional ML Regulatory Pathway
The traditional ML regulatory pathway centers around good machine learning practices (GMLP). One of the highlights of the GMLP guidance is the requirement to demonstrate that training and test datasets are collected independently and represent the intended patient population. This often requires carefully defining and executing a data collection strategy.
Another highlight of the guidance is demonstrating that subject matter experts are involved throughout the process. This usually means that clinical experts are labeling data used to train the AI/ML model, and having multiple independent and qualified labellers will be required as part of a successful submission.
Agentic AI: Challenges and Regulatory Uncertainty
The regulatory pathway for Agentic AI in medical devices is currently unclear. The primary challenges from a regulatory perspective are both design controls around the data inputs used for model training and development as well as defining how to monitor and maintain the safety, efficacy, and performance of these applications that change on their own accord over time. This would require a significant investment in post-production monitoring and analysis as well as the potential introduction of unforeseen risks. The FDA has yet to approve any Agentic AI or LLM based solution as part of a medical device. While the GMLP guidance above may be applied to LLMs and Agentic AI systems, there are still a lot of unanswered questions.
Differences in Training and Validation
The training and validation process for LLMs is quite different from traditional ML. LLMs are often trained on large amounts of text data from multiple sources in order for them to generalize language well enough to perform many types of tasks. They are then improved through reinforcement learning with human feedback (RLHF), which is where humans test the model in various scenarios, providing feedback that is then used to further train the model. This feedback is in the form of text – it could be a document summary or a clinical explanation. The RLHF process is much more involved than data labeling used in traditional ML, and the whole training process typically costs tens of millions of dollars.
AI providers that have followed this process, such as OpenAI and Anthropic, have created models that perform well in many different tasks: mathematical reasoning, text summarization, writing, etc. These models are often applied to many specialized use cases with techniques such as fine-tuning, retrieval augmented generation (RAG), Reason-Act (ReAct) agentic techniques, etc.
Multi-component System Validation
Therefore, in practice, Agentic AI systems consist of many components that are built on top of powerful, general use LLMs. They are often evaluated as a whole, rather than the underlying LLMs. Treating the LLMs as one would a traditional, highly specialized ML model, would miss the many other components that affect the safety and efficacy of the solution.
Future Outlook
Currently, the FDA is inviting proposals to develop clinical AI agents for cardiovascular health. It is expected that within three years, more information will be released on any specific requirements for Agentic AI. It is not clear whether the same GMLP will be used to evaluate the solution or whether some requirements will be adjusted to reflect the different nature of LLMs relative to traditional ML solutions.
Ultimately, while traditional Machine Learning models have a clear, albeit rigorous, regulatory pathway for FDA approval, the landscape for Agentic AI and LLM-based medical devices remains uncertain and challenging. The core principle of demonstrating safety and efficacy under robust software development and Good Machine Learning Practices (GMLP) still applies. However, the unique architectural differences of LLMs, including their general-purpose training, reliance on RLHF, and their integration into complex multi-component agentic systems, suggest that the current GMLP framework may require adaptation. As the FDA continues to engage with this emerging technology, driven by initiatives like ARPA-H’s focus on clinical AI agents, developers must prioritize transparency, incorporate strict human oversight and safety guardrails, and prepare for a future where regulatory requirements will evolve to match the distinct capabilities and risks of Agentic AI in healthcare.
