AI Risk Management Strategies for Healthcare

The EU AI Act isn’t theory anymore — it’s live law. And for Medical AI teams, it just became a business-critical mandate. If your AI product powers diagnostics, clinical decision support, or imaging you’re now officially building a high-risk AI system in the EU. What does that mean? ⚖️ Article 9 — Risk Management System Every model update must link to a live, auditable risk register. Tools like Arterys (Acquired by Tempus AI) Cardio AI automate cardiac function metrics. They must now log how model updates impact critical endpoints like ejection fraction. ⚖️ Article 10 — Data Governance & Integrity Your datasets must be transparent in origin, version, and bias handling. PathAI Diagnostics faced public scrutiny for dataset bias, highlighting why traceable data governance is now non-negotiable. ⚖️ Article 15 — Post-Market Monitoring & Control AI drift after deployment isn’t just a risk — it’s a regulatory obligation. Nature Magazine Digital Medicine published cases of radiology AI tools flagged for post-deployment drift. Continuous monitoring and risk logging are mandatory under Article 61. At lensai.tech, we make this real for medical AI teams: - Risk logs tied to model updates and Jira tasks - Data governance linked with Confluence and MLflow - Post-market evidence generation built into your dev workflow Why this matters: 76% of AI startups fail audits due to lack of traceability. The EU AI Act penalties can reach €35M or 7% of global revenue Want to know how the EU AI Act impacts your AI product? Tag your product below — I’ll share a practical white paper breaking it all down.

Are you curious about how to create safe and effective artificial intelligence and machine learning (AI/ML) devices? Let's demystify the essential guiding principles outlined by the U.S. FDA, Health Canada | Santé Canada, and the United Kingdom’s Medicines and Healthcare products Regulatory Agency (MHRA) for Good Machine Learning Practice (GMLP). These principles aim to ensure the development of safe, effective, and high-quality medical devices. 1. Multi-Disciplinary Expertise Drives Success: Throughout the lifecycle of a product, it's crucial to integrate expertise from diverse fields. This ensures a deep understanding of how a model fits into clinical workflows, its benefits, and potential patient risks. 2. Prioritize Good Software Engineering and Security Practices: The foundation of model design lies in solid software engineering practices, coupled with robust data quality assurance, management, and cybersecurity measures. 3.Representative Data is Key: When collecting clinical study data, it's imperative to ensure it accurately represents the intended patient population. This means capturing relevant characteristics and ensuring an adequate sample size for meaningful insights. 4.Independence of Training and Test Data: To prevent bias, training and test datasets should be independent. While the FDA permits multiple uses of training data, it's crucial to justify each use to avoid inadvertently training on test data. 5. Utilize Best Available Reference Datasets: Developing reference datasets based on accepted methods ensures the collection of clinically relevant and well-characterized data, understanding their limitations. 6. Tailor Model Design to Data and Intended Use: Designing the model should align with available data and intended device usage. Human factors and interpretability should be prioritized, focusing on the performance of the Human-AI team. 7. Test Under Clinically Relevant Conditions: Rigorous testing plans should be in place to assess device performance under conditions reflecting real-world usage, independent of training data. 8. Provide Clear Information to Users: Users should have access to clear, relevant information tailored to their needs, including the product’s intended use, performance characteristics, data insights, limitations, and user interface interpretation. 9. Monitor Deployed Models for Performance: Deployed models should be continuously monitored in real-world scenarios to ensure safety and performance. Additionally, managing risks such as overfitting, bias, or dataset drift is crucial for sustained efficacy. These principles provide a robust framework for the development of AI/ML-driven medical devices, emphasizing safety, efficacy, and transparency. For further insights, dive into the full paper from FDA, MHRA, and Health Canada. #AI #MachineLearning #HealthTech #MedicalDevices #FDA #MHRA #HealthCanada

World Health Organization's latest report on 𝐫𝐞𝐠𝐮𝐥𝐚𝐭𝐢𝐧𝐠 𝐀𝐈 𝐢𝐧 𝐡𝐞𝐚𝐥𝐭𝐡𝐜𝐚𝐫𝐞. Here’s my summary of key takeaways for creating a mature AI ecosystem. 𝐃𝐨𝐜𝐮𝐦𝐞𝐧𝐭𝐚𝐭𝐢𝐨𝐧 𝐚𝐧𝐝 𝐓𝐫𝐚𝐧𝐬𝐩𝐚𝐫𝐞𝐧𝐜𝐲: In the development of health AI systems, developers should maintain detailed records of dataset sources, algorithm parameters, and any deviations from the initial plan to ensure transparency and accountability. 𝐑𝐢𝐬𝐤 𝐌𝐚𝐧𝐚𝐠𝐞𝐦𝐞𝐧𝐭: The development of health AI systems should entail continuous monitoring of risks such as cybersecurity threats, algorithmic biases, and data model underfitting to guarantee patient safety and effectiveness in real-world settings. 𝐀𝐧𝐚𝐥𝐲𝐭𝐢𝐜𝐚𝐥 𝐚𝐧𝐝 𝐂𝐥𝐢𝐧𝐢𝐜𝐚𝐥 𝐕𝐚𝐥𝐢𝐝𝐚𝐭𝐢𝐨𝐧: When validating health AI systems, provide clear information about training data, conduct independent testing with randomized trials for thorough evaluation, and continuously monitor post-deployment for any unforeseen issues. 𝐃𝐚𝐭𝐚 𝐐𝐮𝐚𝐥𝐢𝐭𝐲 𝐚𝐧𝐝 𝐒𝐡𝐚𝐫𝐢𝐧𝐠: Developers of health AI systems should prioritize high-quality data and conduct thorough pre-release assessments to prevent biases or errors, while stakeholders should work to facilitate reliable data sharing in healthcare. 𝐏𝐫𝐢𝐯𝐚𝐜𝐲 𝐚𝐧𝐝 𝐃𝐚𝐭𝐚 𝐏𝐫𝐨𝐭𝐞𝐜𝐭𝐢𝐨𝐧: In the development of a health AI systems, developers should be well-versed in HIPAA regulations and implement robust compliance measures to safeguard patient data, ensuring it aligns with legal requirements and protects against potential harms or breaches. 𝐄𝐧𝐠𝐚𝐠𝐞𝐦𝐞𝐧𝐭 𝐚𝐧𝐝 𝐂𝐨𝐥𝐥𝐚𝐛𝐨𝐫𝐚𝐭𝐢𝐨𝐧: Establish communication platforms for doctors, researchers, and policymakers to streamline the regulatory oversight process, leading to quicker development, adoption, and refinement of safe and responsible health AI systems. 👉 Finally, note that leaders should implement the recommendations holistically. 👉 A holistic approach is essential for building a robust and sustainable AI ecosystem in healthcare. (Source in the comments.)

Last week, an AI medical record summary failed to capture critical information about my dad's condition and next steps in his care. Why do AI tools sometimes "hallucinate" lab results or omit critical context? There are many known (and unknown) risks with AI tools in healthcare and most of these risks are embedded at the research and development phase. 🔍 That means that it is in this phase that scrutiny is warranted because once it's deployed into clinical workflows it's too late. Yet in so many conversations about AI risk in research, I still hear: 💬 “The only real risk is a data breach,” or 💬 “AI is just basic statistics, like regression.” The worst excuse I've ever heard was: 💬 "Doctors make the same mistakes all the time." These statements concern me, and hopefully they concern you too. While I agree many AI tools are relatively low risk, not all are. For example, deep learning and GenAI tools used to summarize patient records can behave in unpredictable and non-linear ways. These #ComplexSystems operate in dynamic, high-stakes clinical environments. They can have real-world consequences for patients and #ResearchParticipants. ⚠️ A small prompt tweak or formatting change in a generative AI summary tool can ripple into misdiagnoses, missed safety alerts, or inappropriate clinical decisions. These aren’t random bugs; they emerge from complex system interactions, like: 🫥 FEEDBACK LOOPS reinforce incorrect predictions Examples: --> “low-risk” labels lead to less monitoring; --> Using AI to screen certain groups for study eligibility but historical screening has systematically excluded minority groups and non-English-speaking ⚖️ EMBEDDED/HISTORICAL BIASES in training data amplify health disparities across race, gender, or disability. 📉 DATA DRIFT: evolving EHR inputs cause the model to misinterpret new formats or trends. 🥴 HALLUCINATION: Fabricating patient details or omitting critical nuances due to token limits or flawed heuristics. ... and so much more... ⚠️ These risks affect patient and research participant safety and jeopardize #ResearchIntegrity. 🏨 If institutions adopt these tools without recognizing their system-level vulnerabilities, the consequences can be profound and hard to trace. That’s why research institutions need: ✅ More technical and algorithmic audits. ✅ Governance frameworks that translate these complex behaviors into plain-language, IRB-ready guidance that centers safety, ethics, and compliance. ✅ To demystify the system-level risks behind these tools. 💡 Fortunately, there's a solution💡 With the right SMEs, we can craft practical, plain-language approaches to improve #IRB review and ethical oversight. Is anyone else working on this at the IRB level? I’d love to compare notes (or maybe even partner on the work!?). #AIinHealthcare #ComplexSystems #IRB #GenerativeAI #ClinicalAI #DigitalHealth #ResponsibleAI #AIEthics #HRPP #AIHSR #SaMD

I’m catching up with my podcasts from last week after being at the #Alliance2024. Everyday AI's episode last Wednesday about AI Governance (link in the comments) is an absolute must listen for companies starting to think about how to incorporate AI into their workflows. Gabriella Kusz shared lots of actionable steps including: Acknowledge the Challenge: Recognize the fast pace of AI advancement and how it outpaces traditional regulatory or standards development processes. Take Action Internally: Proactively form a dedicated task force or working group to focus on AI governance. Multi-Departmental Collaboration: This task force should include representatives from various departments (medical writing, continuing education, publications, marketing, etc.) to provide a range of perspectives on potential risks and benefits. Educate Your Team: Provide team members with resources on AI, generative AI models, and consider regular updates or "brown bag" sessions to stay up-to-date. Start Small, Define Boundaries: Select early use cases with low, acceptable risk levels. Define ethical boundaries for AI deployment even before starting pilot projects. Learn From Mistakes: Embrace an iterative process where pilot projects offer learning opportunities. Adjust approach as needed rather than seeing any initial setbacks as failures. We, as an industry, need to step up and start creating internal rules for ethical AI use, especially for sensitive medical/healthcare content. What resources are you using to stay updated on AI ethics and responsible use in medical communications? In what ways do you think AI could positively transform medical writing and communication? Let's share ideas! #healthcare #medicalwriting #AIethics

Recently, Stanford rolled out a new AI model to help physicians and nurses work together. I was glad to see this shift, as AI was initially limited to improving diagnoses but can do much more. It’s what we imagined with midoc.ai, an integrated health system that collaborates with physicians and nurses to improve patient care. Nurses and clinicians often can’t always keep a close eye on Vital signs. They have regular intervals, but keeping a short loop is sometimes impossible. However, this new algorithm at Stanford Hospital reviews the data every 15 minutes and gives a risk score. Here’s how I think it’s bringing a positive impact onto the scene: → Improved Communication: The model facilitates efficient communication between nurses and physicians. By generating alerts,  it prompts timely discussions about patient care, which might otherwise be delayed due to the busy hospital environment. Initially, the model alerted staff when patients were already deteriorating. Adjustments were made to predict severe outcomes, like ICU transfers instead. This change has led to better proactive care. → Clinical Impact:  In a study involving almost 10,000 patients, Those identified by the AI as at high risk saw a 10.4% reduction in deterioration events (like ICU transfers), which is particularly beneficial for those on the cusp of high risk. → Response to the Model: The reception among healthcare professionals has been generally positive despite some concerns about alert fatigue. Efforts are ongoing to refine the model’s accuracy to boost its reliability and the staff's trust in its predictions. The team at Stanford aims to improve the model's accuracy to enhance trust and effectiveness in preventing patient deterioration. – What are your thoughts on this model?

In the realm of medical devices, adopting a preventive strategy within #riskmanagement is potentialy preferred to a reactive approach that comes into play when it may already be too late. By proactively identifying and addressing potential risks and their probability of occurrence , medical device manufacturers, healthcare providers, and patients can avoid costly and detrimental consequences before they occur. Yes, I’m a true believer that RWE and patient perspective has to be included in risk management analysis. Preventive strategies within a cross functional team involve robust risk assessment protocols, stringent quality control measures, and ongoing monitoring to mitigate risks at every stage of a medical device's lifecycle. This approach not only ensures compliance with regulatory standards but also safeguards patient safety and promotes trust in the healthcare ecosystem. In contrast, a stand alone reactive approach limits the ability to anticipate and prevent risks, often leading to regulatory questions, product recalls, patient and/or end user injuries, and damage to an organization's reputation. By the time reactive measures are implemented, irreversible “harm” may have already been done, resulting in legal liabilities, financial losses, and compromised patient outcomes. Ultimately, by prioritizing a preventive strategy in risk management for medical devices, stakeholders can uphold the highest standards of quality, safety, and efficacy by bolstering continuous improvement and innovation in healthcare.