(206) Design Process for a Machine-Learning Opioid Overdose Prediction and Risk Stratification E-Tool in Electronic Health Record System

Background: The United States opioid public health crisis continues with ~80,400 opioid overdoses in 2021.

Objectives: We aimed to design and evaluate a machine-learning (ML) opioid overdose prediction and risk stratification clinical decision support (CDS) tool in the University of Florida Health (UFHealth) electronic health records (EHR) system.

Methods: Using a user-centered design approach, we conducted 4 in-depth expert-user-feedback panel sessions, with 2-4 UFHealth primary care provider (PCP) participants per session and with each session covering a different topic (i.e., workflow, risk assessment, risk mitigation, and design ideas). Based on the panel’s feedback, we refined the CDS tool, tailoring the alert’s contents and recommended actions to meet the providers’ needs. Then, we conducted 2 sessions of usability testing (4 interviews/session) with additional PCPs, adjusting the CDS tool based on feedback after each wave.

Results: We held 14 interviews and conducted one independent building opportunity (design ideas) with 13 PCPs (62% male, 85% White race, number of patients prescribed opioids per month = 1 – 80). In the panel sessions, PCPs (n=5) identified flaws and suggestions and expressed trust in the tool. Panel sessions revealed that indicating that a patient is high risk could result in referral to addiction services or pain management. In the first session of the usability testing, PCPs (n=4) missed that the alert was triggered by an ML algorithm and the details-page link, acknowledged the importance of including the level of risk (i.e., 1 patient in 125 at high risk of overdose) to draw their attention to the alert, requested a function enabling them to double check their opioid order, and requested to see the alert more than once a year (e.g., every 6 months) per at risk patient. In the second session of usability testing, PCPs (n=4) appreciated the UFHealth specific alert and believed the alert would be beneficial for new or unfamiliar patients. They raised concerns regarding burden on providers who prescribe frequently to high-risk patients and suggested incorporating the details page in training sessions. Some confusion was identified regarding override reasons and suggested actions.

Conclusions: Incorporating in-depth expert user feedback in CDS development identified critical barriers and facilitators for ML-enabled alerts and resulted in a streamlined prototype with higher likelihood of use in clinical settings designed to mitigate unintended impacts on patients. The usability sessions resulted in fine tuning of the alert to increase future acceptance and usability in clinical practice.