(119) Performance of the US Food and Drug Administration’s COVID-19 Disease Severity Categorization for Use in Real-World Data

Background: The US Food and Drug Administration (FDA) issued a guidance to assist sponsors in the clinical development of drugs for treatment or prevention of COVID-19, which included a 5-level disease severity categorization (asymptomatic, mild, moderate, severe, critical).

Objectives: To validate an adapted 5-level disease severity code-based algorithm for use in real-world administrative data against medical record review.

Methods: We sampled 250 community-dwelling Medicare beneficiaries ages ≥18 years with a U07.1 code on an inpatient hospital or ambulatory (outpatient and professional services) Medicare claim between April 1, 2020, and June 18, 2022. We used a stratified sampling approach based on care setting and COVID-19 treatment status to ensure sufficient sample size in each severity category, and adjusted results using sampling weights. Comparing against medical records, we calculated positive predictive values (PPVs) and 95% confidence intervals (CIs) for each severity category, and, to assess the overall ordinal performance, weighted kappa (WK) and macro-averaged mean absolute error (MAE). We conducted pre-specified secondary analyses to improve algorithm performance by using (1) refined severe and critical disease definitions and (2) a combined asymptomatic and mild category.

Results: We received at least one full medical record corresponding to the overall COVID-19 disease episode for 190 (76%) of the 250 sampled beneficiaries. Of these, 158 had a valid positive COVID-19 test and sufficient medical record information to be classified into one of the 5 disease severity categories. The primary analysis resulted in low PPVs except for the moderate category (71%, CI 43-89%). The algorithm with refined severe and critical definitions achieved a high PPV in only some categories, and low-to-moderate PPV in the others: 22% PPV (CI 13-35%) for asymptomatic, 26% PPV (CI 14-44%) for mild, 75% PPV (CI 46-91%) for moderate, 53% PPV (CI 34-71%) for severe, and 83% PPV (CI 56-95%) for critical; the WK and MAE were 55% (CI 47-64%) and 0.65, respectively. When we combined asymptomatic and mild categories, PPV was 77% (CI 66-84%), and WK and MAE were 71% (CI 63-80%) and 0.45, respectively.

Conclusions: The original algorithm demonstrated generally poor performance, while secondary analyses using refined logic achieved moderate performance. Critical and moderate disease may be the most reliably discernible levels. While the asymptomatic and mild categories are difficult to accurately distinguish, their combination achieves a moderate-to-high performance. Researchers should use caution when using administrative claims data to assess COVID-19 disease severity.

Disclaimer: The views expressed are the authors’ and not necessarily those of FDA, CMS, or the VA.