Prediction models for COVID-19 are under development worldwide: which people are especially at risk, which patients with complaints have it, and which characteristics and test results determine the course of the disease? A study of 31 models underlines the need to share more data. “Most predictive models are not yet reliable enough to base medical decisions upon.”
Investigators Dr Laure Wynants of Maastricht University/KU Leuven and Dr Maarten van Smeden of UMC Utrecht are concerned. Together with a group of international investigators, they assessed all available prediction models for the early detection and disease course of coronavirus disease 2019 (COVID-19). They conclude now that some models contain important information for healthcare providers, but that most models are currently based on insufficient scientific evidence. Their paper has been published in the leading medical science journal British Medical Journal (BMJ).
“Good predictive models are urgently needed,” says Maarten. “General practitioners and specialists in hospitals are now using different models under high pressure. Which people are at greater risk of getting COVID-19? Which patients with disease symptoms actually have a COVID-19 infection? And what is the expected course of disease in patients with COVID-19? Specifically, you can think of a model that estimates the risk that a 35-year-old man with shortness of breath actually has a COVID-19 infection. Sometimes it takes too long before you as a doctor get a test result and you want to have confirmation sooner. In such a situation, prediction models come into view, but they are not yet sufficiently accurate to base a medical decision upon. ”
In addition to the individual patient, predictive models are also important for healthcare as a whole. Laure Wynants: “COVID-19 is an acute threat to global public health, with numbers of infections and deaths increasing daily. Since the outbreak at the end of last year, the pandemic has threatened almost all national health systems with overload.” That is why early detection and predictions about disease course are essential to deploy the correct prevention, diagnostics and therapy. The more targeted the effort, the more efficient the care as a whole. Wynants: “For example, you want to prevent patients from ending up in the hospital unnecessarily, but also that patients are wrongly sent home and have to be admitted later.”
The investigators examined 31 predictive models from 27 international studies, most of them (25) from China. Data for these studies were collected between December 8, 2019 and March 15, 2020. Many of these studies have serious shortcomings. “Non-representative control patients, for example,” says Maarten, “and data sets that are too small.” The latter may be a result of the rush to come up with a prediction model. In view of the emergency, it is understandable, he thinks, but this does not alter the fact that the result is scientifically invalid. “For example, there was a study that examined how long patients remained in hospital. When the study was ended after fourteen days, the patients who were still admitted were excluded from the analysis.” Logical consequence: the model, based on this study, underestimates how long COVID-19 patients are hospitalized.
The research did reveal a number of predictive factors that are important for healthcare and healthcare policy. Wynants: “Age is such a factor and gender, for example, but also certain lab values such as C-reactive protein and lactate dehydrogenase.” Still, the forecast models – on average – are based on insufficient scientific evidence. In order to improve quickly, more data is now needed. That is why the researchers call on developers of prediction models to immediately share data publicly: “Share COVID-19 patient data, because only then can we develop, test and apply reliable and widely applicable prediction models in daily practice. The investigators believe that an online location coordinated by an organization such as the WHO is urgently needed.
The investigators are continuing the current study as a so-called living review, with a biweekly update in the BMJ. The aim is to continuously provide healthcare providers and policymakers with up-to-date information about the quality of COVID-19 related prediction models. They will continuously add new available models to their studies with the updates and assess them critically. In doing so, they not only examine the official publications of forecasting models in scientific journals, but also studies that are only available on the Internet in so-called ‘pre-print‘ archives. The models from these ‘pre-prints’ are often already available for use by healthcare providers.
Wynants L, Calster B van, Bonten MJM, et al. Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal. British Medical Journal 2020;369:m1328
