Is it possible to predict COVID-19 trends?

A new preprint available on medRxiv* reports the results of the second part of a two-phase forecasting study on coronavirus disease 2019 (COVID-19) cases and deaths in two countries, Germany and Poland. The study comes from the German and Polish COVID-19 Forecast Hub.

Study: National and subnational short-term forecasting of COVID-19 in Germany and Poland, early 2021. Image Credit: whiteMocca/ Shutterstock

Background

The study was carried out by 15 separate teams of investigators who predicted the incidence of COVID-19 one to four weeks in advance, through January to mid-April 2021. This period saw many non-pharmaceutical interventions (NPIs) implemented for weeks at a time, making it easier to make short-term predictions.

In Poland, however, daycare centers, hair salons, and sports centers were closed, among others, in a set of new restrictions to the existing ones, on March 27, 2020.

Vaccination rollouts in both countries occurred at the beginning of the study period. Only one-sixth and one in twenty had received one or two doses, respectively, by the end of the same period.

What did the study show?

This pre-registered study sought to evaluate forecasts of COVID-19 cases and deaths in Germany and Poland. Ensemble models were associated with high-performing predictions compared to individual or baseline models. Also, contributed models out-performed simple models that carried forward the last observation, in almost all cases, showing themselves capable of forecasting horizons for up to four weeks.

The improvement in accuracy in this second round of forecasting should be judged in terms of the differences in the study period, compared to that of the earlier part of the same study, from October to December 2020. At that time, NPIs were characterized by their fluidity, which meant their trajectory could not be properly judged.

During the earlier period, cases remained more or less stable, which accounted for the success of the last-observation-carried-forward method of evaluation of the baseline model. In the second period, cases moved up or down for longer periods, making the current model more relevant than the baseline.

Conversely, the reliability of the extrapolation approach was not markedly increased by many models, despite including multiple factors. This may be because of all the ways to set parameters for a simple baseline model. One of them is likely to agree with the observational findings at least to an acceptable level. The authors, therefore, prefer the use of a few specified baseline models.

In Germany and Poland, the number of cases first dropped and then went up again because of the emergence of the Alpha variant. Deaths, however, went down throughout most of the study period in Germany but remained stable for a long time before increasing in Poland. Most of the teams produced a better forecast than a simple baseline model up to four weeks in advance.

The best models appeared to be the ensemble methods. Nonetheless, the researchers observed difficulty in predicting larger trends in the caseload reported. While the predictions were made at three- or four-week lead periods, short-term forecasts are most accurate, though not short-term scenarios.

When the results were pooled for both evaluation periods, the researchers found that ensemble forecasts were well-adjusted for deaths, even over longer lead periods, better than baseline or individual models. The prediction for deaths was less accurate, performing higher than the others only at one or two weeks ahead.

Death forecasts were satisfactory in terms of their reliability, compared to baseline models, but for cases, the coverage was sub-par from two weeks in advance. In fact, ensemble scores were below those of the baseline model in accuracy at three to four weeks in advance.

This strengthens our previous conclusion that meaningful case forecasts are only feasible at very short horizons."

Recently, the US COVID-19 Forecast Hub published results showing that ensemble case forecasts were not meaningful beyond a week.  

The ability to predict changes in the trends of cases or deaths is a key area examined in this study. The findings show that case trends are poorly forecast, unlike death trends. This agrees with other studies on short-term forecasts of COVID-19 in real-time, in the US and UK, for instance.

What are the implications?

Epidemiological time series are theoretically forecast by mechanistic or statistical models. In the first case, recent trends, NPIs in force, the population's behavior in the study, and vaccine coverage are all implemented to predict the changes in trends, based on the decrease in susceptibles or the changes observed or even expected in the surrounding factors.

This approach, used with case trend forecasts, finally is reduced to the extrapolation of current trends if care is not taken to factor in changes in NPIs. As such, this work confirms the poor ability at present to predict turning points in such curves, as has been stated theoretically already.

Two microsimulation models, MOCOS-agent1 and ICM-agentModel, successfully predicted a decline in case numbers in Poland. Though this is a positive sign, such examples are rare.

In the second case, a leading indicator is related statistically to the trend of the thing measured. The changes in the trend of this indicator could help establish how it will move in the future, as well. The prediction of deaths falls in this group, using cases and hospitalizations as the leading indicators.

Additional leading indicators such as mobility, insurance claims, or web searches have been tested for their ability to improve prediction success, but so far, not much headway has been made. Another possible advance is incorporating sequencing data, helping predict trends by including the stable directions of growth for different circulating variants.

This is still hampered by the delays in sequencing and reporting, especially if the prediction is to be made in crucial periods.

Once an epidemic occurs, forecasts of coming trends are extensively discussed in the media. However, as this study shows, it is difficult to forecast such turning points, which are uncommon in the general course of the epidemic.

If these trends are evaluated at the front and center of discussion, it may be useful only for those models that tend to predict change points. It is complex to adjust scoring rules to underline these events while adhering to sound theoretical principles – also called "the forecaster's dilemma."

Conclusion

The current study reports the final results of the independent German and Polish COVID-19 Forecast Hub. Thereafter, a European COVID-19 Forecast Hub was set up in April 2021 by the European Center for Disease Prevention and Control (ECDC), and the former has been synchronized with this Hub. The European Hub has the advantage of including hospitalization forecasts, unlike the earlier version.

It is already receiving papers from over 30 teams, including all forecast papers submitted to the earlier Hub and vice versa. The first Hub still receives forecasts at a regional level, which are not incorporated into the European Hub.

This effort shows the benefits of collaboration between forecasting researchers using combined ensemble predictions as a key parameter. When integrated with ongoing surveillance of the forecasting accuracy, the system may still be more useful in planning public health policies and strategies.

*Important notice

medRxiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.

Written by

Dr. Liji Thomas

Dr. Liji Thomas is an OB-GYN, who graduated from the Government Medical College, University of Calicut, Kerala, in 2001. Liji practiced as a full-time consultant in obstetrics/gynecology in a private hospital for a few years following her graduation. She has counseled hundreds of patients facing issues from pregnancy-related problems and infertility, and has been in charge of over 2,000 deliveries, striving always to achieve a normal delivery rather than operative.