What do we want to achieve?

The immunization supply chain

Source: Effective Vaccine Management Assessment (EVM), 2021

Vaccines

Map of Kenya

Hierarchical structure of vaccine data

Classical approaches to forecast vaccine needs

• Demographic methods (developed by WHO, one-size-fits-all model)
  • expected target population
  • coverage: is the expected coverage rate
  • doses/target: is the number of doses per target, as per the national vaccination schedule
  • wastage factor

Forecasts are annual and at the national level

Limitations of current forecasting method

❌ Forecasts are yearly and at the national level(not useful to inform operational / tactical decisions.)

❌ Forecasts are often produced based on unrealistic assumptions.

❌ Forecasts do not acknowledge uncertainty.

❌ Do not capture the information available at multiple hierarchical levels.

❌ Forecasts ignore the hierarchical nature of the problem, not coherent.

❌ Lead to conflicting decisions & lack of coordination.

How to forecast hierarchical time series?

Data

• Vaccine consumption in Kenya from January 2013 until December 2021

• Four type of vaccines

  • Measles
  • Bacillus Calmette–Guérin (BCG) - for tuberculosis
  • DPT, a class of combination vaccines against three infectious diseases in humans: diphtheria-tetanus-pertussis
  • OPV, Oral poliovirus vaccines, used in the fight to eradicate polio

• 306 sub-county, 47 county & 9 regions

• Total of 1452 time series

Doses used (raw data): total and regions

Doses used: National & regions

Doses used: trend and seasonality features

Forecasting setup

• Forecasting method: Exponential Smoothing State Space models
• Forecast horizon: 12 months
• Point and probabilistic forecasts are generated and evaluated for the entire hierarchy

• Forecast evaluation

  • Time series cross-validation with re-estimation
  • Used 36 months as test set

Forecasting performance metrics- point forecast

Forecasting performance metrics- probabilistic forecast

Overall forecast accuracy

Forecast accuracy improvement against the current approach in immunization programs is calculated but not included in the presentation

Conclusions

Benefit of hierarchical forecasting

✅ Plans at any level are based on coherent forecasts and therefore can be aligned.

✅ Hierarchical forecasting framework can be used as a tool to improve coordination between teams across the supply chain at the national, sub-national, regional and local levels.

✅ Result can be more accurate than the independent (base) forecasts.

✅ Hierarchical forecasting framework an be used to create coherent forecast, regardless of how base forecasts are created, even with judgmental forecasts.

Next steps

• Collect data on potential useful predictors such as child population, conflicts, strike, weather conditions, etc

• Build new forecasting models incorporating strong driving factors of vaccine demand

• Evaluate the implication of forecast accuracy on utilities such as cost, service level, coordination, etc by linking to replenishment policies.

Acknowledgement

• John and Snow Inc. (JSI) team

• The representative of the National Vaccine Immunization Program in Kenya

• Developers of Tidyverts and Tidyverse packages

References

• Effective Vaccine Management (EVM 2.0) Assessment Report – 2021, WHO

• Vaccine Logistics, WHO training manual

• • [Probabilistic Forecast Reconciliation For Emergency Services Demand]([https://robjhyndman.com/seminars/fem.html], (Bahman Rostami-Tabar, Rob J Hyndman]

• [Forecasting: Principles and Practice]([https://otexts.com/fpp3/hierarchical.html], (3rd ed, Rob J Hyndman and George Athanasopoulos]

Linear reconciliation methods, Bottom-up and others

Forecast reconciliation approaches combine and reconcile all the base forecasts in order to produce coherent forecasts.

Linear reconciliation methods (Wickramasuriya, Athanasopoulos, and Hyndman 2019) can be written as

$$\large \tilde{{y}}_h = {S}({S}'{W}^{-1}{S})^{-1}{W}^{-1}\hat{{y}}_h ={S}{G}\hat{{y}}_h = {M}\hat{{y}}_h,$$

where ${W}$ is an $n \times n$ positive definite matrix, and $\hat{{y}}_h$ contains the $h$-step forecasts of ${y}_{T+h}$ given data to time $T$.

Producing probabilistic forecasts

• We use bootstrapping to generate probabilistic forecasts:

  • Suppose that $(\hat{{y}}_h^{[1]},\dots,\hat{{y}}_h^{[B]})$ are a set of $B$ simulated sample paths, generated independently from the models used to produce the base forecasts.
  • Then $({S}({S}'{W}^{-1}{S})^{-1}{W}^{-1}\hat{{y}}_h^{[1]},\dots,{S}({S}'{W}^{-1}{S})^{-1}{W}^{-1}\hat{{y}}_h^{[B]})$ provides a set of reconciled sample paths, from which percentiles can be calculated.

Reconciled forecasts

• This approach involves first generating independent base forecast for each series in the hierarchy (i.e. Base)

• As these base forecasts are independently generated they will not be “aggregate consistent” (i.e., they will not add up according to the hierarchical structure). They are not coherent.

• The reconciliation approaches combine the independent base forecasts and generates a set of revised forecasts that are as close as possible to the univariate forecasts but also aggregate consistently with the hierarchical structure.

• Unlike any other existing method, this approach uses all the information available within a hierarchy.