This website does not support Internet Explorer. Please, switch to Edge, Chrome, or Firefox browser to view this page.

Learn about IE support
Best Practices, Knowledge

Measuring forecast stability: Applying the right metric

Business concept. business people discussing the charts and grap
Published: Reading time: 6 min
Stefan de Kok Principal Data Science Architect
Stefan de KokPrincipal Data Science Architect
Published:

Forecast stability measurement

When judging the quality of a forecast, one aspect that is generally ignored is its stability. As explained in my last blog, stability measures how much the forecast changes from one forecast cycle to the next. This blog briefly overviews an intuitive stability metric and how to apply it.

Note on terminology: when speaking of “item,” it refers to a combination of product and location, at any level of aggregation, unless a specific aggregation is indicated. A time series exists as the combination of an item across multiple consecutive periods.

Cycle-over-Cycle Change

Stability in forecasting means the forecast for any item in any given period does not change dramatically between one forecasting cycle to the next. The opposite of that—instability—measures the amount of change between cycles. The Cycle-over-Cycle Change (COCC) is a straightforward metric that measures instability, like a WAPE for error. Its formula is identical to that of WAPE, with two differences:

  • 1.

    Instead of comparing a forecast to an actual, it is compared to a prior forecast.
  • 2.

    It ignores values that do not exist in both forecasts. 

Where t is the periods measured, i is the items (series) measured, c and p are the current forecasting cycle and any given prior forecasting cycle, respectively, and Ip,c is an indicator which is 1 if a value exists in both the current and previous cycles and 0 otherwise.

Basically, it is a sum of absolute changes divided by the sum of the prior forecast for an estimate of the relative change. The forecasts compared within the absolute operator are always for the same item and period. Their difference is their lags. For example, a forecast for item A for March 2022 generated in February will be compared with a forecast for item A for March 2022 generated in January. One is lag 1, the other lag 2. A forecast for March would never be compared to another month.

The above gives a percentage change between two forecasting cycles (any two, these do not need to be consecutive). The same logic can be extended to cover many successive cycles. Where p = c-1, and c iterates through as many cycles as desired. In this case, each of those iterations could provide its own percentage to be tabulated, or an average of those percentages can be used for one overall verdict.

For troubleshooting, the exact percentages can be calculated at any level of aggregation to allow a drill down into the detail where changes are gravest. If more than two forecasting cycles are involved, the average percentages need to be performed separately for meaningful values at each such level of aggregation.

Note: extremely high values are expected for intermittent demand at the detail level. Drill down into detail will only be helpful up to a certain depth. Instead, only the numerator may need to be considered at the most granular levels to assess impact fairly. This is similar to evaluating errors, where a WAPE at some level of aggregation may need to be replaced by MAE in detail for meaningful insights.

How to apply

Like a MAPE or WAPE, COCC will be sensitive to the level of aggregation where the absolute operator is applied. Rather than a problem, this is an opportunity. With the same formula, you could measure the total change over a whole year, the average weekly change within a year, the average monthly change within a year, or the change between two specific cycles.

  • To compare impact per period across two cycles, determine the appropriate level of item aggregation, keep time in the granularity of the forecast, and apply the formula to that period only. The first summation (over t) disappears.
  • To compare average change per period across multiple periods include the absolute values for all the applicable periods and sum over those periods. For an average monthly change, use monthly granularity. For an average weekly change, use weekly granularity. You can do this over any horizon, such as a typical lead time or a whole year.
  • For a total change across a whole year, the period is sized to a year or any 12-month or 52-week period, and the absolute value is taken by item for that single period. In such a case, a slight deviation from the formula can also be made where the periods included in the year total may not necessarily be the same. For example, you could compare the total for Jan-Dec to the total of Feb-Jan. This is not perfect but can provide quick insights. 

The level of item aggregation can also be chosen depending on how the forecast is used. If budget or capacity gets reserved three months out by month, brand, and region based on the forecast, change should be measured at that granularity. In this case, we would sum across three months after taking the absolute values by month, brand, and region. For most other purposes, the appropriate item granularity will likely be the same granularity where the accuracy of the forecast is determined.

The gap between consecutive forecasts may also be considered. For example, if accuracy matters at lags W+2 and W+4, we may want to compare lag W+2 with lag W+4 for the same periods rather than measure change between consecutive weekly cycles. In this case, current forecast c will be at W+4, while prior forecast p will be at W+2 in the formula.

Conclusion

Stability is an aspect of forecast quality that is often ignored. But as I hope to have illustrated here, it is remarkably simple to calculate. There is no practical reason not to measure it. The only thing requiring some thought is at which granularity and across which time horizon to calculate it—precisely the same as you would do to determine forecast accuracy.

Not all change is bad. If information changes, so should a forecast. However, it is surprisingly common that forecasts change much more dramatically than any change in the underlying data would warrant. This signals that overfitting is likely occurring and contributes to the bullwhip effect felt on the supply chain. Both are unwelcome. Both can be identified by measuring the COCC of your forecast and then taking corrective action.

O9 pov supplier collaboration mockup

Multi-tier supply chain collaboration and risk management

Transform your supply chain with real-time collaboration, visibility, and innovation.

I agree to the privacy policy.

About the author

Stefan de Kok Principal Data Science Architect

Stefan de Kok

Principal Data Science Architect

Stefan de Kok consults with o9 Solutions as a Principal Data Science Architect. He has 25 years experience in supply chain planning, optimization and forecasting, across a wide range of industries. As an applied mathematician he works mainly on model design and remediation, algorithm development, and metric design for quality measurement. He is recognized as an innovator on these topics especially in the area of probabilistic planning and forecasting. He is an active blogger and is currently writing a textbook on these topics.

Resources

View our related articles, white papers, use cases & videos

article10 min

What is supply chain?

by o9 Solutions
Highway intersection aerial
Guide

10 Steps for Setting Up an Effective Transformative IBP Process

O9 ibp cheatsheet mockup 3
video

Why Tracy Allen Advocates for Women Leadership in Tech

27 min
Why tracy allen advocates for women leadership in tech thumbnail
video

Solving Supply Chain Challenges with the o9 Digital Brain

22 min
Solving supply chain challenges with the o9 digital brain thumbnail
video

The Fundamentals of Demand Planning Cycles

4 min
The fundamentals of demand planning cycles thumbnail
video

The Future of Data: A Fireside Chat with Filippo Passerini

15 min
The future of data: a fireside chat with filippo passerini thumbnail