How do you measure a demand plan?

Measuring a Demand Plan involves five core elements: Time, level cycle, data, and calculation. All these elements have to come together to truly measure a demand plan. Let's look at each of these elements in turn...

Time is the place in time or ‘lag’ at which the demand plan is measured. This core axis is used to measure short, medium, and long term demand plans. Let's look at an example. The time you are in is called current and each bucket of time into the future is called a Lag.

So the next period is Lag1, then Lag2 and then Lag3 and so on. The higher the lag, the further out the plan. As plans are completed the Lags can be collected and grouped together, so that the business can measure the quality of the plan for different lags. Using these lags, we can measure short, mid, and long-term plan quality.

In this example Lag1, 2 and 3 are part of the short term horizon plan quality measurement. The next element is Level. This is the data aggregation level at which the demand plan is measured. This axis is used to measure low, mid or higher levels of the demand plan where the more detailed information is at the lower level and more aggregated is at the higher level.

Let us look at an example: There are often many hierarchy levels in a demand plan and the measurement level depends on the horizon and granularity of the decision being taken. Although accuracy drops as you look further into the future it increases as you aggregate to higher levels of analysis Post game including exception generation is usually done at lower levels to highlight the problem areas that need specific planner attention. KPI or Key Performance Indicators are generally performed at middle levels This is to summarise data for easier decision making. Management KPIs that reveal how well a plan is performing against overall strategic targets are calculated at higher levels such as category, channel and quarter.

The next measuring element is Cycle. This is tied to the demand planning process phases, activities and steps that repeat on a regular basis (whether daily, weekly or monthly). Cycle information enables the measurement of cycle over cycle volatility in the demand plan or how much the plan is changing for the same period in the future from one cycle to the next This is also known as the Cycle over Cycle Change and is often overlooked. In this example the forecast for April is changing repeatedly through each forecast cycle.

A changing demand signal creates re-work and uncertainty. Measuring and mitigating forecast variation from cycle to cycle will reduce instability and bring confidence to the users of the Demand Plan. The fourth element is Data. Measurement is always the activity of comparing two or more data streams.

Let’s look at some of the data streams that are used to measure the Demand Plan. Comparing Plan to Actuals will measure how well you can interpret and predict customer demand. Selecting these data elements will measure the quality of your predictions. This is the most common data comparison used by organisations to Measure Prediction Quality.

Comparing Plan to Target will measure how the forecast is tracking against the annual budget and whether strategic plans are being met. Select these data elements to Measure Plan Attainment. Comparing Consensus to System Forecast will measure if the overrides being made are improving or hurting the overall plan. So choose these data streams to Measure Forecast Value Add and understand if the forecast process is healthy or not.

And comparing Plan from last cycle to the latest cycle will measure the stability of the plan. Select these data elements to Measure Forecast Volatility and determinine if your plans are introducing bullwhip and creating a lack of trust.

Finally we come to the last element: the Calculation or formula that is used to measure a Demand Plan.

Let’s describe the most common ones: Mean Absolute Deviation or MAD this measures the spread of data from the average error. A lower value indicates greater accuracy and a higher means worse. A simple, entry level calculation. Mean Absolute Percent Error or MAPE provides a simple measurement of accuracy in % terms by totalling up all the variances.

A lower percentage indicates higher accuracy. Commonly used this measurement treats all observations equally and works best if there are no extremes to the data (and no zeros!) Weighted Mean Absolute Percentage Error or WMAPE improves on the MAPE measurement by using volume weights to determine the significance of error when totalling up the variances. This helps when data is intermittent or variable such as when we aggregate high runners together with low runners.

Root Mean Squared Error or RMSE is the square root of the mean average of the square of all the errors and is an estimator for the standard deviation of the distribution of errors. n other words, it tells you how concentrated the data is around the line of best fit. Bias Variance is a delta measurement and shows at aggregate levels how “off” we are from the actuals. This is for planners to understand if their forecast was too high or too low.

Bias Trend is a tracking signal that measures the trend of the forecasting error. This is to identify if any Bias is a one-off event or if there is a consistent (or persistent!) pattern of over or under forecasting. Forecast Value Add or FVA measures the improvements that collaboration and updates make to the forecast.

This identifies if Planners are making good changes or not. Best practice is to calculate this against Statistical and a Naive Forecast to evaluate if your starting point was good too. Cycle over Cycle Change or COCC measures the drift of the plan by specific time buckets from one planning cycle to the next. This is useful for honing in on what has changed and resolving forecast instability.

In summary then, the 5 elements of time, level, cycle, data and calculation have to be combined to enable the measurement of a demand plan. and measurement of a demand plan provides a benchmark of current performance versus targets and improvements in these measurements will lead to lower stock outs, higher fill rates and improved customer satisfaction.