Portolan Operations decision desk
Open Portolan
All guides
Guide · Demand forecasting

Forecast bias, accuracy and value added: what to measure and what to do about overrides

Judge a forecast against a simple benchmark at the horizon of the decision it serves, track bias separately from error, and measure every override against the baseline before deciding whether to keep it. Combinations of simple methods have beaten single complex models in large competitions, and judgmental adjustments help on average at some firms and hurt at others, so the answer is to measure, not to ban or to trust.

Published · Updated

Accuracy depends on the decision

A forecast is an input to a decision, so accuracy only means something relative to that decision: its horizon, the level of detail it needs and the cost of being too high versus too low. A weekly staffing decision and a quarterly purchasing decision need different forecasts. Write the decision down first; otherwise a leaderboard number tells you little.

Bias is not the same as error

Error says how far forecasts miss; bias says whether they miss in the same direction. The playbook lists them as separate measures: forecast bias as the mean signed error over a rolling period, which shows systematic over- or under-planning, and error relative to a seasonal-naive benchmark at the decision's horizon, which shows whether a method earns its complexity.

Inventory policy makes the link concrete: the right safety stock depends on forecast error and lead-time variability, not only on average demand. If you have not measured those, that comes before a software purchase.

Benchmark simple methods before buying complex ones

Across 100,000 series and 61 methods in a large forecasting competition, the strongest results came from combinations of approaches rather than a single complex model, and pure machine-learning entries generally lagged. A later competition on retail sales found some machine-learning methods more accurate than statistical benchmarks, but on a large, well-instrumented retail dataset with many related series. Competition results may not carry over to your data, so the practical step is a holdout test of a simple, well-chosen combination against any sophisticated model on your own series.

Governing overrides

Across more than 60,000 forecasts at four firms, judgmental adjustments improved average accuracy at three of them, but small adjustments and upward adjustments often made forecasts worse. Whether an override helped depended on the firm, the product and the direction of the change.

The evidence points to structure rather than prohibition: allow overrides for material exceptions, record the reason and direction, and compare the adjusted forecast with the baseline so each planner and each kind of override has a track record. Judgment is not the enemy of forecasting; unmeasured judgment is.

The path, step by step

These steps come from the Demand forecasting playbook. Each has a checkpoint you can observe.

  1. Name the decision the forecast serves

    Write down the decision, its horizon, the level of detail required and the cost of being high versus low. A weekly staffing decision and a quarterly purchasing decision need different forecasts.

    CheckpointA one-page forecast brief: decision, horizon, granularity, error cost and current baseline.

    Common failureOptimising statistical accuracy on a series that no planner actually uses.

  2. Establish a reproducible baseline

    Build a seasonal-naive benchmark and one exponential-smoothing or ARIMA challenger. Backtest both on a holdout of at least two full seasonal cycles before touching the live plan.

    CheckpointOut-of-sample error and bias for each model, at the horizon the decision actually uses.

    Common failureChoosing a model by training fit, or by how sophisticated it sounds.

  3. Govern overrides with exception rules

    Allow human edits only for named exceptions, with a reason code, direction and owner. Compare the edited forecast with the untouched baseline every month.

    CheckpointOverride rate, reason mix, and forecast value added for edited versus unedited forecasts.

    Common failureLetting the loudest planner edit anything, then blaming the model when the plan misses.

  4. Run a monthly forecast review that retires complexity

    Review error, bias and value added by model and override. Delete models, hierarchy levels and reports that do not improve a decision.

    CheckpointA short value-added table and a decision log showing what was removed, kept or retested.

    Common failureAdding a new model every quarter while nobody can name the one the business plans on.

What this guide does not settle

The overrides evidence is four firms in supply-chain planning and the competitions use particular datasets and metrics. Accuracy measures the forecast, not the downstream cost of its errors, which is why the decision comes first.

What the research found

Each finding below is quoted from Portolan's research library, with the limits the library records for it. Applying a result from another setting, such as a hospital or a factory, to yours is an analogy, not a proof.

Makridakis et al. · 2020Established

The M4 Competition: 100,000 time series and 61 forecasting methods

Across 100,000 series and 61 methods, the strongest results came from combinations of forecasting approaches rather than a single complex model. The best hybrid was close to 10% more accurate than the combination benchmark, while pure machine-learning entries generally lagged.

Where it stops. Competition datasets and metrics shape the ranking; a method that wins on M4 series may not win on your granular, hierarchical or short-history data.

Makridakis et al. · 2022Experimental

M5 accuracy competition: Results, findings, and conclusions

In a competition built on 42,840 hierarchical Walmart sales series, some machine-learning methods produced less biased and more accurate forecasts than established statistical benchmarks. The result is conditional on a large, well-instrumented retail dataset with many related series.

Where it stops. The competition used one retailer's dataset; the organisers themselves caution that competition results may not generalise, and simple methods can remain competitive at fine granularity.

Fildes et al. · 2009Established

Effective forecasting and judgmental adjustments: An empirical evaluation and strategies for improvement in supply-chain planning

Across more than 60,000 forecasts at four firms, judgmental adjustments improved average accuracy at three firms, but small adjustments and upward adjustments often made forecasts worse. The value of an override depended on the firm, the product and the direction of the change.

Where it stops. Four firms in supply-chain planning, not a universal sample. The study measures accuracy, not the downstream cost of forecast error.

Fildes & Goodwin · 2007Established

Against Your Better Judgment? How Organizations Can Improve Their Use of Management Judgment in Forecasting

Judgmental adjustments to statistical forecasts can improve accuracy, but they are often biased by optimism, anchoring and a desire to please. Organisations get better results when forecasts are structured, assumptions are recorded, and forecasters receive feedback on their track record.

Where it stops. The evidence synthesises forecasting experiments and field studies; effects depend on the incentives and feedback culture around the forecast.

Silver et al. · 2016Established

Inventory and Production Management in Supply Chains

Inventory policy is a trade-off between ordering, holding and shortage costs, and the right safety stock depends on forecast error and lead-time variability, not just average demand. Segmentation by value and uncertainty is more effective than a single policy applied to every item.

Where it stops. This is a textbook synthesis of established models; it assumes reasonably stable demand and reliable data, which many operations lack.