The challenge of predicting forecast errors represents one of the more nuanced problems in applied data science, and the struggles outlined in this post will resonate with anyone who has attempted to model noise itself. The core difficulty here is philosophical as much as technical: when your target variable is fundamentally the difference between a prediction and reality, you are essentially building a model to capture randomness. The author has done everything right from a methodological standpoint—careful feature engineering, proper time-based validation splits, and thorough leakage prevention—yet finds themselves with a model that predicts near-zero for everything. This is not a failure of implementation; it is a structural challenge that deserves deeper examination.

What makes this problem particularly thorny is the inherent signal-to-noise ratio. Forecast errors, by their nature, represent what the original forecasting model failed to capture. If those original forecasts were at all reasonable, the errors should be centered around zero with relatively low variance—which is exactly what the author observes. The Random Forest, trained to minimize error, has essentially learned the most statistically defensible answer: when in doubt, predict the mean. This conservative behavior is a feature of tree-based methods, not a bug. They excel at partitioning signal but struggle when the signal itself is weak or distributed across interactions too complex to capture with standard splitting rules. The diagnostic detail that std(predictions) / std(target) reaches only 0.4 is telling: the model is capturing some structure, but it is systematically under-dispersed.

Several avenues merit exploration beyond the current approach. First, quantile regression or gradient boosting with explicit quantile loss could help the model capture tail behavior rather than just the conditional mean. Second, the author might consider whether the problem is better framed as classification—predicting the sign or magnitude category of the error rather than its exact value. Third, there may be value in explicitly modeling the heteroscedasticity of errors: horizon-specific models or hierarchical approaches could allow different horizons to have different variance structures rather than forcing a single model to average across them. The target encoding for horizon and month is a good instinct, but it may be smoothing away precisely the variation that matters most.

The broader implication here is about the limits of supervised learning when applied to residuals. We often treat prediction errors as just another target to model, but they occupy a peculiar statistical space: they are what remained after one modeling attempt already extracted the learnable patterns. This does not mean the problem is impossible—operational meteorology teams routinely calibrate ensemble forecasts using precisely this kind of error analysis—but it does suggest that the solution may require moving beyond standard regression frameworks. As forecasting systems become more sophisticated and the marginal value of each improvement decreases, the challenge of modeling their residual errors only grows. The author's experience is a useful case study in recognizing when a model is doing exactly what it should, even when the results feel disappointing.

Hey everyone,

I am kind of desperate for help right now on my current project. I'll try and be as clear as possible.

I'm working on a time series backcasting problem. The values I want to backcast are forecasts (not ML forecast, but think of weather forecasts) at different horizon (from 1 to 14). So to be clear, at a date D, I have 14 forecasts (forecast at D+1,..., D+14). I have such forecasts from 2020 to 2026 (each row represents a day, each (date, horizon) key is unique). So I have 14 dates duplicated as blocks because each row consists of on unique(date, horizon) -> target_date. I hope this is clear enough.

So the goal is to backcast those forecasts before 2020 (say 2019-2020 for simplicity). Besides forecasts values and horizon columns, I have "actuals" that are the true measured values for a particular variable (say temperature), and "normals" which is a smooth curves representing the climatology norm for a particular data. This "normals" column captures the seasonality, trend, and every other repetitive and predictable patterns.

So to be clear I have :

* dates (of forecast emission) | actuals | normals | horizon | forecasts *

And to really emphasise this point : dates, actuals and normals are the same for 14 consecutive rows (One row equals one horizon).

The target I want to predict is the following : forecast - actual_at_forecast_date

So i want to predict the true error observed (say i had predicted 20 (forecast) for today and I measure 18 (actual) then my target is +2).

So far, I've done the following :

- Transform target to remove annual seasonality, long-term trend and level-scaling

- Engineered classic features such as anomaly (actual-normal), lagged anomalies, rolling stats (std, mean, median, quantiles)

- Engineered target encoding features such as target_encoding_horizon_x_month

- RandomForest with max_depth 10-15, min_leaf 10, max features "sqrt", n_estimators 300

My train/val folds are reversed because I wanted to best evaluate on a backcasting framework. I made sure there is no leakage.

FINALLY:

My main problem is that, even with a LOT of features combination, trying a LOT of tuning, my prediction is very shallow and shrinking to the mean (the std and q10, q90 are off by a lot). So given I try to predict forecast_error which is centered on 0, I start to think that I only capture noise because my predictions really won't fit anything. MAE is getting worse with higher horizon forecasts which is only natural but even for horizon 1 my prediction is as good as predicting only 0s MAE-wised. Please if anyone has ideas that I can explore on my own I would be so grateful. I know you don't have all the details here but if you have experience with backcasting and has some recommendations I would be so grateful.

Hey everyone,

I'm working on a time series backcasting problem and I'm running into a fairly stubborn issue. I'd really appreciate any insights from people who have worked on similar setups.

Problem setup

I have daily-issued forecasts with multiple horizons:

At each date D, I have forecasts for D+1, ..., D+14
Data spans 2020–2026
Each row is a unique (forecast_date, horizon) pair

Toy example:

forecast_date	horizon	target_date	forecast	actual	normal
2023-01-01	1	2023-01-02	20	18	19
2023-01-01	2	2023-01-03	21	20	19
...	...	...	...	...	...
2023-01-01	14	2023-01-15	25	23	20

Important:

forecast_date, actual, and normal are identical across the 14 horizons
Only horizon, target_date, and forecast vary

Objective

I want to backcast forecast errors before 2020.

Target:

target = forecast − actual(target_date)

So if forecast = 20 and actual = 18 → target = +2.

Features

forecast, horizon
actual, normal
anomaly = actual − normal
lagged anomalies
rolling stats (mean, std, quantiles)
target encoding (e.g. horizon × month)

Model

Random Forest:

max_depth: 10–15
min_samples_leaf: 10
max_features: sqrt
n_estimators: 300

Validation

Time-based splits adapted for backcasting
No leakage (checked carefully)

Main issue

Predictions are very shallow and collapse toward 0:

Very low variance
Poor estimation of tails (q10 / q90)
Even for horizon = 1, performance is close to predicting constant 0 (in MAE)

MAE increases with horizon (expected), but overall performance remains weak.

Diagnostics

std(predictions) / std(target) ≈ 0.4 at best
This ratio decreases with horizon

So the model is clearly under-dispersed.

Interpretation

At this point I suspect:

either the signal is very weak
or the model is too conservative and fails to capture amplitude

Any help, feedback, or ideas to explore would be greatly appreciated.

Thanks a lot.

submitted by /u/Ambitious-Log-5255
[link] [comments]

Tagged with

#rows.com

Backcasting forecast errors: model collapsing to mean [P]