The question
Every month, Eurostat asks European households whether they intend to make a major purchase in the next twelve months. Every month, it also publishes how much was actually bought. The first is what people say; the second is what they did. The obvious question is whether the saying leads the doing.
It is a better question than the one we asked before. Our earlier tests used music as the predictor and failed five times, and we published all five. That piece ended by saying a predictive tier would reopen only on a future pre-registered test that actually cleared. This is that test.
Stated intent is the closest thing there is to a stated version of realised behaviour. If anything we hold were going to lead spending, it was this.
What we tested
We pre-registered the whole specification before fitting anything, and amended it nine times as building the data corrected our own assumptions. Every amendment is published with its date and reason, and eight of the nine were made before any result was seen.
| Element | Locked |
|---|---|
| Predictor | Big-ticket purchase intent, as a gap to each market’s own expanding-window mean |
| Target | Retail trade volume, month on month, seasonally adjusted |
| The bar | The best cheap public alternative — the Economic Sentiment Indicator, industry confidence, mortgage rates, and the series’ own history |
| Sample | 120 out-of-sample months per market, 2015 to 2026, COVID excluded |
| Comparison | Nested — the two models differ by exactly one column |
The bar matters more than it looks. Beating a coin flip proves nothing; the test had to beat what a buyer could already do for free, and the Economic Sentiment Indicator is a demanding opponent because it already contains consumer confidence as a component.
The result
| Market | Change in forecast error | 95% confidence interval |
|---|---|---|
| France | +0.00% | Exactly zero |
| Germany | −1.49% (worse) | −3.92% to +1.01% |
Germany’s interval rules out any improvement of 2% or more. It does not rule out an improvement of under 1%, which would not change anybody’s decision. Neither market cleared the bar, so by the rules we had set in advance we did not run the other markets at all — running them after a failure would have been shopping for one that agreed.
Why France came back as exactly zero
An identical result is suspicious, and we have been caught by it before. An earlier test produced two nulls matching to five decimal places, which turned out to be a bug that made two different models return the same answer. It would have faked a clean negative result.
If results look suspiciously identical, suspect a swallowed exception, not a clean null.
So we checked rather than accepted. The predictor was in the model and did vary. In all 120 windows, the model’s own feature selection set its weight to zero — and in all 120 windows it set every other weight to zero too, falling back to predicting the long-run average. Identical errors are the correct consequence of that, not an error. The test now refuses to report a result at all if the models differ while the errors do not.
That audit turned up something larger than the null. If the model discards every predictor and still beats the alternatives, then the best available forecast of next month’s retail volume is simply its average growth rate. The difficulty is in the target at this horizon, not only in the predictor.
What this does not prove
A forecasting model that fails does not establish that a market cannot be forecast. We tested one predictor, on one target, at one horizon, with one model family, in two markets. Other targets, longer horizons, different transformations and richer information sets are untested, and several are plausible.
We also want to be precise about what a negative result is worth. A test that fails to find an effect is not the same as a test that proves there is none. What the confidence intervals above let us say is bounded: no improvement of 2% or more in Germany, and a predictor that the model rejected outright in France. That is a bounded negative result, not a proof of absence, and we have corrected our own pre-registration where it claimed otherwise.
What it changes
One commitment, and it is deliberately narrow: Cadence will not sell a forecasting capability that has not demonstrated incremental value over the public alternatives a buyer could already use for free. That is a decision about what we sell, not a claim about what is knowable.
A read, not a forecast. Now tested on the signal we actually sell, not only on one we no longer use.
The wider question stays open to genuinely different hypotheses, each with its own pre-registration published before it runs. What is closed is this hypothesis, and the habit of selling a capability before it has earned the claim.
References
- Pre-registration, all nine amendments, the results memo and the code are published with the Cadence methodology.
- Eurostat ei_bsco_m (BS-MP-NY, seasonally adjusted) and sts_trtu_m (VOL_SLS, G47, seasonally and calendar adjusted), 1985 and 1994 onward respectively.
- Diebold, F. X., & Mariano, R. S. (1995). Comparing predictive accuracy. Journal of Business & Economic Statistics, 13(3).