The calibration history is the right thing to insist on, and I would be specific about what you ask for. A provider quoting an error metric on their own back-test is nearly meaningless. What you want is the realised coverage of their published intervals over a long period: if the 90% band contained the outcome 90% of the time, the distribution is usable, and if it contained it 70% of the time, the distribution is decoration. The uncomfortable part is that most forecast products are tuned to minimise average error, which systematically narrows the intervals. The number that sells is accuracy on the median. The number you trade on is honesty at the tails, and those two objectives pull against each other.

narrow bands look confident and confident sells. same failure everywhere

Worth adding that spatial correlation matters as much as the marginal distribution for a portfolio. Every site being individually well-calibrated tells you nothing if the model treats them as independent, because the whole risk in wind is that a stalled front takes an entire region out at once. That is the scenario the position is exposed to and it is the one an ensemble with too little spread will never generate.

空间相关这条提醒得对，我之前只逐个站点看校准，确实忽略了整体。回头把同一天多个站点的误差拉出来算一下相关性，如果模型里假设是独立的，那我算出来的组合风险就是系统性偏低的。