{"paper_id":"2511.13616","title":"Statistical and economic evaluation of forecasts in electricity markets: beyond RMSE and MAE","authors":["Katarzyna Maciejowska","Arkadiusz Lipiecki","Bartosz Uniejewski"],"paper_date":"2025-11-17","paper_type":"empirical","paper_url":"https://arxiv.org/pdf/2511.13616v1","page_url":"https://thequant.space/flowcharts/statistical-and-economic-evaluation-of-forecasts-in-electric/","scores":{"empirical_rigor":8,"math_complexity":5.5,"quadrant":"Holy Grail"},"proposes":{"status":"available","statement":"The paper proposes two new measures describing dispersion of errors and association of forecasts to capture economic value for BESS arbitrage.","quote":"We propose and examine two measures describing dispersion of errors and association of forecasts, which reflect statistical properties of the electricity price forecasts that are particularly relevant for the BESS decision-making process.","summary":"In recent years, a rapid development of forecasting methods has led to an increase in the accuracy of predictions. In the literature, forecasts are typically evaluated using metrics such as Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE). While appropriate for statistical assessment…","strategy":"BESS arbitrage strategy"},"assumptions":{"status":"available","items":[{"assumption":"The battery completes one full charge–discharge cycle per day and does not transfer energy between consecutive days.","type":"execution","stated":"explicit","quote":"Specifically, we assume that the battery completes one full charge–discharge cycle per day and does not transfer energy between consecutive days."},{"assumption":"No additional constraints on the minimum or maximum state of charge are imposed.","type":"execution","stated":"explicit","quote":"Furthermore, no additional constraints on the minimum or maximum state of charge are imposed, although such limits are typically required in real‑world applications to extend battery lifetime."},{"assumption":"Battery degradation is not modeled explicitly; it is indirectly reflected through reduced operating costs.","type":"execution","stated":"explicit","quote":"Finally, given the relatively short evaluation period considered in this study, battery degradation is not modeled explicitly within the decision problem; instead, it is indirectly reflected through reduc…"},{"assumption":"BESS earns income solely from day‑ahead market arbitrage, buying low and selling high.","type":"market","stated":"explicit","quote":"It is assumed that BESS earns income from the arbitrage in DA market. BESS aims at buying energy in periods of low prices and selling at hours of high prices."},{"assumption":"Operating costs are a levelled cost per unit of energy throughput, incorporating investment, O&M, and degradation.","type":"model","stated":"explicit","quote":"The operating costs and battery efficiency are taken from [6] and represent the levelled cost per unit of energy throughput. This takes into account investment and O&M costs, as well as implicitly incorporating battery degradation through the assumed number of charge–discharge cycles over the lifetime of the battery."},{"assumption":"The BESS specifications (energy capacity 3 MWh, power ratings 3 MW and 1 MW) are representative of grid‑scale lithium‑ion batteries.","type":"model","stated":"explicit","quote":"Both batteries are assumed to have the same energy capacity of 3 MWh. They differ in terms of the power ratings that are 3 MW and 1 MW, for BESS‑a and BESS‑b respectively."}]},"datasets":{"status":"available","items":[{"name":"German‑Luxembourg day‑ahead electricity price","provider":"ENTSO‑E transparency platform","kind":"market-prices","access":"public","period":"2016-01-02 to 2024-12-31","universe":"Germany–Luxembourg bidding zone (Austria included until Oct 2018)","quote":"The core series is the hourly day‑ahead price for the Germany–Luxembourg bidding zone (Austria included until October 2018), sourced from the ENTSO‑E transparency platform."},{"name":"System load, solar and wind forecasts","provider":"ENTSO‑E","kind":"fundamentals","access":"public","period":"2016-01-02 to 2024-12-31","universe":null,"quote":"We add ENTSO‑E day‑ahead forecasts of system load, solar generation, and aggregated wind generation (onshore and offshore), with solar and wind combined into a single renewable generation series."},{"name":"Commodity futures (coal, gas, oil, EUA)","provider":"Investing.com","kind":"macro","access":"public","period":"2016-01-02 to 2024-12-31","universe":null,"quote":"Broader market influences are represented by commodity futures prices for coal (API2), natural gas (TTF), crude oil (Brent), and EU emission allowances (EUA), obtained from Investing.com."}],"known":[],"sample":{"start":"2016-01","end":"2024-12","market":"Germany–Luxembourg day-ahead electricity market","asset_class":"other","frequency":"intraday"},"data_resolution":"intraday-bars"},"transaction_costs":{"status":"available","modeled":"yes","headline_net_of_costs":null,"detail":"Operating costs are set at 11.63 EUR/MWh, representing levelled cost per unit of energy throughput.","quote":"... are taken from [6] and represent the levelled cost per unit of energy throughput.","source":"evidence_card","rule":"R2","candidates":{"front_matter":"unclear","evidence_card":"yes","evidence_card_quoted":true},"card_detail":"Operating costs are set at 11.63 EUR/MWh, representing levelled cost per unit of energy throughput.","card_quote":"... are taken from [6] and represent the levelled cost per unit of energy throughput.","meaning":"costs are part of the reported headline results"},"results":{"status":"available","evaluation_type":"trading-pnl","findings":[{"finding":"Traditional accuracy metrics (RMSE, MAE) are only weakly correlated with BESS profits.","kind":"performance","quote":"We provide evidence that classical forecast accuracy measures, such as the RMSE or the MAE, are only weakly correlated with BESS profits."},{"finding":"Dispersion‑ and association‑based measures (Cov‑e, Corr‑f) show a strong correlation with BESS profitability.","kind":"performance","quote":"In particular, measures capturing intraday error dispersion (Cov‑e) and the association between predicted and realized prices (Corr‑f) exhibit a strong relationship with BESS profitability."},{"finding":"Choosing the best‑performing forecasting model can increase profits by 16–58% over 2021–2024 compared to the poorest model.","kind":"performance","quote":"Selecting the best‑performing model can increase profits by 16–58% during 2021–2024 compared to the poorest‑performing model."},{"finding":"NARX delivers the highest statistical accuracy, while LEAR‑based forecasts generate the highest profits.","kind":"predictive-accuracy","quote":"Among the analyzed model classes, NARX delivers the highest statistical accuracy, while LEAR‑based forecasts generate the highest profits."},{"finding":"The pooled estimator is an attractive alternative to the commonly used heterogeneous approach.","kind":"statistical","quote":"With respect to model specification, the pooled estimator emerges as an attractive alternative to the commonly used heterogeneous approach."}],"reported_performance":{"strategy":"BESS arbitrage strategy","sharpe":null,"sharpe_benchmark":null,"annual_return_pct":null,"total_return_pct":null,"max_drawdown_pct":null},"out_of_sample":{"method":"unclear","headline_out_of_sample":null}},"independent_evidence":{"status":"available","supports":[],"contradicts":[{"paper":{"paper_id":"2611.17170","title":"Improving Algorithmic Trading for BESS on Intraday Electricity Markets through Distributional Price Forecasts","paper_date":"2026-11-12","authors":["Simon Hirsch","David Schaurecker","David Wozabal"],"empirical_rigor":8,"page_url":"https://thequant.space/flowcharts/improving-algorithmic-trading-for-bess-on-intraday-electricity-markets-through/"},"question":"How does forecast accuracy impact BESS profitability in intraday electricity markets?","this_paper_finding":"Traditional accuracy metrics (RMSE, MAE) are only weakly correlated with BESS profits.","other_paper_finding":"Forecast errors and instability are strongly negatively correlated with daily profits, indicating predictive accuracy drives returns.","other_paper_quote":"a strong negative correlation between error metrics, forecast stability, and daily profits, indicating that predictive accuracy is a driving factor in returns.","note":"A finds a weak correlation between RMSE/MAE and profits, while B finds a strong negative correlation between forecast errors/instability and daily profits.","confidence":0.5}],"qualifies":[],"compared_with":39,"citations":{"cited_by_in_corpus":5,"by_independent_authors":4,"how_cited":{"used_as_baseline":0,"uses_method_or_data":1,"extends":1,"discusses_findings":1,"mentions":1,"unclassified":1},"independent_papers":[{"paper":{"paper_id":"2611.17170","title":"Improving Algorithmic Trading for BESS on Intraday Electricity Markets through Distributional Price Forecasts","paper_date":"2026-11-12","authors":["Simon Hirsch","David Schaurecker","David Wozabal"],"empirical_rigor":8,"page_url":"https://thequant.space/flowcharts/improving-algorithmic-trading-for-bess-on-intraday-electricity-markets-through/"},"how":"discusses_findings","quote":"... and find only modest levels of correlation between forecast errors and profits. They conclude that traditional $L_1$ and $L_2$ metrics have no strong association with profits."},{"paper":{"paper_id":"2604.19580","title":"Probabilistic Forecasting for Day-ahead Electricity Prices, Battery Trading Strategies and the Economic Evaluation of Predictive Accuracy","paper_date":"2026-04-21","authors":["Simon Hirsch","Florian Ziel"],"empirical_rigor":6,"page_url":"https://thequant.space/flowcharts/probabilistic-forecasting-for-day-ahead-electricity-prices-battery-trading/"},"how":"extends","quote":null},{"paper":{"paper_id":"2602.10071","title":"Deep Learning for Electricity Price Forecasting: A Review of Day-Ahead, Intraday, and Balancing Electricity Markets","paper_date":"2026-02-10","authors":["Runyao Yu","Derek W. Bunn","Julia Lin","Jochen Stiasny","Fabian Leimgruber","Tara Esterl"],"empirical_rigor":6,"page_url":"https://thequant.space/flowcharts/deep-learning-for-electricity-price-forecasting-a-review-of-day-ahead-intraday/"},"how":"uses_method_or_data","quote":null}],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":false,"code_url":null,"note":"No repository presented by the authors as their own was found in the paper. Absence here is not proof that no code exists."},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":1,"extracted_at":"2026-11-14T17:21:52Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}