Overfitting

Interpretable Hypothesis-Driven Trading:A Rigorous Walk-Forward Validation Framework for Market Microstructure Signals ArXiv ID: 2512.12924 “View on arXiv” Authors: Gagan Deep, Akash Deep, William Lamptey Abstract We develop a rigorous walk-forward validation framework for algorithmic trading designed to mitigate overfitting and lookahead bias. Our methodology combines interpretable hypothesis-driven signal generation with reinforcement learning and strict out-of-sample testing. The framework enforces strict information set discipline, employs rolling window validation across 34 independent test periods, maintains complete interpretability through natural language hypothesis explanations, and incorporates realistic transaction costs and position constraints. Validating five market microstructure patterns across 100 US equities from 2015 to 2024, the system yields modest annualized returns (0.55%, Sharpe ratio 0.33) with exceptional downside protection (maximum drawdown -2.76%) and market-neutral characteristics (beta = 0.058). Performance exhibits strong regime dependence, generating positive returns during high-volatility periods (0.60% quarterly, 2020-2024) while underperforming in stable markets (-0.16%, 2015-2019). We report statistically insignificant aggregate results (p-value 0.34) to demonstrate a reproducible, honest validation protocol that prioritizes interpretability and extends naturally to advanced hypothesis generators, including large language models. The key empirical finding reveals that daily OHLCV-based microstructure signals require elevated information arrival and trading activity to function effectively. The framework provides complete mathematical specifications and open-source implementation, establishing a template for rigorous trading system evaluation that addresses the reproducibility crisis in quantitative finance research. For researchers, practitioners, and regulators, this work demonstrates that interpretable algorithmic trading strategies can be rigorously validated without sacrificing transparency or regulatory compliance. ...

The Red Queen’s Trap: Limits of Deep Evolution in High-Frequency Trading ArXiv ID: 2512.15732 “View on arXiv” Authors: Yijia Chen Abstract The integration of Deep Reinforcement Learning (DRL) and Evolutionary Computation (EC) is frequently hypothesized to be the “Holy Grail” of algorithmic trading, promising systems that adapt autonomously to non-stationary market regimes. This paper presents a rigorous post-mortem analysis of “Galaxy Empire,” a hybrid framework coupling LSTM/Transformer-based perception with a genetic “Time-is-Life” survival mechanism. Deploying a population of 500 autonomous agents in a high-frequency cryptocurrency environment, we observed a catastrophic divergence between training metrics (Validation APY $>300%$) and live performance (Capital Decay $>70%$). We deconstruct this failure through a multi-disciplinary lens, identifying three critical failure modes: the overfitting of \textit{“Aleatoric Uncertainty”} in low-entropy time-series, the \textit{“Survivor Bias”} inherent in evolutionary selection under high variance, and the mathematical impossibility of overcoming microstructure friction without order-flow data. Our findings provide empirical evidence that increasing model complexity in the absence of information asymmetry exacerbates systemic fragility. ...

In-Sample and Out-of-Sample Sharpe Ratios for Linear Predictive Models ArXiv ID: 2501.03938 “View on arXiv” Authors: Unknown Abstract We study how much the in-sample performance of trading strategies based on linear predictive models is reduced out-of-sample due to overfitting. More specifically, we compute the in- and out-of-sample means and variances of the corresponding PnLs and use these to derive a closed-form approximation for the corresponding Sharpe ratios. We find that the out-of-sample “replication ratio” diminishes for complex strategies with many assets based on many weak rather than a few strong trading signals, and increases when more training data is used. The substantial quantitative importance of these effects is illustrated with a simulation case study for commodity futures following the methodology of Gârleanu and Pedersen, and an empirical case study using the dataset compiled by Goyal, Welch and Zafirov. ...

Double Descent in Portfolio Optimization: Dance between Theoretical Sharpe Ratio and Estimation Accuracy ArXiv ID: 2411.18830 “View on arXiv” Authors: Unknown Abstract We study the relationship between model complexity and out-of-sample performance in the context of mean-variance portfolio optimization. Representing model complexity by the number of assets, we find that the performance of low-dimensional models initially improves with complexity but then declines due to overfitting. As model complexity becomes sufficiently high, the performance improves with complexity again, resulting in a double ascent Sharpe ratio curve similar to the double descent phenomenon observed in artificial intelligence. The underlying mechanisms involve an intricate interaction between the theoretical Sharpe ratio and estimation accuracy. In high-dimensional models, the theoretical Sharpe ratio approaches its upper limit, and the overfitting problem is reduced because there are more parameters than data restrictions, which allows us to choose well-behaved parameters based on inductive bias. ...