Paper: arXiv 2403.14695
Authors: Denis Levchenko, Efstratios Rappos, Shabnam Ataee, Biagio Nigro, Stephan Robert-Nicoud
Abstract
Neural architecture search (NAS) emerged as a way to automatically optimize neural networks for a specific task and dataset. Despite an abundance of research on NAS for images and natural language applications, similar studies for time series data are lacking. Among NAS search spaces, chain-structured are the simplest and most applicable to small datasets like time series. We compare three popular NAS strategies on chain-structured search spaces: Bayesian optimization (specifically Tree-structured Parzen Estimator), the hyperband method, and reinforcement learning in the context of financial time series forecasting. These strategies were employed to optimize simple well-understood neural architectures like the MLP, 1D CNN, and RNN, with more complex temporal fusion transformers (TFT) and their own optimizers included for comparison. We find Bayesian optimization and the hyperband method performing best among the strategies, and RNN and 1D CNN best among the architectures, but all methods were very close to each other with a high variance due to the difficulty of working with financial datasets. We discuss our approach to overcome the variance and provide implementation recommendations for future users and researchers.
Complexity vs Empirical Score
- Math Complexity: 4.5/10
- Empirical Rigor: 7.0/10
- Quadrant: Street Traders — empirical and applied, lower math complexity
Why this score: The paper involves advanced hyperparameter optimization techniques and neural network architecture search, but focuses on practical implementation with real-world financial datasets, backtesting, and specific performance metrics like F1 and AUC.
Research Flowchart
flowchart TD
A["Research Goal: Optimize Neural Networks<br>for Financial Time Series Forecasting"] --> B["Methodology: Chain-Structured NAS<br>with 3 Strategies & 5 Architectures"]
B --> C["Data Inputs: Financial Datasets<br>(High Variance, Small Size)"]
C --> D{"Computational Process"}
D --> E["Bayesian Optimization<br>(Tree-structured Parzen Estimator)"]
D --> F["Hyperband Method<br>(Resource-based Pruning)"]
D --> G["Reinforcement Learning<br>(Policy Gradient)"]
E & F & G --> H["Key Outcomes"]
H --> I["Best Strategies: Bayesian & Hyperband"]
H --> J["Best Architectures: RNN & 1D CNN"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper evaluates three NAS strategies on chain-structured search spaces for financial time series forecasting and reports that Bayesian optimization and hyperband perform best, with RNN and 1D CNN yielding the top architectures.
Source sentence
We find Bayesian optimization and the hyperband method performing best among the strategies, and RNN and 1D CNN best among the architectures, but all methods were very close to each other with a high variance due to the difficulty of working with financial datasets.
Assumes 3
- Data are split chronologically with 70% training, 20% validation, and 10% test. data
Source sentence
The data was split into (in historical order): first 70% for training, the following 20% for validation used in the evaluation during optimization runs, and the last 10% for final testing on unseen data.
- Financial time series are noisy and non‑stationary, leading to low signal‑to‑noise ratios. market
Source sentence
The problem is challenging due to the inherently noisy and non-stationary nature of financial time series.
- Binary classification performance is evaluated primarily with AUC, balanced accuracy and F1, assuming these metrics reflect predictive usefulness. statistical
Source sentence
We mostly focused on AUC as we found it to be the hardest metric to optimize, but we paid close attention to bACC and F1 too and report…
Data
- Germany dataset market-prices unknown
Source sentence
Our best results were obtained on the Germany dataset, with time-derived redundant features removed but no PCA applied.
- Japan dataset market-prices unknown
Source sentence
For the Japan dataset, the redundant time-derived features removed and PCA applied retaining most of the information in the dataset while further cutting the number of features approximately in half gave the best results.
- US dataset market-prices unknown
Source sentence
The US dataset was most challenging for us. Although tuned 1d CNNs gave an AUC score of 0.6 ± 0.02 on validation data (not used for training)
Costs
Not applicable
Finds 5
- On the Germany dataset, an LSTM tuned by hyperband achieved an average AUC of 0.56 ± 0.05 and balanced accuracy of 0.54 ± 0.04 over 50 test runs. performance
Source sentence
The LSTM with parameters selected by the hyperband method applied to the unseen test data achieved an AUC score of 0.56 on average over 50 test runs starting from different random seeds (see Section 5), with 0.05 standard deviation. The same architecture showed balanced accuracy score of 0.54 ± 0.04.
- A 1D CNN found by Bayesian optimization on the Germany dataset obtained AUC 0.54 ± 0.05 and a high F1 of 0.6 ± 0.06. performance
Source sentence
1D CNNs were also able to achieve good performance on this dataset, with the best architecture provided by Bayesian optimization giving an AUC score of 0.54 ± 0.05 and a high F1 score of 0.6 ± 0.06.
- For the Japan dataset, the best model was a 1D CNN from Bayesian optimization with AUC 0.54 ± 0.03 and F1 0.65 ± 0.02. performance
Source sentence
The best performing architecture was a 1D CNN coming from Bayesian optimization, achieving an AUC score of 0.54 ± 0.03 over 50 test runs, with a high F1 score of 0.65 ± 0.02.
- On the US dataset, no architecture achieved average test AUC above 0.5, indicating poor predictive performance. negative
Source sentence
on average no architecture could achieve AUC over 0.5 on the test dataset.
- Overall, Bayesian optimization and hyperband outperformed reinforcement learning, while LSTMs and 1D CNNs consistently beat simple feed‑forward networks across datasets. performance
Source sentence
For search strategy, overall the hyperband method and Bayesian optimization showed better performance than the reinforcement learning based approach, but all three were very close to each other.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 25 related papers). Coverage is partial, so this is neither replication nor refutation.
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (88)
- CNNpred: CNN-based stock market prediction using a diverse set of variables (2019) — link
- Neural Architecture Search: A Survey (2019) — link
- Neural Architecture Search with Reinforcement Learning (2017)
- Algorithms for Hyper-Parameter Optimization (2011) — link
- Practical Bayesian Optimization of Machine Learning Algorithms (2012) — link
- Gaussian processes for machine learning. (2006)
- Taking the Human Out of the Loop: A Review of Bayesian Optimization (2016) — link
- A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning (2010)
and 80 more
- Towards an Empirical Foundation for Assessing Bayesian Optimization of Hyperparameters (2013) — link
- Optuna: A Next-generation Hyperparameter Optimization Framework (2019)
- KerasTuner (2019) — link
- Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization (2018)
- Automated Machine Learning in Action (2022)
- Human-level control through deep reinforcement learning (2015)
- Q-learning (1992)
- On-line Q-learning using connectionist systems (1994)
- Mastering the game of Go with deep neural networks and tree search (2016)
- A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play (2018)
- Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates (2017)
- Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor (2018)
- Informing sequential clinical decision-making through reinforcement learning: an empirical study (2011)
- Supervised reinforcement learning with recurrent neural network for dynamic treatment recommendation (2018)
- Extremely randomized trees (2006)
- Deep reinforcement learning with double q-learning (2016)
- Addressing function approximation error in actor-critic methods (2018)
- The sharpe ratio (1998)
- Principal component analysis (1987)
- Long short-term memory (1997)
- Temporal Fusion Transformers for interpretable multi-horizon time series forecasting (2021) — link
- Deep reinforcement learning that matters (2018)
- Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning (2021)
- A distributional perspective on reinforcement learning (2017)
- Implicit quantile networks for distributional reinforcement learning (2018)
- On calibration of modern neural networks (2017)
- Neural Architecture Search: Insights from 1000 papers (2023) — link
- The Transformer Network for the Traveling Salesman Problem (2021) — link
- Combining Reinforcement Learning and Optimal Transport for the Traveling Salesman Problem (2022) — link
- Pointerformer: Deep Reinforced Multi-Pointer Transformer for the Traveling Salesman Problem (2023) — link
- A Lightweight CNN-Transformer Model for Learning Traveling Salesman Problems (2023) — link
- Reinforcement Learning-based Non-Autoregressive Solver for Traveling Salesman Problems (2023) — link
- Solving Quadratic Traveling Salesman Problem with Deep Reinforcement Learning (2022) — link
- Learning Heuristics for the TSP by Policy Gradient (2018)
- Revised Note on Learning Algorithms for Quadratic Assignment with Graph Neural Networks (2018) — link
- Solving the Quadratic Assignment Problem using Deep Reinforcement Learning (2023) — link
- RL 4 CO : a Unified Reinforcement Learning for Combinatorial Optimization Library (2023) — link
- Attention-based Reinforcement Learning for Combinatorial Optimization: Application to Job Shop Scheduling Problem (2024) — link
- Accelerating Quadratic Optimization with Reinforcement Learning (2021) — link
- Pointer Networks (2015) — link
- Neural Combinatorial Optimization with Reinforcement Learning (2017) — link
- Unsupervised Machine Learning for the Quadratic Assignment Problem (2023) — link
- PyTorch Forecasting (2023) — link
- Deep neural networks for quantum circuit mapping (2021) — link
- neural-architecture-search (2018)
- Reinforcement Learning for Solving the Vehicle Routing Problem (2018) — link
- Vehicle Routing Problem Using Reinforcement Learning: Recent Advancements (2022)
- The Traveling Salesman Problem: A Computational Study (2006)
- Proximal Policy Optimization Algorithms (2017) — link
- POPMUSIC for the Travelling Salesman Problem (2019) — link
- A Linearithmic Heuristic for the Travelling Salesman Problem (2022) — link
- \textbf Design of Heuristic Algorithms for Hard Optimization (2023) — link
- On the Emerging Potential of Quantum Annealing Hardware for Combinatorial Optimization (2022) — link
- MQT QMAP: Efficient Quantum Circuit Mapping (2023) — link
- Machine Learning Optimization of Quantum Circuit Layouts (2023) — link
- Optimization of quantum circuit mapping using gate transformation and commutation (2020) — link
- A dataset for quantum circuit mapping (2021) — link
- The complexity of quantum circuit mapping with fixed parameters (2022) — link
- A Dynamic Look-Ahead Heuristic for the Qubit Mapping Problem of NISQ Computers (2020) — link
- Reinforcement Learning and DEAR Framework for Solving the Qubit Mapping Problem (2022)
- Resource Bounds for Quantum Circuit Mapping via Quantum Circuit Complexity (2024) — link
- The symmetric quadratic traveling salesman problem (2013) — link
- Exact algorithms and heuristics for the Quadratic Traveling Salesman Problem with an application in bioinformatics (2014) — link
- A Polyhedral Study of the Quadratic Traveling Salesman Problem (2016)
- An Efficient Hybrid Genetic Algorithm for the Quadratic Traveling Salesman Problem (2023) — link
- Algorithms and Experimental Study for the Traveling Salesman Problem of Second Order (2008)
- A class of exponential neighbourhoods for the quadratic travelling salesman problem (2020) — link
- Geometric and LP-based heuristics for angular travelling salesman problems in the plane (2019) — link
- Nearest Neighbor Transformation of Quantum Circuits in 2D Architecture (2020) — link
- Machine Learning into Metaheuristics: A Survey and Taxonomy (2021) — link
- Machine learning enhancing metaheuristics: a systematic review (2023) — link
- The ant system: Optimization by a colony of cooperating agents (1996)
- Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performance (2023) — link
- Particle swarm optimization-based automatic parameter selection for deep neural networks and its applications in large-scale and high-dimensional data (2017) — link
- Neural Predictor for Neural Architecture Search (2020)
- Neural Architecture Search without Training (2021) — link
- Deep learning with long short-term memory networks for financial market predictions (2018) — link
- Classification-based financial markets prediction using deep neural networks (2017) — link
- Evolving Deep Convolutional Neural Networks for Image Classification (2017) — link
- Pattern Recognition and Machine Learning (Information Science and Statistics) (2006)
0 of 88 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.