Paper: arXiv 2410.01843

Authors: Ahmad Makinde

Abstract

Several studies have discussed the impact different optimization techniques in the context of time series forecasting across different Neural network architectures. This paper examines the effectiveness of Adam and Nesterov’s Accelerated Gradient (NAG) optimization techniques on LSTM and GRU neural networks for time series prediction, specifically stock market time-series. Our study was done by training LSTM and GRU models with two different optimization techniques - Adam and Nesterov Accelerated Gradient (NAG), comparing and evaluating their performance on Apple Inc’s closing price data over the last decade. The GRU model optimized with Adam produced the lowest RMSE, outperforming the other model-optimizer combinations in both accuracy and convergence speed. The GRU models with both optimizers outperformed the LSTM models, whilst the Adam optimizer outperformed the NAG optimizer for both model architectures. The results suggest that GRU models optimized with Adam are well-suited for practitioners in time-series prediction, more specifically stock price time series prediction producing accurate and computationally efficient models. The code for the experiments in this project can be found at https://github.com/AhmadMak/Time-Series-Optimization-Research Keywords: Time-series Forecasting, Neural Network, LSTM, GRU, Adam Optimizer, Nesterov Accelerated Gradient (NAG) Optimizer

Complexity vs Empirical Score

  • Math Complexity: 3.5/10
  • Empirical Rigor: 7.0/10
  • Quadrant: Street Traders — empirical and applied, lower math complexity

Why this score: The paper applies standard neural network architectures and optimization algorithms with relatively basic mathematical exposition, while the empirical component is strong due to the use of real financial data (Apple stock), specific performance metrics (RMSE), and a public code repository.

Research Flowchart

  flowchart TD
  A["Research Goal: Compare optimizers on RNNs for stock forecasting"] --> B{"Methodology"}
  B --> C["Data: Apple Inc. closing prices 2014-2024"]
  B --> D["Models: LSTM vs GRU"]
  B --> E["Optimizers: Adam vs Nesterov Accelerated Gradient"]
  
  C & D & E --> F["Training & Evaluation: RMSE & Convergence Speed"]
  
  F --> G{"Key Findings"}
  G --> H["GRU-Adam: Lowest RMSE, Best Performance"]
  G --> I["GRU > LSTM regardless of optimizer"]
  G --> J["Adam > Nesterov regardless of architecture"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper compares Adam and Nesterov Accelerated Gradient optimizers on LSTM and GRU networks for stock price time‑series forecasting.

Source sentence
This paper examines the effectiveness of Adam and Nesterov’s Accelerated Gradient (NAG) optimization techniques on LSTM and GRU neural networks for time series prediction, specifically stock market time‑series.

Assumes 3

  • The study is limited to stock market data due to limited computational resources. data
    Source sentence
    The experiments carried out in this study however are only limited to stock market data due to limited computational resources.
  • The dataset is small, which may affect validation loss variability. data
    Source sentence
    Another limitation was that the data used for training, validation and testing was not very big.
  • Only daily closing prices of Apple Inc. are used, assuming they represent the underlying price dynamics. market
    Source sentence
    We will be using a dataset from Yahoo Finance containing the daily closing prices of Apple Inc.’s stock between 2014 and the present month (August 2024).

Data

Backtest sample: Apple Inc. closing price · equities · daily

  • Apple Inc. daily closing prices · Yahoo Finance market-prices public
    2014-2024 · Apple Inc.
    Source sentence
    We will be using a dataset from Yahoo Finance containing the daily closing prices of Apple Inc.’s stock between 2014 and the present month (August 2024)

Recognised sources: Yahoo Finance

Costs

Not applicable The paper reports no trading or execution result.

Finds 4

  • GRU model optimized with Adam achieved the lowest RMSE (172.4037), outperforming all other model‑optimizer combinations. predictive-accuracy
    Source sentence
    The GRU model optimized with Adam produced the lowest RMSE, outperforming the other model-optimizer combinations in both accuracy and convergence speed.
  • GRU models outperformed LSTM models for both Adam and NAG optimizers. performance
    Source sentence
    The GRU models with both optimizers outperformed the LSTM models, whilst the Adam optimizer outperformed the NAG optimizer for both model architectures.
  • Adam optimizer consistently yielded lower RMSE and final losses than NAG across both architectures. performance
    Source sentence
    Regarding their impact as optimizers, it has been observed that the use of Adam as an optimizer always results in decreased values of RMSE and final losses as opposed to NAG.
  • RMSE values: LSTM‑Adam 176.6259, LSTM‑NAG 187.6416, GRU‑Adam 172.4037, GRU‑NAG 182.2570. predictive-accuracy
    Source sentence
    RMSE values for each model and optimizer are summarized in the table. Adam’s GRU has the lowest RMSE (172.4037)

Checked by others

Contradicted by

  • Same Error, Different Function: The Optimizer as an Implicit Prior in Financial Time Series (2026)
    On the question: Which optimizer yields better forecasting performance on financial time series?
    Compare the findings

    This paper: Adam optimizer consistently yielded lower RMSE and final losses than NAG across both architectures.

    That paper: Predictive accuracy (NMSE) remains unchanged across architectures and optimizers despite functional divergence.

    A reports Adam gives lower RMSE and final loss than NAG; B reports NMSE unchanged across optimizers, contradicting A.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (28)

  1. A method for unconstrained convex minimization problem with the rate of convergence O (1/k2) (1983)
  2. Adam: A Method for Stochastic Optimization (2014) — link
  3. Ian Goodfellow, Yoshua Bengio, and Aaron Courville: Deep learning: The MIT Press, 2016, 800 pp, ISBN: 0262035618 (2018)
  4. An overview of gradient descent optimization algorithms (2016) — link
  5. Recurrent Neural Networks for Time Series Forecasting: Current status and future directions (2021) — link
  6. Deep Learning for Time Series Forecasting: Predict the Future with MLPs … - Jason Brownlee - Google Books (2018) — link
  7. On the importance of initialization and momentum in deep learning (2013) — link
  8. Advances in Optimizing Recurrent Networks (2012) — link
and 20 more
  1. On the Convergence of Adam and Beyond (2019) — link
  2. Learning precise timing with LSTM recurrent networks (2003) — link
  3. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling (2014) — link
  4. A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction (2017)
  5. WeatherBench: A benchmark dataset for data-driven weather forecasting (2020)
  6. Short-Term Residential Load Forecasting based on LSTM Recurrent Neural Network (2019)
  7. Time Series Forecasting With Deep Learning: A Survey (2020)
  8. Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks (2018)
  9. Forecasting stock prices with long-short term memory neural network based on attention mechanism (2020) — link
  10. Using long short-term memory networks to predict energy consumption of air-conditioning systems (2020) — link
  11. Data Mining: Concepts and Techniques (2011)
  12. Tensorflow: Large-scale machine learning on heterogeneous distributed systems (2016) — link
  13. Rethinking the inception architecture for computer vision (2016)
  14. Random search for hyper-parameter optimization. (2012)
  15. Large-scale machine learning with stochastic gradient descent (2010)
  16. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation (2014) — link
  17. Attention is all you need (2017)
  18. Long Short-Term Memory (1997)
  19. Deep learning on mobile devices: a review (2019)
  20. Show, attend and tell: Neural image caption generation with visual attention (2015) — link

0 of 28 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.