Paper: arXiv 2601.16446

Authors: George Awiakye-Marfo, Elijah Agbosu, Victoria Mawuena Barns, Samuel Asante Gyamerah

Abstract

Deep learning models are effective for sequential data modeling, yet commonly used activation functions such as ReLU, LeakyReLU, and PReLU often exhibit gradient instability when applied to noisy, non-stationary financial time series. This study introduces BrownianReLU, a stochastic activation function induced by Brownian motion that enhances gradient propagation and learning stability in Long Short-Term Memory (LSTM) networks. Using Monte Carlo simulation, BrownianReLU provides a smooth, adaptive response for negative inputs, mitigating the dying ReLU problem. The proposed activation is evaluated on financial time series from Apple, GCB, and the S&P 500, as well as LendingClub loan data for classification. Results show consistently lower Mean Squared Error and higher $R^2$ values, indicating improved predictive accuracy and generalization. Although ROC-AUC metric is limited in classification tasks, activation choice significantly affects the trade-off between accuracy and sensitivity, with Brownian ReLU and the selected activation functions yielding practically meaningful performance.

Complexity vs Empirical Score

  • Math Complexity: 5.0/10
  • Empirical Rigor: 4.5/10
  • Quadrant: Lab Rats — theoretically deep, empirically untested

Why this score: The paper introduces a novel stochastic activation function (Brownian ReLU) by leveraging Monte Carlo simulations of Brownian motion for negative input regions. While the mathematical derivation of the gradient is clearly stated, the empirical validation lacks rigorous backtesting frameworks, out-of-sample statistical significance tests, and detailed financial context.

Research Flowchart

  flowchart TD
    A[Research Goal: Mitigate Gradient Instability & Dying ReLU in LSTMs for Noisy Financial Data] --> B[Introduce BrownianReLU: Stochastic Activation Induced by Brownian Motion]
    B --> C[Methodology: Monte Carlo Simulations for Smooth Adaptive Negative Input Response]
    
    D[(Data Inputs: Apple, GCB, S&P 500 Equities & LendingClub Loans)] --> E[Train & Evaluate LSTM Models Across Activation Functions]
    C --> E
    
    E --> F[Outcomes: Lower MSE, Higher R² Values, Improved Gradient Stability & Generalization]

Cited References (34)

  1. Activation functions in deep learning: A comprehensive survey and benchmark (2022)
  2. Deep learning (2016)
  3. Why tanh: choosing a sigmoidal function (1992)
  4. Rectified linear units improve restricted boltzmann machines (2010)
  5. The perceptron: a probabilistic model for information storage and organization in the brain. (1958)
  6. An activation function adapting training algorithm for sigmoidal feedforward networks (2004)
  7. A case for the self-adaptation of activation functions in FFANNs (2004)
  8. A class+ 1 sigmoidal activation functions for FFANNs (2003)
and 26 more
  1. Deep Learning Activation Functions: Fixed-Shape, Parametric, Adaptive, Stochastic, Miscellaneous, Non-Standard, Ensemble (2024) — link
  2. Three Decades of Activations: A Comprehensive Survey of 400 Activation Functions for Neural Networks (2024) — link
  3. Spin-wave coupled spin torque oscillators for artificial neural network (2018)
  4. Gradient-based learning applied to document recognition (1998)
  5. New activation functions for single layer feedforward neural network (2021)
  6. Revise saturated activation functions (2016) — link
  7. LiSHT: Non-parametric linearly scaled hyperbolic tangent activation function for neural networks (2022)
  8. Study of the Effect of Combining Activation Functions in a Convolutional Neural Network (2020)
  9. Soft-root-sign: A new bounded neural activation function (2020)
  10. Sigmoid-Gumbel: Yeni Bir Hibrit Aktivasyon Fonksiyonu (2022)
  11. Imagenet classification with deep convolutional neural networks (2012)
  12. An improved human activity recognition technique based on convolutional neural network (2023)
  13. Rectifier nonlinearities improve neural network acoustic models (2013)
  14. Empirical evaluation of rectified activations in convolutional network (2015) — link
  15. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification (2015)
  16. Bounded activation functions for enhanced training stability of deep neural networks on visual pattern recognition problems (2016)
  17. Deep neural networks with elastic rectified linear units for object recognition (2018)
  18. Fast and accurate deep network learning by exponential linear units (elus) (2015) — link
  19. Noisy activation functions (2016)
  20. Searching for activation functions (2017) — link
  21. Mish: A self regularized non-monotonic activation function (2019) — link
  22. Incorporating second-order functional knowledge for better option pricing (2000)
  23. Long Short-term Memory (1997)
  24. Gaussian error linear units (gelus) (2016) — link
  25. Brownian motion and stochastic calculus (2014)
  26. Adam: A method for stochastic optimization (2014) — link

0 of 34 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.