Paper: arXiv 2509.17964

Authors: Yang Li, Zhi Chen, Steve Y. Yang, Ruixun Zhang

Abstract

Traditional stochastic control methods in finance rely on simplifying assumptions that often fail in real world markets. While these methods work well in specific, well defined scenarios, they underperform when market conditions change. We introduce FinFlowRL, a novel framework for financial stochastic control that combines imitation learning with reinforcement learning. The framework first pretrains an adaptive meta policy by learning from multiple expert strategies, then finetunes it through reinforcement learning in the noise space to optimize the generation process. By employing action chunking, that is generating sequences of actions rather than single decisions, it addresses the non Markovian nature of financial markets. FinFlowRL consistently outperforms individually optimized experts across diverse market conditions.

Complexity vs Empirical Score

  • Math Complexity: 7.0/10
  • Empirical Rigor: 6.0/10 (capped from 6.5: transaction costs not modelled; see penalty flags)
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper introduces an innovative combination of flow matching, noise-space reinforcement learning, and action chunking for financial stochastic control, showing strong conceptual novelty. The mathematical framework combines Hawkes processes, mean-flow identities, and jump-diffusions, while empirical evaluation is extensive across simulated market regimes, though synthetic backtesting slightly limits real-world validation.

Research Flowchart

  flowchart TD
    A["Research Goal: Adaptive Stochastic Control in Non-Markovian Financial Markets"] --> B["Data Inputs: Expert Strategies & Market Noise Space"]
    B --> C["Phase 1: Meta-Policy Pretraining via Imitation Learning"]
    C --> D["Action Chunking: Generate Multi-Step Action Sequences"]
    D --> E["Phase 2: Reinforcement Learning Finetuning in Noise Space"]
    E --> F["Key Outcome: FinFlowRL Outperforms Single Experts Across Conditions"]

Cited References (153)

  1. Fractional Brownian motions, fractional noises and applications (1968)
  2. Reinforcement learning with action chunking (2025) — link
  3. Mean Flows for One-step Generative Modeling (2025) — link
  4. Steering Your Diffusion Policy with Latent Space Reinforcement Learning (2025) — link
  5. Finagent: A multimodal foundation agent for financial trading: Tool-augmented, diversified, and generalist (2024)
  6. Flow Q-Learning (2025) — link
  7. Steering Your Diffusion Policy with Latent Space Reinforcement Learning (2025) — link
  8. KAN: Kolmogorov-Arnold Networks (2024) — link
and 145 more
  1. Static Hedging of Standard Options (2014) — link
  2. Financial trading as a game: A deep reinforcement learning approach (2018) — link
  3. Automatic financial trading agent for low-risk portfolio management using deep reinforcement learning (2019) — link
  4. Adaptive quantitative trading: An imitative deep reinforcement learning approach (2020)
  5. An Overview of Machine Learning, Deep Learning, and Reinforcement Learning-Based Techniques in Quantitative Finance: Recent Progress and Challenges (2023)
  6. CHATREPORT: Democratizing Sustainability Disclosure Analysis through LLM-based Tools (2023) — link
  7. Improving Factuality and Reasoning in Language Models through Multiagent Debate (2023) — link
  8. Replication and analysis of Ebbinghaus’ forgetting curve (2015)
  9. Generative agents: Interactive simulacra of human behavior (2023) — link
  10. Code Llama: Open Foundation Models for Code (2023) — link
  11. Improving language understanding by generative pre-training (2018)
  12. GPT-4 Technical Report (2023) — link
  13. FinGPT: Open-Source Financial Large Language Models — → 2306.06031
  14. Bloomberggpt: A large language model for finance (2023) — link
  15. Sentiment-influenced trading system based on multimodal deep reinforcement learning (2021)
  16. A Deep Multimodal Reinforcement Learning System Combined with CNN and LSTM for Stock Trading (2019) — link
  17. Design and implementation of a multiagent stock trading system (2012)
  18. MSPM: A modularized and scalable multi-agent reinforcement learning-based system for financial portfolio management (2022)
  19. A multi-agent Q-learning framework for optimizing stock trading systems (2002)
  20. A multi-agent deep reinforcement learning framework for algorithmic trading in financial markets (2022)
  21. Modelling stock markets by multi-agent reinforcement learning (2021)
  22. word2vec Explained: deriving Mikolov et al.’s negative-sampling word-embedding method (2014) — link
  23. Glove: Global vectors for word representation (2014)
  24. Bert: Pre-training of deep bidirectional transformers for language understanding (2018) — link
  25. Long short-term memory based recurrent neural network architectures for large vocabulary speech recognition (2014) — link
  26. Interactive narrative: A novel application of artificial intelligence for computer games (2012)
  27. Weaviate — link
  28. Cohere: Privacy Management in Large Scale Systems (2023) — link
  29. Human memory: A proposed system and its control processes (1968)
  30. What are the differences between long-term, short-term, and working memory? (2008)
  31. Llama: Open and efficient foundation language models (2023) — link
  32. Generative AI for End-to-End Limit Order Book Modelling: A Token-Level Autoregressive Generative Model of Message Flow Using a Deep State Space Network (2023) — link
  33. Modelling microstructure noise with mutually exciting point processes (2013)
  34. Modeling foreign exchange market activity around macroeconomic news: Hawkes-process approach (2015)
  35. Modelling security market events in continuous time: Intensity based, multivariate point process models (2007)
  36. Limit theorems for nearly unstable Hawkes processes (2016)
  37. Exact simulation of Hawkes process with exponentially decaying intensity (2013)
  38. Bayesian Estimation of the ETAS Model for Earthquake Occurrences (2021)
  39. Bayesian Inference for Hawkes Processes (2013) — link
  40. Sora by OpenAI (2024) — link
  41. Mamba: Linear-Time Sequence Modeling with Selective State Spaces (2023) — link
  42. Benchmark dataset for mid-price forecasting of limit order book data with machine learning methods (2018) — link
  43. Forecasting Stock Prices from the Limit Order Book Using Convolutional Neural Networks (2017) — link
  44. DeepLOB: Deep Convolutional Neural Networks for Limit Order Books (2019) — link
  45. BDLOB: Bayesian Deep Convolutional Neural Networks for Limit Order Books (2018) — link
  46. Transformers for Limit Order Books (2020) — link
  47. Mean flows for one-step generative modeling (2025) — link
  48. Flow-Based Policy for Online Reinforcement Learning (2025) — link
  49. Denoising diffusion probabilistic models (2020)
  50. Earnhft: Efficient hierarchical reinforcement learning for high frequency trading (2024)
  51. Reinforcement Learning in High-frequency Market Making — → 2407.21025
  52. Empirical market microstructure: The institutions, economics, and econometrics of securities trading (2007)
  53. Score-Based Generative Modeling through Stochastic Differential Equations (2021)
  54. Reverse-Time Diffusion Equation Models (1982)
  55. Market microstructure: A survey (2000)
  56. Trading and exchanges: Market microstructure for practitioners (2002)
  57. High-frequency trading in a limit order book (2008)
  58. Buy low, sell high: A high frequency trading perspective (2014)
  59. High frequency market making: Implications for liquidity (2017)
  60. Dealing with the inventory risk: a solution to the market making problem (2013)
  61. Reinforcement learning: An introduction (2018)
  62. Deep reinforcement learning in high frequency trading (2018) — link
  63. Reinforcement learning in robotics: A survey (2013)
  64. Deep unsupervised learning using nonequilibrium thermodynamics (2015)
  65. Planning with diffusion for flexible behavior synthesis (2022) — link
  66. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion (2024)
  67. Diffusion policies as an expressive policy class for offline reinforcement learning (2022) — link
  68. One step diffusion via shortcut models (2024) — link
  69. U-net: Convolutional networks for biomedical image segmentation (2015)
  70. Continuous auctions and insider trading (1985)
  71. Informed speculation with imperfect competition (1989)
  72. Feature-wise transformations for visual reasoning (2018)
  73. Attention is all you need (2017)
  74. Nice: Non-linear independent components estimation (2014) — link
  75. Score-based generative modeling through stochastic differential equations (2020) — link
  76. Handbook of Stochastic Methods (2004)
  77. The Fokker-Planck Equation: Methods of Solution and Applications (1996)
  78. Numerical Solution of Stochastic Differential Equations (1999)
  79. Markov chain Monte Carlo with approximate proposals (2015)
  80. Large deviations for diffusion processes with unbounded coefficients (2006)
  81. Optimal Trade Execution (1996)
  82. Optimal execution and transaction costs (1996)
  83. Volume Weighted Average Price: A Primer (2018)
  84. Reinforcement Learning: An Introduction (1998)
  85. Human-level control through deep reinforcement learning (2015)
  86. Continuous control with deep reinforcement learning (2015) — link
  87. Denoising diffusion probabilistic models (2020) — link
  88. Title of the Paper (2019)
  89. A multivariate Hawkes process approach for modeling financial market activity (2014)
  90. Hawkes processes for energy markets: Modeling negative and positive jumps (2021)
  91. Long short-term memory (1997)
  92. Learning phrase representations using RNN encoder-decoder for statistical machine translation (2014)
  93. HiPPO: A continuous-time sequence model (2020) — link
  94. Mamba2: Enhancing state-space models for continuous memory (2023)
  95. Chain-of-thought prompting elicits reasoning in large language models (2022) — link
  96. Chain of Continuous Thought: Novel Paradigm with Enhanced LLM Reasoning in Continuous Latent Space (2023)
  97. Optimal high-frequency trading in a pro rata microstructure with predictive information (2015)
  98. Multi-Agent Systems for Financial Trading: A Survey (2021)
  99. Language models are few-shot learners (2020) — link
  100. Advanced Continuous Reasoning in Latent Spaces (2024) — link
  101. Price dynamics in a Markovian limit order market (2013)
  102. Incorporating order-flow into optimal execution (2016)
  103. Risk metrics and fine tuning of high-frequency trading strategies (2015)
  104. Optimal dealer pricing under transactions and return uncertainty (1981)
  105. Training Large Language Models to Reason in a Continuous Latent Space (2024) — link
  106. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality (2024) — link
  107. HiPPO: Recurrent Memory with Optimal Polynomial Projections (2020) — link
  108. FlowHFT: Flow Policy Induced Optimal High-Frequency Trading under Diverse Market Conditions — → 2505.05784
  109. FlowOE: Imitation Learning with Flow Policy from Ensemble RL Experts for Optimal Execution under Heston Volatility and Concave Market Impacts — → 2506.05755
  110. Flow Matching for Generative Modeling (2022) — link
  111. Proximal Policy Optimization Algorithms (2017) — link
  112. Point spectra of some mutually exciting point processes (1971)
  113. Hawkes processes in finance (2015)
  114. Option pricing when underlying stock returns are discontinuous (1976)
  115. Optimal execution of portfolio transactions (2001)
  116. Market making under a weakly consistent limit order book model (2019)
  117. High-frequency market making to large institutional trades (2019)
  118. High-frequency market-making with inventory constraints and directional bets (2012) — link
  119. Brownian motion (2010)
  120. Lectures on the Poisson process (2017)
  121. The Financial Mathematics of Market Liquidity: From optimal execution to market making (2016)
  122. The problem of calibrating an agent-based model of high-frequency trading (2016) — link
  123. Deep reinforcement learning for active high frequency trading (2021) — link
  124. Reinforcement learning: A survey (1996)
  125. Introduction to reinforcement learning (2020)
  126. Markov decision processes (1990)
  127. Reinforcement learning for stock prediction and high-frequency trading with T+ 1 rules (2022)
  128. Deep reinforcement learning for trading (2019) — link
  129. Frequant: A reinforcement-learning based adaptive portfolio optimization with multi-frequency decomposition (2024)
  130. Inside the black box: A simple guide to quantitative and high-frequency trading (2013)
  131. MacroHFT: Memory Augmented Context-aware Reinforcement Learning On High Frequency Trading (2024)
  132. Flow matching in latent space (2023) — link
  133. Flow matching guide and code (2024) — link
  134. Flowpolicy: Enabling fast and robust 3d flow-based policy via consistency flow matching for robot manipulation (2025)
  135. Shortcut to greatness: The new thinking and the revolution in Soviet foreign policy (2003)
  136. Lifetime portfolio selection under uncertainty: The continuous-time case (1969)
  137. The pricing of options and corporate liabilities (1973)
  138. Volatility is rough (2022)
  139. bContinuous Auctions and Insider Trading, 0 Econometrica, November 1985, 53 (6), 1315~ 1335., bInformed Speculation with Imperfect Competition, 0 Review of Economic Studies, July 1989, 56 (3), 317~ 356. and F. Albert Wang, bSpeculation Duopoly With Agreement to Disagree: Can Overcon7 fidence Survive the Market Test (1997)
  140. A Non (Random Walk Down Wall (Street, Princeton Uni (versity Press, 1999 (1999)
  141. Giving content to investor sentiment: The role of media in the stock market (2007)
  142. Algorithmic and high-frequency trading (2015)
  143. Hedging options for a large investor and forward-backward SDE’s (1996)
  144. Fundamentals of stochastic filtering (2009)
  145. A new approach to the economic analysis of nonstationary time series and the business cycle (1989)

4 of 153 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.