Paper: arXiv 2509.23280

Authors: Yilie Huang

Abstract

This paper proposes a novel approach for Asset-Liability Management (ALM) by employing continuous-time Reinforcement Learning (RL) with a linear-quadratic (LQ) formulation that incorporates both interim and terminal objectives. We develop a model-free, policy gradient-based soft actor-critic algorithm tailored to ALM for dynamically synchronizing assets and liabilities. To ensure an effective balance between exploration and exploitation with minimal tuning, we introduce adaptive exploration for the actor and scheduled exploration for the critic. Our empirical study evaluates this approach against two enhanced traditional financial strategies, a model-based continuous-time RL method, and three state-of-the-art RL algorithms. Evaluated across 200 randomized market scenarios, our method achieves higher average rewards than all alternative strategies, with rapid initial gains and sustained superior performance. The outperformance stems not from complex neural networks or improved parameter estimation, but from directly learning the optimal ALM strategy without learning the environment.

Complexity vs Empirical Score

  • Math Complexity: 8.0/10
  • Empirical Rigor: 7.5/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper provides a strong theoretical bridge by formalizing continuous-time reinforcement learning for ALM via a linear-quadratic framework. It combines rigorous stochastic control calculus with practical, data-driven algorithm design, demonstrating solid empirical performance against multiple standard benchmarks across 200 market scenarios.

Research Flowchart

  flowchart TD
    A[Research Goal: Dynamic Asset-Liability Management via Continuous-Time RL] --> B[Methodology: Linear-Quadratic Soft Actor-Critic Framework]
    B --> C[Inputs: Interim & Terminal Objectives + Asset/Liability Market Data]
    C --> D[Computational Engine: Model-Free RL with Adaptive & Scheduled Exploration]
    D --> E[Empirical Evaluation: Tested across 200 Randomized Market Scenarios]
    E --> F[Outcomes: Outperforms Traditional & Model-Based RL Strategies via Direct Policy Learning]

Cited References (270)

  1. The patent holder’s dilemma: Buy, sell, or troll? (2007) — link
  2. Deciding equivalances among conjunctive aggregate queries (2007) — link
  3. Special issue: Digital Libraries (1996)
  4. Understanding Policy-Based Networking (2001)
  5. First-Order Dynamic Logic (1979) — link
  6. The title of book one (2007) — link
  7. The title of book two (2008) — link
  8. Achieving application requirements (1990) — link
and 262 more
  1. Statecarts in use: structured analysis and object-orientation (1998) — link
  2. The Art of Computer Programming, Vol. 1: Fundamental Algorithms (3rd. ed.) (1997)
  3. The Art of Computer Programming (1998)
  4. An experiment in bibliographic mark-up: Parsing metadata for XML export (2010) — link
  5. Catch me, if you can: Evading network signatures with web-based polymorphic worms (2007)
  6. Catch me, if you can: Evading network signatures with web-based polymorphic worms (2008)
  7. Catch me, if you can: Evading network signatures with web-based polymorphic worms (2009)
  8. Predicate Path expressions (1979) — link
  9. LOGICS of Programs: AXIOMATICS and DESCRIPTIVE POWER (1978)
  10. Optimal Motion Control of a Ground Vehicle (2003)
  11. Algorithms for Closest-Point Problems (Computational Geometry) (1985)
  12. Introduction to Bayesian Statistics (2001) — link
  13. CLIFFORD: a Maple 11 Package for Clifford Algebra Computations, version 11 (2007) — link
  14. Stats and Analysis (2006) — link
  15. A more perfect union (2008) — link
  16. The fountain of youth (2009)
  17. Solder man (2003) — link
  18. Interview with Bill Kinder: January 13, 2005 (2005) — link
  19. The Enabling of Digital Libraries (2008)
  20. (old) Finding minimum congestion spanning trees (2000) — link
  21. (new) Finding minimum congestion spanning trees (2000) — link
  22. (old) Distributed data source verification in wireless sensor networks (2009) — link
  23. (new) Distributed data source verification in wireless sensor networks (2009) — link
  24. Portalis: using competitive online interactions to support aid initiatives for the homeless (2008) — link
  25. Visual Basic 6: Design, Specification, and Objects with Other (1999)
  26. The Latex Web Companion: Integrating TEX, HTML, and XML (1999)
  27. Vertex Types in Book-Embeddings (1987)
  28. Vertex Types in Book-Embeddings (1987)
  29. CHI ‘08: CHI ‘08 extended abstracts on Human factors in computing systems (2008)
  30. Algorithms for Closest-Point Problems (Computational Geometry) (1985)
  31. IEEE TCSC Executive Committee (2004) — link
  32. Distributed systems (2nd Ed.) (1993)
  33. New Algorithms for Dependency-Directed Backtracking (Master’s thesis) (1986)
  34. New Algorithms for Dependency-Directed Backtracking (Master’s thesis) (1986)
  35. Seminumerical Algorithms (1981)
  36. E-commerce and cultural values (2001) — link
  37. Chapter 9 (2002) — link
  38. The implementation of electronic commerce in SMEs in Singapore (Incoll) (2003) — link
  39. E-commerce and cultural values - (InBook-num-in-chap) (2004) — link
  40. E-commerce and cultural values (Inbook-text-in-chap) (2005) — link
  41. E-commerce and cultural values (Inbook-num chap) (2006) — link
  42. A library-based synthesis methodology for reversible logic (2010)
  43. Synthesis of Reversible Circuit Using Cycle-Based Approach (2010)
  44. Algorithmic Enumeration of Ideal Classes for Quaternion Orders (2010) — link
  45. Chapter II: Notes on data structuring (1972) — link
  46. Transcript of question and answer session (1981) — link
  47. Go to statement considered harmful (1979) — link
  48. Three-dimensional virtual acoustic displays (1992) — link
  49. Managerial expert systems and organizational change: some critical research issues (1987) — link
  50. Simplified Structured COBOL with Microsoft/MicroFocus COBOL (1990)
  51. A multi-radio unification protocol for IEEE 802.11 wireless networks (2004)
  52. Wireless Sensor Networks: A Survey (2002)
  53. A Survey on Wireless Multimedia Sensor Networks (2007)
  54. SSCH : Slotted Seeded Channel Hopping for Capacity Improvement in IEEE 802.11 Ad-Hoc Wireless Networks (2004)
  55. XBOW Sensor Motes Specifications (2008)
  56. Overview of Sensor Networks (2004)
  57. CodeBlue : Sensor Networks for Medical Care (2008)
  58. Investigating Network Architectures for Body Sensor Networks (2007) — link
  59. Channel-Hopping Multiple Access (2000)
  60. Body Sensor Networks (2008)
  61. Mapping Powerlists onto Hypercubes (1994)
  62. Automatic Parallelization for Distributed-Memory Multiprocessing Systems (1989)
  63. User recovery and reversal in interactive systems (1984)
  64. Generalizing specifications for uniformly implemented loops (1985)
  65. Towards monolingual programming environments (1985)
  66. The \TeX book (1984)
  67. Distributed algorithms for finding centers and medians in networks (1984)
  68. \it \LaTeX : A Document Preparation System (1986)
  69. Program transformations in a denotational setting (1985)
  70. Seminumerical Algorithms (1981)
  71. A high-level approach to computer document formatting (1980)
  72. A multifrequency MAC specially designed for wireless sensor network applications (2010) — link
  73. Institutional members of the \TeX Users Group (2017) — link
  74. acmart— C lass for typesetting publications of ACM (2017) — link
  75. Reasoning About Naming Systems (1993) — link
  76. Babel, a Multilingual Style-Option System for Use with LaTeX’s Standard Document Styles (1991)
  77. A Methodology for Implementing Highly Concurrent Data Objects (1993) — link
  78. Publication quality tables in \LaTeX (2005)
  79. Using the amsthm Package (2015)
  80. R: A Language and Environment for Statistical Computing (2019) — link
  81. UMass Citation Field Extraction Dataset (2013) — link
  82. Citation concept analysis (CCA) —A new form of citation analysis revealing the usefulness of concepts for other researchers illustrated by two exemplary case studies including classic books by Thomas S.~Kuhn and Karl R.~Popper (2019) — link
  83. Learning Soft Linear Constraints with Application to Citation Field Extraction (2014) — link
  84. Maintaining Discrete Probability Distributions Optimally (1993)
  85. Improved estimation of the covariance matrix of stock returns with an application to portfolio selection (2003)
  86. Optimal versus naive diversification: How inefficient is the 1/N portfolio strategy? (2009)
  87. Portfolio selection (1952)
  88. Bayes- S tein estimation for portfolio analysis (1986)
  89. Common risk factors in the returns on stocks and bonds (1993)
  90. Asset allocation: combining investor views with market equilibrium (1990)
  91. Distributionally robust mean-variance portfolio selection with W asserstein distances (2021)
  92. Capital asset prices: A theory of market equilibrium under conditions of risk (1964)
  93. Risk reduction in large portfolios: Why imposing the wrong constraints helps (2003)
  94. Estimation with Quadratic Loss (1961)
  95. Minimum-variance portfolios in the US equity market (2006)
  96. A simplified model for portfolio analysis (1963)
  97. Individual comparisons by ranking methods (1992)
  98. Policy evaluation and temporal-difference learning in continuous time and space: A martingale approach (2022)
  99. Policy gradient and actor-critic learning in continuous time and space: Theory and algorithms (2022)
  100. Reinforcement learning in continuous time and space (2000)
  101. Reinforcement learning in continuous time and space: A stochastic control approach (2020)
  102. Continuous-time mean–variance portfolio selection: A reinforcement learning framework (2020)
  103. Reinforcement Learning: An Introduction (2018)
  104. Stochastic Controls: Hamiltonian Systems and HJB Equations (1999)
  105. Controlled Markov Processes and Viscosity Solutions (2006)
  106. Investment Science (1998)
  107. A closed-form solution for options with stochastic volatility with applications to bond and currency options (1993)
  108. Smart “predict, then optimize” (2022)
  109. Optimal dynamic portfolio selection: Multiperiod mean-variance formulation (2000)
  110. Continuous-time mean-variance portfolio selection: A stochastic LQ framework (2000)
  111. Portfolio selection in stochastic environments (2007)
  112. Markowitz’s mean-variance portfolio selection with regime switching: A continuous-time model (2003)
  113. Numerical solution of the Hamilton–Jacobi–Bellman formulation for continuous-time mean–variance asset allocation under stochastic volatility (2016)
  114. Mean–variance portfolio selection of cointegrated assets (2011)
  115. Mean-variance portfolio selection with random parameters in a complete market (2002)
  116. A dynamic mean-variance analysis for log returns (2021)
  117. Dynamic mean-variance asset allocation (2010)
  118. Portfolio and consumption decisions under mean-reverting returns: An exact solution for complete markets (2002)
  119. On estimating the expected return on the market: An exploratory investigation (1980)
  120. On the sensitivity of mean-variance-efficient portfolios to changes in asset means: Some analytical and computational results (1991)
  121. The sampling error in estimates of mean-variance efficient portfolio weights (1999)
  122. On portfolio optimization: Forecasting covariances and choosing the risk model (1999)
  123. A generalized approach to portfolio optimization: Improving performance by constraining portfolio norms (2009)
  124. The effect of errors in means, variances, and covariances on optimal portfolio choice (2013)
  125. Mean–Variance Optimization for Asset Allocation (2021)
  126. On the existence of optimal relaxed controls of stochastic partial differential equations (1992)
  127. On stochastic relaxed control for partially observed diffusions (1984)
  128. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor (2018)
  129. Maximum entropy inverse reinforcement learning. (2008)
  130. Sensitivity analysis for mean-variance portfolio problems (1991)
  131. Dynamic Nonmyopic Portfolio Behavior (1996)
  132. Consumption and portfolio decisions when expected returns are time varying (1999)
  133. Human-level control through deep reinforcement learning (2015)
  134. End-to-end training of deep visuomotor policies (2016)
  135. A survey on policy search for robotics (2013)
  136. Estimating Portfolio and Consumption Choice: A Conditional E uler Equations Approach (1999)
  137. Vast portfolio selection with gross-exposure constraints (2012)
  138. Variable Selection for Portfolio Choice (2001)
  139. Strategic Asset Allocation: Portfolio Choice for Long-term Investors (2002)
  140. Continuous-time Markowitz’s model with transaction costs (2010)
  141. Continuous-time mean-variance efficiency: the 80% rule (2006)
  142. A theory of Markovian time-inconsistent stochastic control in discrete time (2014)
  143. Mean–variance portfolio optimization with state-dependent risk aversion (2014)
  144. Myopia and inconsistency in dynamic utility maximization (1955)
  145. Being serious about non-commitment: subgame perfect equilibrium in continuous time (2006)
  146. Time-consistent portfolio management (2012)
  147. Consumption and portfolio rules for time-inconsistent investors (2010)
  148. Investment and consumption without commitment (2008)
  149. The golden rule when preferences are time inconsistent (2010)
  150. Time-inconsistent stochastic linear-quadratic control: characterization and uniqueness of equilibrium (2017)
  151. Time-inconsistent stochastic linear–quadratic control (2012)
  152. Time-inconsistent recursive stochastic optimal control problems (2017)
  153. Time-inconsistent optimal control problems and the equilibrium HJB equation (2012) — link
  154. Mean-variance portfolio selection with dynamic targets for expected terminal wealth (2022)
  155. Portfolio analysis under uncertain means, variances, and covariances (1974)
  156. Inadmissibility of the usual estimator for the mean of a multivariate normal distribution (1956)
  157. Portfolio selection with parameter and model uncertainty: A multi-prior approach (2007)
  158. Robust portfolio selection problems (2003)
  159. Maxmin expected utility with a non-unique prior (1989)
  160. Portfolio diversification and model uncertainty: A robust dynamic mean-variance approach (2022)
  161. Robust Markowitz mean-variance portfolio selection under ambiguous covariance matrix (2019)
  162. Machine learning and portfolio optimization (2018)
  163. Robust satisficing (2022)
  164. Nonlinear shrinkage of the covariance matrix for portfolio selection: Markowitz meets Goldilocks (2017)
  165. High dimensional covariance matrix estimation using a factor model (2008)
  166. FinBrain: when finance meets AI 2.0 (2019)
  167. Portfolio management using reinforcement learning (2016)
  168. An algorithm for trading and portfolio management using Q-learning and Sharpe ratio maximization (2000)
  169. Optimal asset allocation using adaptive dynamic programming (1996)
  170. Machine learning for trading (2017)
  171. Algorithmic aspects of mean–variance optimization in Markov decision processes (2013)
  172. The variance of discounted Markov decision processes (1982)
  173. TD algorithm for the variance of return and mean-variance reinforcement learning (2001)
  174. Variance-penalized reinforcement learning for risk-averse asset allocation (2000)
  175. Variance adjusted actor critic algorithms (2013) — link
  176. Variance-constrained actor-critic algorithms for discounted and average reward MDPs (2016)
  177. Actor-critic algorithms for risk-sensitive MDPs (2013)
  178. Reinforcement learning: An introduction (2018)
  179. A stochastic approximation algorithm with varying bounds (1995)
  180. General bounds and finite-time improvement for the Kiefer-Wolfowitz stochastic approximation algorithm (2011)
  181. A stochastic approximation method (1951)
  182. Stochastic Approximation (2003)
  183. Stochastic approximation: A dynamical systems viewpoint (2009)
  184. An overview of stochastic approximation (2014)
  185. Generalized instrumental variables estimation of nonlinear rational expectations models (1982)
  186. A practical end-to-end inventory management model with deep learning (2023)
  187. A convergence theorem for non negative almost supermartingales and some applications (1971)
  188. Dynamic Programming and Optimal Control (1995)
  189. Inventory and production under uncertainty: The multi-item case (1973)
  190. Provably global convergence of actor-critic: A case for linear quadratic regulator with ergodic cost (2019)
  191. Global convergence of two-timescale actor-critic for solving linear quadratic regulator (2023)
  192. Single timescale actor-critic method to solve the linear quadratic regulator with convergence guarantees (2023)
  193. A finite-time analysis of two time-scale actor-critic methods (2020)
  194. Actor-critic algorithms (1999)
  195. Doubly robust off-policy actor-critic: Convergence and optimality (2021)
  196. Achieving mean–variance efficiency by continuous-time reinforcement learning (2022)
  197. Stochastic linear quadratic regulators with indefinite control weight costs (1998)
  198. Stochastic controls: Hamiltonian systems and HJB equations (2012)
  199. Investment Science (1998)
  200. Logarithmic regret for episodic continuous-time linear-quadratic reinforcement learning over a finite-time horizon (2022)
  201. Asynchronous methods for deep reinforcement learning (2016)
  202. Continuous control with deep reinforcement learning (2015) — link
  203. Mastering the game of Go without human knowledge (2017)
  204. Mastering the game of Go with deep neural networks and tree search (2016)
  205. Deep direct reinforcement learning for financial signal representation and trading (2016)
  206. A deep reinforcement learning framework for the financial portfolio management problem (2017) — link
  207. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates (2017)
  208. A systematic review on reinforcement learning-based robotics within the last decade (2020)
  209. Linear matrix inequalities, R iccati equations, and indefinite stochastic linear quadratic controls (2000)
  210. Regret bounds for the adaptive control of linear quadratic systems (2011)
  211. Improved regret bounds for T hompson sampling in linear quadratic control problems (2018)
  212. Policy gradient methods for the noisy linear quadratic regulator over a finite horizon (2021)
  213. Optimal Scheduling of Entropy Regularizer for Continuous-Time Linear-Quadratic Reinforcement Learning (2024)
  214. Reinforcement learning in continuous time: Advantage updating (1994)
  215. Online actor–critic algorithm to solve the continuous-time infinite horizon optimal control problem (2010)
  216. Policy iterations for reinforcement learning problems in continuous time and space—Fundamental theory and methods (2021)
  217. Hamilton-Jacobi deep Q-learning for deterministic continuous-time systems with Lipschitz continuous controls (2021)
  218. Reinforcement learning applied to linear quadratic regulation (1992)
  219. Global convergence of policy gradient for linear-quadratic mean-field control/game in continuous time (2021)
  220. Global convergence of policy gradient methods for the linear quadratic regulator (2018)
  221. Derivative-free methods for policy optimization: Guarantees for linear quadratic systems (2019)
  222. Online Policy Gradient for Model Free Learning of Linear Quadratic Regulators with $\sqrt T$ Regret (2021)
  223. Online linear quadratic control (2018)
  224. Learning Linear-Quadratic Regulators Efficiently with only $\sqrt T $ Regret (2019)
  225. q- L earning in continuous time (2023)
  226. Optimal control: Linear quadratic methods (2007)
  227. Dynamic programming and optimal control: Volume I (2012)
  228. Reinforcement learning for continuous-time optimal execution: actor-critic algorithm and error analysis (2023)
  229. Learning equilibrium mean-variance strategy (2023)
  230. Actor-Critic learning for mean-field control in continuous time (2023) — link
  231. Continuous-time q-learning for McKean-Vlasov control problems — → 2306.16208
  232. Policy gradient methods for reinforcement learning with function approximation (1999)
  233. Policy gradient in continuous time (2006)
  234. Making deep Q -learning methods robust to time discretization (2019)
  235. Time discretization-invariant safe action repetition for policy gradient methods (2021)
  236. Fast global convergence of natural policy gradient methods with entropy regularization (2022)
  237. Provably efficient exploration in policy optimization (2020)
  238. A theoretical analysis of optimistic proximal policy optimization in linear Markov decision processes (2023)
  239. Constant proportion portfolio insurance for fixed-income investment (1989)
  240. Theory of constant proportion portfolio insurance (1992)
  241. Multiplier optimization for constant proportion portfolio insurance (cppi) strategy (2020)
  242. A new choice of dynamic asset management: the variable proportion portfolio insurance (2008)
  243. Contingent immunization—Part I: Risk control procedures (1982)
  244. Contingent Immunization—Part II: Problem Areas (1983)
  245. Proximal policy optimization algorithms (2017) — link
  246. Time-consistent optimal portfolio strategy for asset-liability management under mean-variance criterion (2013)
  247. Mean–variance asset–liability management: Cointegrated assets and insurance liability (2012)
  248. Continuous-time mean–variance asset–liability management with endogenous liabilities (2013)
  249. Mean–variance asset–liability management under constant elasticity of variance process (2016)
  250. Strategies and techniques for asset-liability management: an overview (1993)
  251. The matching of assets to liabilities (1984)
  252. Key rate durations: Measures of interest rate risks (1992)
  253. Deep reinforcement learning for trading (2019) — link
  254. A deep reinforcement learning approach to asset-liability management (2019)
  255. Application of deep reinforcement learning in asset liability management (2023)
  256. Data-Driven Exploration for a Class of Continuous-Time Linear–Quadratic Reinforcement Learning Problems (2025) — link
  257. Sublinear regret for a class of continuous-time linear-quadratic reinforcement learning problems (2025)
  258. Mean–Variance Portfolio Selection by Continuous-Time Reinforcement Learning: Algorithms, Regret Analysis, and Empirical Study — → 2412.16175
  259. Optimal dynamic mean-variance asset-liability management under the Heston model (2018)
  260. Non-homogeneous stochastic LQ control with regime switching and random coefficients (2022) — link
  261. Mean–variance asset–liability management problem under non-Markovian regime-switching models (2020)
  262. Regret of exploratory policy improvement and $ q $-learning (2024) — link

2 of 270 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.