Paper: arXiv 2507.07159

Authors: Nishan Ranabhat, Behnam Javanparast, David Goerz, Estelle Inack

Abstract

Portfolio optimization is a routine asset management operation conducted in financial institutions around the world. However, under real-world constraints such as turnover limits and transaction costs, its formulation becomes a mixed-integer nonlinear program that current mixed-integer optimizers often struggle to solve. We propose mapping this problem onto a classical Ising-like Hamiltonian and solving it with Variational Neural Annealing (VNA), via its classical formulation implemented using autoregressive neural networks. We demonstrate that VNA can identify near-optimal solutions for portfolios comprising more than 2,000 assets and yields performance comparable to that of state-of-the-art optimizers, such as Mosek, while exhibiting faster convergence on hard instances. Finally, we present a dynamical finite-size scaling analysis applied to the S&P 500, Russell 1000, and Russell 3000 indices, revealing universal behavior and polynomial annealing time scaling of the VNA algorithm on portfolio optimization problems.

Complexity vs Empirical Score

  • Math Complexity: 8.5/10
  • Empirical Rigor: 8.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper introduces high-level mathematical concepts such as Ising Hamiltonian mapping, variational neural annealing, and dynamical finite-size scaling, demonstrating substantial theoretical depth. It validates its approach with large-scale backtests on major indices (S&P 500, Russell 1000/3000) and benchmarks against commercial solvers like Mosek, showing strong empirical implementation and performance analysis.

Research Flowchart

  flowchart TD
  A["Research Goal<br>Optimize portfolios with constraints<br>over 2,000 assets"] --> B["Methodology<br>Variational Neural Annealing<br>Autoregressive Neural Networks"]
  B --> C["Formulation<br>Map to Ising-like Hamiltonian<br>Mixed-Integer Nonlinear Program"]
  C --> D["Data Sources<br>S&P 500, Russell 1000,<br>Russell 3000 Indices"]
  D --> E["Computational Process<br>Dynamical Finite-Size Scaling Analysis"]
  E --> F{"Findings"}
  F --> G["Near-optimal solutions<br>Comparable to Mosek<br>Faster on hard instances"]
  F --> H["Scaling Laws<br>Universal behavior revealed<br>Polynomial annealing time"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes mapping large-scale portfolio optimization onto a classical Ising-like Hamiltonian and solving it with Variational Neural Annealing (VNA) using autoregressive neural networks.

Source sentence
We propose mapping this problem onto a classical Ising-like Hamiltonian and solving it with Variational Neural Annealing (VNA), via its classical formulation implemented using autoregressive neural networks.

Assumes 6

  • The covariance matrix C is positive semi‑definite, making the continuous‑valued objective convex. model
    Source sentence
    The covariance matrix C is positive semi-definite by construction, and the objective function is therefore convex.
  • Asset returns are estimated from monthly closing prices using logarithmic returns over a five‑year look‑back window. data
    Source sentence
    Closing prices with monthly granularity are used to compute expected returns and risk terms. µ is computed as a logarithm returns, since it is time-additive, and the covariance matrix C is built with a five-year look-back window.
  • Only assets that are present throughout the entire ten‑year trading period are included. execution
    Source sentence
    Only assets that are consistently present during the entire trading period are considered.
  • Transaction costs consist of a variable cost proportional to trade volume and a fixed cost proportional to the number of trades. market
    Source sentence
    The transaction cost can be of two types: the first is variable cost, which is proportional to the volume of the trade, and the second is fixed cost, which is proportional to the number of trades.
Show 2 more
  • The VNA algorithm’s runtime scales linearly with the number of assets per gradient step (O(N)). statistical
    Source sentence
    Other factors include the computational complexity of VNA, which is of O(N) for each gradient descent step when implemented with a RNN.
  • The normalization penalty term can be adjusted to incorporate transaction costs, assuming the penalty parameters can be made arbitrarily large. model
    Source sentence
    the normalization penalty term is altered to include the transaction costs, ρn(1T x + Costv + Costf −1)2.

Data

  • S&P 500 historical prices market-prices public
    ten years · 478 stocks (S&P 500 constituents present throughout period)
    Source sentence
    Our dataset is made up of historical prices over ten years of the S&P 500, Russell 1000 and Russell 3000 indices.
  • Russell 1000 historical prices market-prices public
    ten years · 857 stocks (Russell 1000 constituents present throughout period)
    Source sentence
    Our dataset is made up of historical prices over ten years of the S&P 500, Russell 1000 and Russell 3000 indices.
  • Russell 3000 historical prices market-prices public
    ten years · 2008 stocks (Russell 3000 constituents present throughout period)
    Source sentence
    Our dataset is made up of historical prices over ten years of the S&P 500, Russell 1000 and Russell 3000 indices.

Costs

Modeled Costs are part of the reported headline results.

Finds 5

  • VNA identifies near‑optimal solutions for portfolios with more than 2,000 assets and achieves performance comparable to state‑of‑the‑art mixed‑integer solvers such as Mosek. performance
    Source sentence
    We demonstrate that VNA can identify near-optimal solutions for portfolios comprising more than 2,000 assets and yields performance comparable to that of state-of-the-art optimizers, such as Mosek, while exhibiting faster convergence on hard instances.
  • On hard instances VNA reaches a relative gap of 0.01 % in roughly half the time required by Mosek, even when Mosek’s stopping criterion is relaxed to 1 %. performance
    Source sentence
    Increasing the relative gap requirement by two orders of magnitude still resulted in Mosek having twice the time-to-solution as VNA.
  • The average Sharpe ratios obtained by VNA are statistically indistinguishable from those obtained by Mosek across S&P 500, Russell 1000 and Russell 3000 datasets. performance
    Source sentence
    The Sharpe ratio difference is minimal, indicating that VNA can yield comparable results to Mosek across different trading periods, sometimes with slightly better results.
  • Finite‑size scaling analysis reveals universal polynomial annealing‑time scaling with critical exponents µ = −0.58 ± 0.08 and κ = 0.16 ± 0.09. theoretical
    Source sentence
    The critical exponents extracted from this collapse are µ = −0.58 ± 0.08 and κ = 0.16 ± 0.09.
  • VNA’s runtime per gradient step scales linearly with the number of assets (O(N)), enabling scalability to thousands of assets. performance
    Source sentence
    Other factors include the computational complexity of VNA, which is of O(N) for each gradient descent step when implemented with a RNN.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 42 related papers). Coverage is partial, so this is neither replication nor refutation.

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (99)

  1. Spin glass theory and beyond: an introduction to the replica method and its applications (1987)
  2. A measure of data collapse for scaling (2001) — link
  3. Solvable model of a spin-glass (1975) — link
  4. Spin glasses: a computational challenge for the 21st century (2002) — link
  5. Rational decisions, random matrices and spin glasses (1998) — link
  6. Portfolio selection (1952) — link
  7. Tax-Aware Portfolio Construction via Convex Optimization (2021) — link
  8. Ising formulations of many NP problems (2014) — link
and 91 more
  1. On the computational complexity of Ising spin glass models (1982) — link
  2. Computational study of a family of mixed-integer quadratic programming problems (1996)
  3. Mixed-integer nonlinear optimization (2013)
  4. Structure and binding of Lennard–Jones clusters: $13\leq N\leq 147$ (1987) — link
  5. Thermodynamics and the global optimization of Lennard-Jones clusters (1998) — link
  6. Optimization in computational systems biology (2008) — link
  7. Optimization Algorithms for Computational Systems Biology (2017) — link
  8. Application of combinatorial optimization strategies in synthetic biology (2020) — link
  9. A survey of job shop scheduling problem: The types and models (2022) — link
  10. Healthcare scheduling in optimization context: a review (2021) — link
  11. The flexible job shop scheduling problem: A review (2024) — link
  12. Optimization by simulated annealing (1983) — link
  13. A Survey of simulated annealing applications to operations research problems (1994) — link
  14. A tutorial survey of theory and applications of simulated annealing (1985) — link
  15. Optimization by simulated annealing: an experimental evaluation; Part II, graph coloring and number partitioning (1991) — link
  16. Robust design of multimachine power system stabilizers using simulated annealing (2000) — link
  17. Strategic portfolio optimization using simulated, digital, and quantum annealing (2022) — link
  18. Universal nonequilibrium quantum dynamics in imaginary time (2011) — link
  19. Dynamic scaling at classical phase transitions approached through nonequilibrium quenching (2014) — link
  20. A quantum approximate optimization algorithm (2014) — link
  21. A variational eigenvalue solver on a photonic quantum processor (2014) — link
  22. A review on Quantum Approximate Optimization Algorithm and its variants (2024) — link
  23. Challenges and opportunities in quantum optimization (2024) — link
  24. Branch-and-bound digitized counterdiabatic quantum optimization (2025) — link
  25. From portfolio optimization to quantum blockchain and security: A systematic review of quantum computing in finance — → 2307.01155
  26. Quantum computational finance: quantum algorithm for portfolio optimization (2024) — link
  27. Wasserstein solution quality and the Quantum Approximate Optimization Algorithm: a portfolio optimization case study (2022) — link
  28. Benchmarking the performance of portfolio optimization with QAOA (2022) — link
  29. Best practices for portfolio optimization by quantum computing, experimented on real quantum devices (2023) — link
  30. Quantum walk-based portfolio optimisation (2021) — link
  31. Quantum computing for finance: overview and prospects (2019) — link
  32. Quantum computing for finance: state-of-the-art and future prospects (2020) — link
  33. Quantum computing for finance (2023) — link
  34. Reverse quantum annealing approach to portfolio optimization problems (2019) — link
  35. Quantum annealing with manufactured spins (2011) — link
  36. A coherent quantum annealer with Rydberg atoms (2017) — link
  37. Solving the optimal trading trajectory problem using a quantum annealer (2015) — link
  38. Quantum portfolio optimization with investment bands and target volatility (2021) — link
  39. Portfolio optimization of 60 stocks using classical and quantum algorithms (2020) — link
  40. Quantum critical dynamics in a 5,000-qubit programmable spin glass (2023) — link
  41. Scaling advantage in approximate optimization with quantum annealing (2025) — link
  42. Pushing the boundary of quantum advantage in hard combinatorial optimization with probabilistic computers (2025) — link
  43. Supplementing recurrent neural networks with annealing to solve combinatorial optimization problems (2023) — link
  44. Lattice protein folding with variational annealing (2025) — link
  45. Sparse autoregressive neural networks for classical spin systems (2024) — link
  46. Message passing variational autoregressive network for solving intractable Ising models (2024) — link
  47. A method for quantifying the generalization capabilities of generative models for solving Ising models (2024)
  48. Efficient optimization of variational autoregressive networks with natural gradient (2025) — link
  49. Variational neural annealing (2021) — link
  50. Enhancing combinatorial optimization with classical and quantum generative models (2024) — link
  51. Solving statistical mechanics using variational autoregressive networks (2019) — link
  52. Recurrent neural network wave functions (2020) — link
  53. Iterative retraining of quantum spin models using recurrent neural networks (2020) — link
  54. The density-matrix renormalization group in the age of matrix product states (2011) — link
  55. Quantum-inspired meta-heuristic approaches for a constrained portfolio optimization problem (2024) — link
  56. Dynamic portfolio optimization with real datasets using quantum processors and quantum-inspired tensor networks (2022) — link
  57. Decomposition pipeline for large-scale portfolio optimization with applications to near-term quantum computing — → 2409.10301
  58. Ising-model optimizer with parallel-trial bit-sieve engine (2018)
  59. Physics-inspired optimization for quadratic unconstrained problems using a digital annealer (2019) — link
  60. Solving the optimal trading trajectory problem using simulated bifurcation (2020) — link
  61. Real-time trading system based on selections of potentially profitable, uncorrelated, and balanced stocks by NP-hard combinatorial optimization — → 2307.06339
  62. Index tracking using Shapley additive explanations and one-dimensional pointwise convolutional autoencoders (2024) — link
  63. Deep learning for portfolio optimization (2020) — link
  64. StockGPT: a GenAI model for stock prediction and trading — → 2404.05101
  65. MILLION: a general multi-objective framework with controllable risk for portfolio management — → 2412.03038
  66. Graph neural networks for deep portfolio optimization (2023) — link
  67. Neural networks for portfolio analysis with cardinality constraints (2024) — link
  68. Hybrid data-driven and deep learning based portfolio optimization (2024) — link
  69. l1-regularization in portfolio selection with machine learning (2022) — link
  70. Enhancing Markowitz’s portfolio selection paradigm with machine learning (2024) — link
  71. End-to-End learning for stochastic optimization: a bayesian perspective (2023)
  72. E2EAI: End-to-End deep learning framework for active investing — → 2305.16364
  73. Generative AI for End-to-End limit order book modelling: a token-level autoregressive generative model of message flow using a deep state space network — → 2309.00638
  74. End-to-end risk budgeting portfolio optimization with neural networks (2024) — link
  75. Deep reinforcement learning for optimal portfolio allocation: a comparative study with mean-variance optimization (2023) — link
  76. Advancing investment frontiers: industry-grade deep reinforcement learning for portfolio optimization (2024) — link
  77. Deep reinforcement learning and mean-variance strategies for responsible portfolio optimization (2024) — link
  78. FreQuant: a reinforcement-learning based adaptive portfolio optimization with multi-frequency decomposition (2024) — link
  79. Deep learning in finance: a survey of applications and techniques (2024) — link
  80. Large Language Models in finance: a survey — → 2311.10723
  81. From deep learning to LLMs: a survey of AI in quantitative investment — → 2503.21422
  82. Recurrent neural network wave functions for Rydberg atom arrays on kagome lattice (2024) — link
  83. Investigating topological order using recurrent neural networks (2023) — link
  84. Leveraging recurrence in neural network wavefunctions for large-scale simulations of Heisenberg antiferromagnets: the triangular lattice (2025) — link
  85. Nonlinear programming (1997)
  86. Numerical Optimization (2006) — link
  87. Learning Constrained Optimization with Deep Augmented Lagrangian Methods (2024) — link
  88. An adaptive augmented Lagrangian method for training physics and equality constrained artificial neural networks (2023) — link
  89. Gurobi Optimizer
  90. IBM ILOG CPLEX Optimizer
  91. MOSEK Optimization Suite

9 of 99 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.