Paper: arXiv 2306.12639

Authors: Cassidy K. Buhler, Hande Y. Benson

Abstract

The Markowitz mean-variance portfolio optimization model aims to balance expected return and risk when investing. However, there is a significant limitation when solving large portfolio optimization problems efficiently: the large and dense covariance matrix. Since portfolio performance can be potentially improved by considering a wider range of investments, it is imperative to be able to solve large portfolio optimization problems efficiently, typically in microseconds. We propose dimension reduction and increased sparsity as remedies for the covariance matrix. The size reduction is based on predictions from machine learning techniques and the solution to a linear programming problem. We find that using the efficient frontier from the linear formulation is much better at predicting the assets on the Markowitz efficient frontier, compared to the predictions from neural networks. Reducing the covariance matrix based on these predictions decreases both runtime and total iterations. We also present a technique to sparsify the covariance matrix such that it preserves positive semi-definiteness, which improves runtime per iteration. The methods we discuss all achieved similar portfolio expected risk and return as we would obtain from a full dense covariance matrix but with improved optimizer performance.

Complexity vs Empirical Score

  • Math Complexity: 7.0/10
  • Empirical Rigor: 6.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper is mathematically dense, featuring formal optimization formulations and linear algebra concepts like positive semi-definiteness. It also demonstrates high empirical rigor by testing on real financial data (S&P 500) with specific runtime improvements, though it lacks the full backtest/trading implementation details of a ‘Street Trader’.

Research Flowchart

  flowchart TD
  A["Research Goal:<br>Efficiently solve large-scale<br>portfolio optimization problems<br>overcoming dense covariance matrix issues"] --> B["Key Methodology:<br>Dimension Reduction & Sparsification<br>using ML predictions & LP solutions"]

  B --> C["Input Data:<br>Portfolio assets data<br>Expected returns &<br>full covariance matrix"]

  C --> D["Core Process:<br>Predict asset importance<br>via Linear Programming<br>& Neural Networks"]

  D --> E["Covariance Matrix<br>Transformation:<br>1. Reduce size (select assets)<br>2. Sparsify (preserve PSD)"]

  E --> F["Compute Outcomes:<br>Risk (Variance) & Return<br>Optimization Runtime & Iterations"]

  F --> G["Key Findings:<br>LP predictions outperform Neural Networks<br>Reduced matrices yield similar risk/return<br>Runtime & iterations significantly improved"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes dimension reduction and increased sparsity of the covariance matrix to improve large‑scale portfolio optimization.

Source sentence
We propose dimension reduction and increased sparsity as remedies for the covariance matrix.

Assumes 4

  • Predictive models (ML or LP) will sufficiently match the assets in the optimal Markowitz portfolios. model
    Source sentence
    Our hypothesis is that the predictive models will give a sufficient match to the investments in the optimal portfolios.
  • Correlations close to zero can be set to zero while preserving positive semidefiniteness of the covariance matrix. model
    Source sentence
    Correlations close to zero can be replaced by zero, which yields a sparse matrix. Care must be taken to retain positive semidefiniteness.
  • Large correlations and variances are sufficient to determine the optimal portfolios and represent their risk. model
    Source sentence
    The main hypothesis for this technique is that the variance and large correlations are sufficient to determine the optimal portfolios and represent their risk.
  • Daily closing prices of S&P 500 constituents provide a representative dataset for testing the methods. data
    Source sentence
    Our financial data, which consists of daily closing prices for individual stocks in the S&P 500 index.

Data

  • S&P 500 daily closing prices (374 stocks) market-prices unknown
    S&P 500 constituents
    Source sentence
    The efficient frontier for 374 stocks selected from the S&P500 using the…
  • S&P 500 daily closing prices (497 assets, 502 days) market-prices unknown
    November 21 2017 to November 19 2019 · S&P 500 constituents
    Source sentence
    We repeated the above experiment with a shorter dataset that has more stocks: 497 assets for 502 days starting November 21 2017 to November 19th 2019.

Recognised sources: Yahoo Finance

Costs

Not modeled A trading or execution result is reported without costs.

Finds 3

  • Reducing the covariance matrix based on ML or LP predictions decreases both runtime and total iterations while preserving portfolio risk and return. performance
    Source sentence
    Reducing the covariance matrix based on these predictions decreases both runtime and total iterations.
  • All three proposed methods achieve similar expected risk and return as the full dense covariance matrix but with improved optimizer performance. performance
    Source sentence
    The methods we discuss all achieved similar portfolio expected risk and return as we would obtain from a full dense covariance matrix but with improved optimizer performance.
  • Sparsification by correlation reduces runtime per iteration while increasing total runtime per iteration, and higher sparsity leads to fewer optimizer iterations. performance
    Source sentence
    Reducing the size of the covariance matrix decreased the number of iterations and CPU run‑time, yet increased the runtime per iteration.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 18 related papers). Coverage is partial, so this is neither replication nor refutation.

Code

https://github.com/cassiebuhler/PODS — the authors' own repository

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.