Improving Bayesian Optimization for Portfolio Management with an Adaptive Scheduling

ArXiv ID: 2504.13529 “View on arXiv”

Authors: Unknown

Abstract

Existing black-box portfolio management systems are prevalent in the financial industry due to commercial and safety constraints, though their performance can fluctuate dramatically with changing market regimes. Evaluating these non-transparent systems is computationally expensive, as fixed budgets limit the number of possible observations. Therefore, achieving stable and sample-efficient optimization for these systems has become a critical challenge. This work presents a novel Bayesian optimization framework (TPE-AS) that improves search stability and efficiency for black-box portfolio models under these limited observation budgets. Standard Bayesian optimization, which solely maximizes expected return, can yield erratic search trajectories and misalign the surrogate model with the true objective, thereby wasting the limited evaluation budget. To mitigate these issues, we propose a weighted Lagrangian estimator that leverages an adaptive schedule and importance sampling. This estimator dynamically balances exploration and exploitation by incorporating both the maximization of model performance and the minimization of the variance of model observations. It guides the search from broad, performance-seeking exploration towards stable and desirable regions as the optimization progresses. Extensive experiments and ablation studies, which establish our proposed method as the primary approach and other configurations as baselines, demonstrate its effectiveness across four backtest settings with three distinct black-box portfolio management models.

Keywords: Bayesian Optimization, Black-box Optimization, Portfolio Management, Exploration-Exploitation, Sample Efficiency, Multi-Asset

Complexity vs Empirical Score

• Math Complexity: 7.5/10
• Empirical Rigor: 8.0/10
• Quadrant: Holy Grail
• Why: The paper involves advanced Bayesian optimization concepts like Lagrangian estimators, TPE, and importance sampling, requiring solid mathematical foundations. It also demonstrates empirical rigor through extensive experiments and ablation studies across multiple backtest settings with real-world portfolio models.

  flowchart TD
    A["Research Goal: Stable & Sample-Efficient<br>Black-Box Portfolio Optimization"] --> B{"Key Methodology:<br>TPE-AS Framework"}
    B --> C["Adaptive Scheduling<br>Dynamically balances<br>Exploration vs. Exploitation"]
    C --> D["Weighted Lagrangian Estimator<br>Minimizes Performance Variance<br>+ Maximizes Return"]
    D --> E["Computational Process:<br>Bayesian Optimization with<br>Limited Evaluation Budgets"]
    E --> F["Data Inputs:<br>3 Black-Box Models<br>4 Backtest Settings"]
    F --> G["Outcomes:<br>Superior Stability &<br>Sample Efficiency"]