Paper: arXiv 2603.09301

Authors: Hanyong Cho, Jang Ho Kim

Abstract

This study introduces a benchmark framework for evaluating the financial decision-making capabilities of large language models (LLMs) through portfolio optimization problems with mathematically explicit solutions. Unlike existing financial benchmarks that emphasize language-processing tasks, the proposed framework directly tests optimization-based reasoning in investment contexts. A large set of multiple-choice questions is generated by varying objectives, candidate assets, and investment constraints, with each problem designed to include a unique correct solution and systematically constructed alternatives. Experimental results comparing GPT-4, Gemini 1.5 Pro, and Llama 3.1-70B reveal distinct performance patterns: GPT achieves the highest accuracy in risk-based objectives and remains stable under constraints, Gemini performs well in return-based tasks but struggles under other conditions, and Llama records the lowest overall performance. These findings highlight both the potential and current limitations of LLMs in applying quantitative reasoning to finance, while providing a scalable foundation for developing LLM-based services in portfolio management.

Complexity vs Empirical Score

  • Math Complexity: 4.0/10
  • Empirical Rigor: 6.0/10 (capped from 7.5: no out-of-sample test; see penalty flags)
  • Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: This paper presents a novel benchmark framework (PortBench) for evaluating LLMs on quantitative portfolio optimization rather than standard NLP tasks. While the mathematical concepts rely on standard mean-variance and convex optimization formulations, the empirical evaluation across 9,500 generated questions and multiple state-of-the-art LLMs is rigorous and highly reproducible via a public GitHub repository.

Research Flowchart

  flowchart TD
    A[Goal: Evaluate LLMs on Quantitative Portfolio Optimization] --> B[Data Input: Varying Objectives, Assets & Investment Constraints]
    B --> C[Framework Creation: Generate MCQs with Unique Exact Solutions & Distractors]
    C --> D[Model Testing: Evaluate GPT-4, Gemini 1.5 Pro & Llama 3.1-70B]
    D --> E[Outcome: GPT-4 Leads in Risk Objectives & Constraint Stability]
    D --> F[Outcome: Gemini Excels in Return Tasks; Llama 3.1 Lowest Overall]