Paper: arXiv 2411.16569

Authors: Jerick Shi, Burton Hollifield

Abstract

Predicting the movement of the stock market and other assets has been valuable over the past few decades. Knowing how the value of a certain sector market may move in the future provides much information for investors, as they use that information to develop strategies to maximize profit or minimize risk. However, market data are quite noisy, and it is challenging to choose the right data or the right model to create such predictions. With the rise of large language models, there are ways to analyze certain data much more efficiently than before. Our goal is to determine whether the GPT model provides more useful information compared to other traditional transformer models, such as the BERT model. We shall use data from the Federal Reserve Beige Book, which provides summaries of economic conditions in different districts in the US. Using such data, we then employ the LLM’s to make predictions on the correlations. Using these correlations, we then compare the results with well-known strategies and determine whether knowing the economic conditions improves investment decisions. We conclude that the Beige Book does contain information regarding correlations amongst different assets, yet the GPT model has too much look-ahead bias and that traditional models still triumph.

Complexity vs Empirical Score

  • Math Complexity: 3.0/10
  • Empirical Rigor: 6.0/10 (capped from 7.0: transaction costs not modelled; see penalty flags)
  • Quadrant: Street Traders — empirical and applied, lower math complexity (effective empirical rigor capped from 7.0)

Why this score: The paper uses standard statistical methods (linear regression) and treats LLM outputs as classification/continuous signals, lacking advanced mathematical derivations. However, it involves real financial data, backtesting portfolio strategies, and implementing ML models with specific prompts, indicating significant implementation and empirical validation efforts.

Research Flowchart

  flowchart TD
  A["Research Goal:<br>Predict Market Movement<br>using Beige Book"] --> B["Data Input:<br>Federal Reserve Beige Book<br>Economic Summaries"]
  B --> C["Methodology:<br>Apply GPT vs BERT Models<br>for Sentiment Analysis"]
  C --> D["Computational Process:<br>Extract Asset Correlations<br>Test for Predictive Power"]
  D --> E{"Analysis of Outcomes"}
  E -->|"Look-ahead Bias"| F["Outcome: GPT Model<br>High Bias, Limited Utility"]
  E -->|"Traditional Analysis"| G["Outcome: BERT/Traditional Models<br>Outperform for Strategy"]
  F & G --> H["Conclusion:<br>Economic Conditions Contain Signal,<br>but LLMs Require Rigorous Validation"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes to evaluate look-ahead bias in GPT models and compare GPT versus BERT using Beige Book data for portfolio construction.

Source sentence
From this report, we make 3 main contributions. First, we show that regardless of how well the prompt is, there is always an unavoidable look-ahead bias when using the GPT model.

Assumes 4

  • The GPT-3.5 model is trained only on data up to September 2021. model
    Source sentence
    since the GPT-3.5 model is ’trained up to September 2021’, we shall use 1980-01 to 2021-09 as the training set
  • Historical stock and bond price series are accurate and can be used for linear regression without adjustment. data
    Source sentence
    We obtain historical stock prices directly from Yahoo Finance, where we take the prices from ˆGSPC from the first day of 1985 to the last day of 2023.
  • Rolling averages of volatility and correlations are sufficient inputs for the portfolio optimization. execution
    Source sentence
    σS ←rollingAverage(month, Stock) σB ←rollingAverage(month, Bond) ρmonth ←askGPT(beige data[month], ρ−3::−1)
  • The out‑of‑sample R² of 0.9901 indicates the bond regression model predicts AGG prices accurately. statistical
    Source sentence
    Our best result chooses the corporate index, 1-year, 2-year, 5-year, 7-year, and 10-year bill returns with an out-of-sample R2 of 0.9901.

Data

Backtest sample: US stocks and bonds · fixed-income · monthly

  • ˆGSPC · Yahoo Finance market-prices public
    1985-01-01 to 2023-12-31 · S&P 500 index
    Source sentence
    We obtain historical stock prices directly from Yahoo Finance, where we take the prices from ˆGSPC from the first day of 1985 to the last day of 2023.
  • AGG · Yahoo Finance market-prices public
    mid‑2003 to 2023-12-31 · Aggregate bond ETF
    Source sentence
    The bond data comes from the same source under the ticker AGG, but the data only span from the middle of 2003 to the end of 2023.
  • FRED bill returns & corporate index · Federal Reserve Economic Data (FRED) macro public
    1980 onward · U.S. Treasury bills (1‑,2‑,5‑,7‑,10‑,20‑,30‑year) and ICE BofA US Corporate Index
    Source sentence
    we were able to get 1-year, 2-year, 5-year, 7-year, 10-year, 20-year, 30-year bill returns as well as the ICE BofA US Corporate Index Total Return Index Value. These data go back to 1980.
  • Beige Book articles · Federal Reserve Bank of Minneapolis text-news public
    5 months per year, 12 states each month
    Source sentence
    Finally, the beige book is publicly available on the Federal Reserve Bank of Minneapolis. Each year, 5 months per year, and 12 states for each month, an article is published.

Costs

Not modeled A trading or execution result is reported without costs.

Finds 5

  • The GPT model exhibits unavoidable look‑ahead bias, as evidenced by higher out‑of‑sample RMSE compared to training RMSE. negative
    Source sentence
    If the error has a significant increase out of sample, that indicates that there is evidence of look-ahead bias.
  • BERT consistently outperforms GPT in portfolio simulations, delivering higher Sharpe ratios both pre‑ and post‑Covid. performance
    Source sentence
    We notice that in both periods, the BERT model tends to perform the best, whereas the GPT model sometimes even performs worse than simple rolling covariances.
  • For the 2‑Variable portfolio, BERT achieves a Sharpe ratio of 1.9208 pre‑Covid versus 0.445 for GPT. performance
    Source sentence
    Period Baseline BERT GPT Pre‑Covid 0.423 1.9208 0.445
  • For the Multi‑Variable portfolio, BERT’s Sharpe ratio is 0.790 post‑Covid, while GPT’s is –1.009. performance
    Source sentence
    TABLE X: Sharpe Ratios for Multi‑Variable Portfolios
  • The bond regression model achieves an out‑of‑sample R² of 0.9901, indicating very accurate price predictions for AGG. predictive-accuracy
    Source sentence
    Our best result chooses the corporate index, 1-year, 2-year, 5-year, 7-year, and 10-year bill returns with an out-of-sample R2 of 0.9901.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 25 related papers). Coverage is partial, so this is neither replication nor refutation.

Cited by 1 paper in the archive (1 by independent authors)

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (3)

  1. Stock trend prediction using news sentiment analysis (2016) — link
  2. Beige Book. Federal Reserve Board - Beige Book
  3. The Ghost in the Machine: Generating Beliefs with Large Language Models

0 of 3 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.