{"paper_id":"2411.16569","title":"Predictive Power of LLMs in Financial Markets","authors":["Jerick Shi","Burton Hollifield"],"paper_date":"2024-11-25","paper_type":"empirical","paper_url":"https://arxiv.org/pdf/2411.16569v1","page_url":"https://thequant.space/flowcharts/predictive-power-of-llms-in-financial-markets/","scores":{"empirical_rigor":6,"math_complexity":3,"quadrant":"Street Traders"},"proposes":{"status":"available","statement":"The paper proposes to evaluate look-ahead bias in GPT models and compare GPT versus BERT using Beige Book data for portfolio construction.","quote":"From this report, we make 3 main contributions. First, we show that regardless of how well the prompt is, there is always an unavoidable look-ahead bias when using the GPT model.","summary":"Predicting the movement of the stock market and other assets has been valuable over the past few decades. Knowing how the value of a certain sector market may move in the future provides much information for investors, as they use that information to develop strategies to maximize profit or…","strategy":"BERT-based portfolio allocation using Beige Book correlations"},"assumptions":{"status":"available","items":[{"assumption":"The GPT-3.5 model is trained only on data up to September 2021.","type":"model","stated":"explicit","quote":"since the GPT-3.5 model is ’trained up to September 2021’, we shall use 1980-01 to 2021-09 as the training set"},{"assumption":"Historical stock and bond price series are accurate and can be used for linear regression without adjustment.","type":"data","stated":"implied","quote":"We obtain historical stock prices directly from Yahoo Finance, where we take the prices from ˆGSPC from the first day of 1985 to the last day of 2023."},{"assumption":"Rolling averages of volatility and correlations are sufficient inputs for the portfolio optimization.","type":"execution","stated":"implied","quote":"σS ←rollingAverage(month, Stock) σB ←rollingAverage(month, Bond) ρmonth ←askGPT(beige data[month], ρ−3::−1)"},{"assumption":"The out‑of‑sample R² of 0.9901 indicates the bond regression model predicts AGG prices accurately.","type":"statistical","stated":"implied","quote":"Our best result chooses the corporate index, 1-year, 2-year, 5-year, 7-year, and 10-year bill returns with an out-of-sample R2 of 0.9901."}]},"datasets":{"status":"available","items":[{"name":"ˆGSPC","provider":"Yahoo Finance","kind":"market-prices","access":"public","period":"1985-01-01 to 2023-12-31","universe":"S&P 500 index","quote":"We obtain historical stock prices directly from Yahoo Finance, where we take the prices from ˆGSPC from the first day of 1985 to the last day of 2023.","known_id":"yahoo-finance"},{"name":"AGG","provider":"Yahoo Finance","kind":"market-prices","access":"public","period":"mid‑2003 to 2023-12-31","universe":"Aggregate bond ETF","quote":"The bond data comes from the same source under the ticker AGG, but the data only span from the middle of 2003 to the end of 2023.","known_id":"yahoo-finance"},{"name":"FRED bill returns & corporate index","provider":"Federal Reserve Economic Data (FRED)","kind":"macro","access":"public","period":"1980 onward","universe":"U.S. Treasury bills (1‑,2‑,5‑,7‑,10‑,20‑,30‑year) and ICE BofA US Corporate Index","quote":"we were able to get 1-year, 2-year, 5-year, 7-year, 10-year, 20-year, 30-year bill returns as well as the ICE BofA US Corporate Index Total Return Index Value. These data go back to 1980.","known_id":"fred"},{"name":"Beige Book articles","provider":"Federal Reserve Bank of Minneapolis","kind":"text-news","access":"public","period":null,"universe":"5 months per year, 12 states each month","quote":"Finally, the beige book is publicly available on the Federal Reserve Bank of Minneapolis. Each year, 5 months per year, and 12 states for each month, an article is published."}],"known":[],"sample":{"start":null,"end":null,"market":"US stocks and bonds","asset_class":"fixed-income","frequency":"monthly"},"data_resolution":"daily"},"transaction_costs":{"status":"available","modeled":"no","headline_net_of_costs":false,"detail":null,"quote":null,"source":"front_matter","rule":"R1","candidates":{"front_matter":"no","evidence_card":"no","evidence_card_quoted":false},"meaning":"a trading or execution result is reported without costs"},"results":{"status":"available","evaluation_type":"trading-pnl","findings":[{"finding":"The GPT model exhibits unavoidable look‑ahead bias, as evidenced by higher out‑of‑sample RMSE compared to training RMSE.","kind":"negative","quote":"If the error has a significant increase out of sample, that indicates that there is evidence of look-ahead bias."},{"finding":"BERT consistently outperforms GPT in portfolio simulations, delivering higher Sharpe ratios both pre‑ and post‑Covid.","kind":"performance","quote":"We notice that in both periods, the BERT model tends to perform the best, whereas the GPT model sometimes even performs worse than simple rolling covariances."},{"finding":"For the 2‑Variable portfolio, BERT achieves a Sharpe ratio of 1.9208 pre‑Covid versus 0.445 for GPT.","kind":"performance","quote":"Period Baseline BERT GPT Pre‑Covid 0.423 1.9208 0.445"},{"finding":"For the Multi‑Variable portfolio, BERT’s Sharpe ratio is 0.790 post‑Covid, while GPT’s is –1.009.","kind":"performance","quote":"TABLE X: Sharpe Ratios for Multi‑Variable Portfolios"},{"finding":"The bond regression model achieves an out‑of‑sample R² of 0.9901, indicating very accurate price predictions for AGG.","kind":"predictive-accuracy","quote":"Our best result chooses the corporate index, 1-year, 2-year, 5-year, 7-year, and 10-year bill returns with an out-of-sample R2 of 0.9901."}],"reported_performance":{"strategy":"BERT-based portfolio allocation using Beige Book correlations","sharpe":null,"sharpe_benchmark":null,"annual_return_pct":null,"total_return_pct":null,"max_drawdown_pct":null},"out_of_sample":{"method":"holdout","headline_out_of_sample":true}},"independent_evidence":{"status":"none_found","supports":[],"contradicts":[],"qualifies":[],"compared_with":25,"citations":{"cited_by_in_corpus":1,"by_independent_authors":1,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":0,"mentions":1,"unclassified":0},"independent_papers":[],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":false,"code_url":null,"note":"No repository presented by the authors as their own was found in the paper. Absence here is not proof that no code exists."},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":1,"extracted_at":"2026-11-14T17:52:38Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}