Paper: arXiv 2412.19245
Authors: Kemal Kirtac, Guido Germano
Abstract
We investigate the efficacy of large language models (LLMs) in sentiment analysis of U.S. financial news and their potential in predicting stock market returns. We analyze a dataset comprising 965,375 news articles that span from January 1, 2010, to June 30, 2023; we focus on the performance of various LLMs, including BERT, OPT, FINBERT, and the traditional Loughran-McDonald dictionary model, which has been a dominant methodology in the finance literature. The study documents a significant association between LLM scores and subsequent daily stock returns. Specifically, OPT, which is a GPT-3 based LLM, shows the highest accuracy in sentiment prediction with an accuracy of 74.4%, slightly ahead of BERT (72.5%) and FINBERT (72.2%). In contrast, the Loughran-McDonald dictionary model demonstrates considerably lower effectiveness with only 50.1% accuracy. Regression analyses highlight a robust positive impact of OPT model scores on next-day stock returns, with coefficients of 0.274 and 0.254 in different model specifications. BERT and FINBERT also exhibit predictive relevance, though to a lesser extent. Notably, we do not observe a significant relationship between the Loughran-McDonald dictionary model scores and stock returns, challenging the efficacy of this traditional method in the current financial context. In portfolio performance, the long-short OPT strategy excels with a Sharpe ratio of 3.05, compared to 2.11 for BERT and 2.07 for FINBERT long-short strategies. Strategies based on the Loughran-McDonald dictionary yield the lowest Sharpe ratio of 1.23. Our findings emphasize the superior performance of advanced LLMs, especially OPT, in financial market prediction and portfolio management, marking a significant shift in the landscape of financial analysis tools with implications to financial regulation and policy analysis.
Complexity vs Empirical Score
- Math Complexity: 6.5/10
- Empirical Rigor: 6.0/10 (capped from 8.0: reported Sharpe ratio of 3.05 is implausibly high for daily or lower-frequency data; see penalty flags)
- Quadrant: Holy Grail — high math complexity, high empirical rigor (effective empirical rigor capped from 8.0)
Why this score: The paper employs sophisticated NLP and econometric models (BERT, OPT, linear regressions with fixed effects) and advanced statistical techniques, indicating high mathematical complexity. It demonstrates strong empirical rigor by using a large dataset of nearly a million news articles, reporting precise metrics like accuracy (74.4%), Sharpe ratios (3.05), and transaction costs, and outlining a backtest-ready trading strategy.
Research Flowchart
flowchart TD A["Research Goal<br>Test LLM efficacy in<br>financial sentiment analysis"] --> B["Data Input<br>965,375 news articles<br>2010-2023"] B --> C["Methodology<br>Compare LLMs: BERT, OPT, FINBERT<br>vs Traditional Dictionary"] C --> D["Computational Process<br>Sentiment Scoring &<br>Regression Analysis"] D --> E["Key Outcomes"] E --> F["Optimal Model<br>OPT (GPT-3 based)<br>74.4% accuracy"] E --> G["Portfolio Performance<br>OPT Long-Short Strategy<br>Sharpe Ratio: 3.05"] E --> H["Traditional Model Failure<br>LM Dictionary<br>50.1% accuracy, No Returns Link"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes using large language models for sentiment analysis of U.S. financial news and building long‑short trading strategies based on the derived sentiment scores.
Source sentence
We analyse the performance of the large language models (LLMs) OPT, BERT, and FinBERT, alongside the traditional Loughran-McDonald dictionary, in the sentiment analysis of 965,375 U.S. financial news articles from 2010 to 2023.
Assumes 5
- Trading incurs a flat transaction cost of 10 basis points per trade. execution
Source sentence
To make our simulation more aligned with actual trading conditions, we included a transaction cost of 10 basis points for each trade, accounting for the typical costs traders would encounter in the market.
- Portfolios are rebalanced daily using market‑value weighting and are self‑financing long‑short strategies. execution
Source sentence
The long‑short portfolios are self‑financing strategies that simultaneously involve taking long positions in stocks with the highest 20% sentiment scores and short positions in stocks with the lowest 20% sentiment scores.
- News articles are filtered for uniqueness using a cosine similarity threshold of 0.8 within the prior five business days. data
Source sentence
We avoid redundancy by using a novelty score based on the similarity between articles: if a new article is too similar to an older article published within the past 20 days according to a cosine similarity score of 0.8 or more, we exclude it.
- Only news linked to individual stocks with available three‑day returns are included. data
Source sentence
We only consider news articles related to individual stocks with available three‑day returns.
Show 1 more
- Trading decisions are timed to news releases: pre‑6 am news trades at market open, 6 am‑4 pm news trades same‑day close, post‑4 pm news trades next day open. execution
Source sentence
For news appearing between 6 am and 4 pm, we initiate a trade with closing prices of the same day and exit the trade the next trading day. Any news coming in after 4 pm was used for trades at the start of the next trading day.
Data
2021-08 – 2023-07 · U.S. stocks · equities · daily
- CRSP daily stock file market-prices commercial
Source sentence
In our research, we primarily use two datasets: one from the Center for Research in Security Prices (CRSP) that includes daily stock returns, and another from Refinitiv with global news.
- Refinitiv news dataset text-news commercial
Source sentence
The news data from Refinitiv comprises detailed articles and quick alerts, focusing on companies based in the U.S.
- Filtered news articles text-news commercial
Source sentence
After applying our filters, we were left with 965,375 articles. Our sample dataset is summarized in Table 1.
Refinitiv / LSEG (Datastream, Eikon, Tick History) CRSP Financial PhraseBank
Costs
Modeled
10 basis points per trade, applied to each side of the transaction.
Source sentence
To make our simulation more aligned with actual trading conditions, we included a transaction cost of 10 basis points for each trade, accounting for the typical costs traders would encounter in the market.
Finds 5
Sharpe 3.05 (benchmark 1.23) · total return 355.0% · max drawdown 18.6%
- OPT achieves the highest sentiment prediction accuracy of 74.4%, outperforming BERT (72.5%) and FinBERT (72.2%). predictive-accuracy
Source sentence
Specifically, OPT, which is a GPT‑3 based LLM, shows the highest accuracy in sentiment prediction with an accuracy of 74.4%, slightly ahead of BERT (72.5%) and FINBERT (72.2%).
- The long‑short OPT strategy yields a Sharpe ratio of 3.05, far above BERT (2.11), FinBERT (2.07) and the Loughran‑McDonald dictionary (1.23). performance
Source sentence
In portfolio performance, the long‑short OPT strategy excels with a Sharpe ratio of 3.05, compared to 2.11 for BERT and 2.07 for FINBERT long‑short strategies. Strategies based on the Loughran‑McDonald dictionary yield the lowest Sharpe ratio of 1.23.
- From August 2021 to July 2023 the OPT long‑short strategy generated a 355% cumulative gain. performance
Source sentence
From August 2021 to July 2023, this strategy produces an impressive 355% gain, outperforming other strategies and traditional market portfolios.
- Regression coefficients for OPT sentiment scores are positive and significant (0.274 and 0.254) in predicting next‑day returns. statistical
Source sentence
Regression analyses highlight a robust positive impact of OPT model scores on next‑day stock returns, with coefficients of 0.274 and 0.254 in different model specifications.
- The Loughran‑McDonald dictionary model shows low predictive power (50.1% accuracy) and no significant relationship with stock returns. negative
Source sentence
In contrast, the Loughran‑McDonald dictionary model demonstrates considerably lower effectiveness with only 50.1% accuracy
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 45 related papers). Coverage is partial, so this is neither replication nor refutation.
12 papers in the archive : Market-Dependent Communication in Multi-Agent Alpha …
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.