Paper: arXiv 2407.18103
Authors: Tian Guo, Emmanuel Hauptmann
Abstract
Large language models (LLMs) and their fine-tuning techniques have demonstrated superior performance in various language understanding and generation tasks. This paper explores fine-tuning LLMs for stock return forecasting with financial newsflow. In quantitative investing, return forecasting is fundamental for subsequent tasks like stock picking, portfolio optimization, etc. We formulate the model to include text representation and forecasting modules. We propose to compare the encoder-only and decoder-only LLMs, considering they generate text representations in distinct ways. The impact of these different representations on forecasting performance remains an open question. Meanwhile, we compare two simple methods of integrating LLMs’ token-level representations into the forecasting module. The experiments on real news and investment universes reveal that: (1) aggregated representations from LLMs’ token-level embeddings generally produce return predictions that enhance the performance of long-only and long-short portfolios; (2) in the relatively large investment universe, the decoder LLMs-based prediction model leads to stronger portfolios, whereas in the small universes, there are no consistent winners. Among the three LLMs studied (DeBERTa, Mistral, Llama), Mistral performs more robustly across different universes; (3) return predictions derived from LLMs’ text representations are a strong signal for portfolio construction, outperforming conventional sentiment scores.
Complexity vs Empirical Score
- Math Complexity: 5.0/10
- Empirical Rigor: 6.0/10 (capped from 8.0: transaction costs not modelled; see penalty flags)
- Quadrant: Holy Grail — high math complexity, high empirical rigor (effective empirical rigor capped from 8.0)
Why this score: The paper exhibits moderate mathematical complexity with conceptual modeling of LLM architectures and forecasting modules, but the core methodology relies on applying existing transformer models. It demonstrates high empirical rigor through detailed backtesting on real news and investment universes, evaluating portfolio performance (long-only/long-short) with specific out-of-sample results and comparative metrics.
Research Flowchart
flowchart TD
A["Research Goal: Fine-tune LLMs for Stock Return Prediction<br>using Financial Newsflow"] --> B{"Data & Models"};
B --> B1["Input: Financial Newsflow &<br>Investment Universe Data"];
B --> B2["LLM Architectures:<br>Encoder vs. Decoder"];
B --> B3["Forecasting Module:<br>LLM Representation Integration"];
B1 & B2 & B3 --> C["Computational Process:<br>Fine-tuning LLMs for Return Forecasting"];
C --> D{"Key Outcomes"};
D --> D1["Aggregated LLM representations<br>enhance portfolio performance"];
D --> D2["Mistral: Most robust across universes<br>Decoder LLMs excel in large universes"];
D --> D3["LLM predictions outperform<br>conventional sentiment scores"];
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes an LLM-based return prediction model that includes text representation and forecasting modules.
Source sentence
We design an LLM-based return prediction model comprising the text representation and the forecasting modules.
Assumes 6
- An investment universe of stocks is defined and used for stock‑picking. market
Source sentence
Assume an investment universe consisting of a set of stocks denoted by U = {s}S s=1, where s represents the stock index.
- Return forecasts are n‑step forward returns of each stock. model
Source sentence
let rs,t+n ∈R be the n-step forward return of stock s w.r.t. timestep t.
- Portfolios are formed by selecting stocks in the top (9th) and bottom (0th) deciles of predicted returns and are equally weighted. execution
Source sentence
The long-only portfolio is built by taking the stocks with the return predictions falling in the top (9th) decile of prediction rankings. The long-short portfolios take the stocks in the top (9th) and bottom (0th) deciles. The stocks in all portfolios are equally weighted.
- Portfolios are rebalanced monthly. execution
Source sentence
We perform backtesting to evaluate the portfolios in monthly rebalancing.
Show 2 more
- Aggregating token‑level embeddings (e.g., by averaging) provides a valid representation for forecasting. model
Source sentence
we present a simple alternative to bottleneck representation, i.e., allowing the forecasting module to aggregate the representations of all tokens. This can be done using various methods like averaging...
- Appending an EOS token to sequences is beneficial for encoder‑only LLMs and is applied to both encoder‑only and decoder‑only models for fair comparison. model
Source sentence
In experiments, we observed that appending the EOS token is more helpful for encoder-only LLMs. For a comparison on the same ground, we append EOS tokens for both encoder-only and decoder-only LLMs...
Data
2015 – 2020 · North American stocks · equities · monthly
- Company-level financial newsflow data text-news commercial
Source sentence
We use company-level financial newsflow data from 2003 to 2019 provided by a financial data vendor.
- North American, European, Emerging market investment universes market-prices unknown
Source sentence
we have two investment universe datasets of the North American (NA), European (EU), and Emerging (EM) markets, which consist of dates, stock identifiers, and the true monthly forward returns of corresponding stocks and dates.
Costs
Not modeled
Finds 5
Sharpe 1.49 (benchmark 0.68) · annual return 27.0%
- Aggregated token‑level representations generally improve the performance of long‑only and long‑short portfolios. performance
Source sentence
aggregated representations from LLMs’ token-level embeddings generally produce the return predictions that enhance the performance of long-only and long-short portfolios
- Decoder‑only LLMs yield stronger portfolios in relatively large investment universes, while no consistent winner appears in small universes. performance
Source sentence
in the relatively large investment universe, the decoder LLMs-based prediction model leads to stronger portfolios, whereas in the small universes, there are no consistent winners
- Mistral shows robust performance across different universes compared with DeBERTa and Llama. performance
Source sentence
Among the three LLMs studied (DeBERTa, Mistral, Llama), Mistral performs more robustly across different universes
- Return predictions from LLM text representations outperform conventional sentiment scores for portfolio construction. performance
Source sentence
return predictions derived from LLMs’ text representations are a strong signal for portfolio construction, outperforming conventional sentiment scores
- Aggregated representations produce higher decile returns even when decile RMSE is higher, indicating return predictions can be useful despite lower predictive accuracy. statistical
Source sentence
... the aggregated representation model has a higher decile return, but a higher RMSE, in the 9th decile
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 38 related papers). Coverage is partial, so this is neither replication nor refutation.
6 papers in the archive : Predicting Liquidity-Aware Bond Yields using Causal GANs and …
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (80)
- Multifactor explanations of asset pricing anomalies (1996)
- Foundations of factor investing (2013)
- Asset management: A systematic approach to factor investing (2014)
- Daily market news sentiment and stock prices (2019)
- Measuring news sentiment (2022)
- A hierarchical multi-task approach for learning embeddings from semantic tasks (2019)
- Wider and deeper LLM networks are fairer LLM evaluators (2023) — link
- Spae: Semantic pyramid autoencoder for multimodal generation with frozen llms (2024)
and 72 more
- Novel volatility forecasting using deep learning–Long Short Term Memory Recurrent Neural Networks (2019)
- Hierarchical complementary attention network for predicting stock price movements with news (2018)
- Textual analysis of stock market prediction using breaking financial news: The AZFin text system (2009)
- Quantifying the relationship between financial news and the stock market (2013)
- EAN: Event Attention Network for Stock Price Trend Prediction based on Sentimental Embedding (2019)
- Predicting short-term stock prices using ensemble methods and online data sources (2018)
- Listening to chaotic whispers: A deep learning framework for news-oriented stock trend prediction (2018)
- Stock movement prediction from tweets and historical prices (2018)
- What You Say and How You Say It Matters: Predicting Financial Risk Using Verbal and Vocal Cues (2019)
- Hierarchical Multi-Scale Gaussian Transformer for Stock Movement Prediction. (2020)
- Deep attentive learning for stock movement prediction from social media text and company correlations (2020)
- Factorvae: A probabilistic dynamic factor model based on variational autoencoder for predicting cross-sectional stock returns (2022)
- Efficient integration of multi-order dynamics and internal dynamics in stock movement prediction (2023)
- Ask" who", not" what": Bitcoin volatility forecasting with twitter data (2023)
- RWKV: Reinventing RNNs for the Transformer Era (2023) — link
- Leveraging BERT to Improve the FEARS Index for Stock Forecasting (2019)
- HTML: Hierarchical Transformer-based Multi-task Learning for Volatility Prediction (2020)
- Group, Extract and Aggregate: Summarizing a Large Amount of Finance News for Forex Movement Prediction (2019) — link
- A survey of large language models (2023) — link
- Large language models in finance: A survey — → 2311.10723
- Attention is all you need (2017)
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019)
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019)
- Deberta: Decoding-enhanced bert with disentangled attention (2020) — link
- Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing (2021) — link
- Text embeddings by weakly-supervised contrastive pre-training (2022) — link
- Improving text embeddings with large language models (2023) — link
- Multilingual e5 text embeddings: A technical report (2024) — link
- Llm2vec: Large language models are secretly powerful text encoders (2024) — link
- NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (2024) — link
- Improving Language Understanding by Generative Pre-Training (2018)
- Llama 2: Open foundation and fine-tuned chat models (2023) — link
- Mistral 7B (2023) — link
- Transformers need glasses! Information over-squashing in language tasks (2024) — link
- Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension (2019) — link
- Exploring the limits of transfer learning with a unified text-to-text transformer (2020)
- Sentence Bottleneck Autoencoders from Transformer Language Models (2021)
- SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval (2023)
- Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters (2020)
- Supervised contrastive learning for pre-trained language model fine-tuning (2020) — link
- Lora: Low-rank adaptation of large language models (2021) — link
- Finetuned language models are zero-shot learners (2021) — link
- Parameter-efficient fine-tuning of large-scale pre-trained language models (2023)
- Dora: Weight-decomposed low-rank adaptation (2024) — link
- Scaling instruction-finetuned language models (2024)
- Learning to generate explainable stock predictions using self-reflective large language models — → 2402.03659
- Ploutos: Towards interpretable stock movement prediction with financial large language model — → 2403.00782
- Language models are unsupervised multitask learners (2019)
- Language models are few-shot learners (2020)
- Large language models are zero-shot reasoners (2022)
- Chain-of-thought prompting elicits reasoning in large language models (2022)
- Harnessing LLMs for temporal data-a study on explainable financial time series forecasting (2023)
- LLMFactor: Extracting Profitable Factors through Prompts for Explainable Stock Movement Prediction (2024) — link
- Financial statement analysis with large language models — → 2407.17866
- Vader: A parsimonious rule-based model for sentiment analysis of social media text (2014)
- FinVADER: Financial Sentiment Analysis (2023) — link
- FinBERT: Financial Sentiment Analysis with Pre-trained Language Models (2019) — link
- Finbert: A pretrained language model for financial communications (2020) — link
- Finbert: A pre-trained financial language representation model for financial text mining (2021)
- FinBERT-FOMC: Fine-Tuned FinBERT Model with sentiment focus method for enhancing sentiment analysis of FOMC minutes (2023)
- Hybrid neural networks for learning the trend in time series (2017)
- Bitcoin volatility forecasting with a glimpse into buy and sell orders (2018)
- Sensing Social Media Signals for Cryptocurrency News (2019)
- Esg2risk: A deep learning framework from esg news to stock volatility prediction (2020) — link
- Simple and scalable predictive uncertainty estimation using deep ensembles (2017)
- A simple baseline for Bayesian uncertainty in deep learning (2019)
- Preconditioned stochastic gradient Langevin dynamics for deep neural networks (2016)
- Exploring interpretable LSTM neural networks over multi-variable data (2019)
- The benefits of socially responsible investing: An active manager’s perspective (2015)
- Can ESG add alpha? An analysis of ESG tilt and momentum strategies (2016)
- Impact of ESG factors on firm risk in Europe (2016)
- Every little helps? ESG news and stock market reaction (2019)
4 of 80 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.