Paper: arXiv 2407.18103

Authors: Tian Guo, Emmanuel Hauptmann

Abstract

Large language models (LLMs) and their fine-tuning techniques have demonstrated superior performance in various language understanding and generation tasks. This paper explores fine-tuning LLMs for stock return forecasting with financial newsflow. In quantitative investing, return forecasting is fundamental for subsequent tasks like stock picking, portfolio optimization, etc. We formulate the model to include text representation and forecasting modules. We propose to compare the encoder-only and decoder-only LLMs, considering they generate text representations in distinct ways. The impact of these different representations on forecasting performance remains an open question. Meanwhile, we compare two simple methods of integrating LLMs’ token-level representations into the forecasting module. The experiments on real news and investment universes reveal that: (1) aggregated representations from LLMs’ token-level embeddings generally produce return predictions that enhance the performance of long-only and long-short portfolios; (2) in the relatively large investment universe, the decoder LLMs-based prediction model leads to stronger portfolios, whereas in the small universes, there are no consistent winners. Among the three LLMs studied (DeBERTa, Mistral, Llama), Mistral performs more robustly across different universes; (3) return predictions derived from LLMs’ text representations are a strong signal for portfolio construction, outperforming conventional sentiment scores.

Complexity vs Empirical Score

  • Math Complexity: 5.0/10
  • Empirical Rigor: 6.0/10 (capped from 8.0: transaction costs not modelled; see penalty flags)
  • Quadrant: Holy Grail — high math complexity, high empirical rigor (effective empirical rigor capped from 8.0)

Why this score: The paper exhibits moderate mathematical complexity with conceptual modeling of LLM architectures and forecasting modules, but the core methodology relies on applying existing transformer models. It demonstrates high empirical rigor through detailed backtesting on real news and investment universes, evaluating portfolio performance (long-only/long-short) with specific out-of-sample results and comparative metrics.

Research Flowchart

  flowchart TD
  A["Research Goal: Fine-tune LLMs for Stock Return Prediction<br>using Financial Newsflow"] --> B{"Data & Models"};
  B --> B1["Input: Financial Newsflow &<br>Investment Universe Data"];
  B --> B2["LLM Architectures:<br>Encoder vs. Decoder"];
  B --> B3["Forecasting Module:<br>LLM Representation Integration"];
  B1 & B2 & B3 --> C["Computational Process:<br>Fine-tuning LLMs for Return Forecasting"];
  C --> D{"Key Outcomes"};
  D --> D1["Aggregated LLM representations<br>enhance portfolio performance"];
  D --> D2["Mistral: Most robust across universes<br>Decoder LLMs excel in large universes"];
  D --> D3["LLM predictions outperform<br>conventional sentiment scores"];

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes an LLM-based return prediction model that includes text representation and forecasting modules.

Source sentence
We design an LLM-based return prediction model comprising the text representation and the forecasting modules.

Assumes 6

  • An investment universe of stocks is defined and used for stock‑picking. market
    Source sentence
    Assume an investment universe consisting of a set of stocks denoted by U = {s}S s=1, where s represents the stock index.
  • Return forecasts are n‑step forward returns of each stock. model
    Source sentence
    let rs,t+n ∈R be the n-step forward return of stock s w.r.t. timestep t.
  • Portfolios are formed by selecting stocks in the top (9th) and bottom (0th) deciles of predicted returns and are equally weighted. execution
    Source sentence
    The long-only portfolio is built by taking the stocks with the return predictions falling in the top (9th) decile of prediction rankings. The long-short portfolios take the stocks in the top (9th) and bottom (0th) deciles. The stocks in all portfolios are equally weighted.
  • Portfolios are rebalanced monthly. execution
    Source sentence
    We perform backtesting to evaluate the portfolios in monthly rebalancing.
Show 2 more
  • Aggregating token‑level embeddings (e.g., by averaging) provides a valid representation for forecasting. model
    Source sentence
    we present a simple alternative to bottleneck representation, i.e., allowing the forecasting module to aggregate the representations of all tokens. This can be done using various methods like averaging...
  • Appending an EOS token to sequences is beneficial for encoder‑only LLMs and is applied to both encoder‑only and decoder‑only models for fair comparison. model
    Source sentence
    In experiments, we observed that appending the EOS token is more helpful for encoder-only LLMs. For a comparison on the same ground, we append EOS tokens for both encoder-only and decoder-only LLMs...

Data

Backtest sample: 2015 – 2020 · North American stocks · equities · monthly

  • Company-level financial newsflow data · financial data vendor text-news commercial
    2003-2019
    Source sentence
    We use company-level financial newsflow data from 2003 to 2019 provided by a financial data vendor.
  • North American, European, Emerging market investment universes market-prices unknown
    NA, EU, EM markets
    Source sentence
    we have two investment universe datasets of the North American (NA), European (EU), and Emerging (EM) markets, which consist of dates, stock identifiers, and the true monthly forward returns of corresponding stocks and dates.

Costs

Not modeled A trading or execution result is reported without costs.

Finds 5

Reported performance (LLM-based return prediction with aggregated/bottleneck representations): Sharpe 1.49 (benchmark 0.68) · annual return 27.0%

  • Aggregated token‑level representations generally improve the performance of long‑only and long‑short portfolios. performance
    Source sentence
    aggregated representations from LLMs’ token-level embeddings generally produce the return predictions that enhance the performance of long-only and long-short portfolios
  • Decoder‑only LLMs yield stronger portfolios in relatively large investment universes, while no consistent winner appears in small universes. performance
    Source sentence
    in the relatively large investment universe, the decoder LLMs-based prediction model leads to stronger portfolios, whereas in the small universes, there are no consistent winners
  • Mistral shows robust performance across different universes compared with DeBERTa and Llama. performance
    Source sentence
    Among the three LLMs studied (DeBERTa, Mistral, Llama), Mistral performs more robustly across different universes
  • Return predictions from LLM text representations outperform conventional sentiment scores for portfolio construction. performance
    Source sentence
    return predictions derived from LLMs’ text representations are a strong signal for portfolio construction, outperforming conventional sentiment scores
  • Aggregated representations produce higher decile returns even when decile RMSE is higher, indicating return predictions can be useful despite lower predictive accuracy. statistical
    Source sentence
    ... the aggregated representation model has a higher decile return, but a higher RMSE, in the 9th decile

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 38 related papers). Coverage is partial, so this is neither replication nor refutation.

Cited by 6 papers in the archive (5 by independent authors): Predicting Liquidity-Aware Bond Yields using Causal GANs and … (discusses its findings)

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (80)

  1. Multifactor explanations of asset pricing anomalies (1996)
  2. Foundations of factor investing (2013)
  3. Asset management: A systematic approach to factor investing (2014)
  4. Daily market news sentiment and stock prices (2019)
  5. Measuring news sentiment (2022)
  6. A hierarchical multi-task approach for learning embeddings from semantic tasks (2019)
  7. Wider and deeper LLM networks are fairer LLM evaluators (2023) — link
  8. Spae: Semantic pyramid autoencoder for multimodal generation with frozen llms (2024)
and 72 more
  1. Novel volatility forecasting using deep learning–Long Short Term Memory Recurrent Neural Networks (2019)
  2. Hierarchical complementary attention network for predicting stock price movements with news (2018)
  3. Textual analysis of stock market prediction using breaking financial news: The AZFin text system (2009)
  4. Quantifying the relationship between financial news and the stock market (2013)
  5. EAN: Event Attention Network for Stock Price Trend Prediction based on Sentimental Embedding (2019)
  6. Predicting short-term stock prices using ensemble methods and online data sources (2018)
  7. Listening to chaotic whispers: A deep learning framework for news-oriented stock trend prediction (2018)
  8. Stock movement prediction from tweets and historical prices (2018)
  9. What You Say and How You Say It Matters: Predicting Financial Risk Using Verbal and Vocal Cues (2019)
  10. Hierarchical Multi-Scale Gaussian Transformer for Stock Movement Prediction. (2020)
  11. Deep attentive learning for stock movement prediction from social media text and company correlations (2020)
  12. Factorvae: A probabilistic dynamic factor model based on variational autoencoder for predicting cross-sectional stock returns (2022)
  13. Efficient integration of multi-order dynamics and internal dynamics in stock movement prediction (2023)
  14. Ask" who", not" what": Bitcoin volatility forecasting with twitter data (2023)
  15. RWKV: Reinventing RNNs for the Transformer Era (2023) — link
  16. Leveraging BERT to Improve the FEARS Index for Stock Forecasting (2019)
  17. HTML: Hierarchical Transformer-based Multi-task Learning for Volatility Prediction (2020)
  18. Group, Extract and Aggregate: Summarizing a Large Amount of Finance News for Forex Movement Prediction (2019) — link
  19. A survey of large language models (2023) — link
  20. Large language models in finance: A survey — → 2311.10723
  21. Attention is all you need (2017)
  22. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019)
  23. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019)
  24. Deberta: Decoding-enhanced bert with disentangled attention (2020) — link
  25. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing (2021) — link
  26. Text embeddings by weakly-supervised contrastive pre-training (2022) — link
  27. Improving text embeddings with large language models (2023) — link
  28. Multilingual e5 text embeddings: A technical report (2024) — link
  29. Llm2vec: Large language models are secretly powerful text encoders (2024) — link
  30. NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (2024) — link
  31. Improving Language Understanding by Generative Pre-Training (2018)
  32. Llama 2: Open foundation and fine-tuned chat models (2023) — link
  33. Mistral 7B (2023) — link
  34. Transformers need glasses! Information over-squashing in language tasks (2024) — link
  35. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension (2019) — link
  36. Exploring the limits of transfer learning with a unified text-to-text transformer (2020)
  37. Sentence Bottleneck Autoencoders from Transformer Language Models (2021)
  38. SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval (2023)
  39. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters (2020)
  40. Supervised contrastive learning for pre-trained language model fine-tuning (2020) — link
  41. Lora: Low-rank adaptation of large language models (2021) — link
  42. Finetuned language models are zero-shot learners (2021) — link
  43. Parameter-efficient fine-tuning of large-scale pre-trained language models (2023)
  44. Dora: Weight-decomposed low-rank adaptation (2024) — link
  45. Scaling instruction-finetuned language models (2024)
  46. Learning to generate explainable stock predictions using self-reflective large language models — → 2402.03659
  47. Ploutos: Towards interpretable stock movement prediction with financial large language model — → 2403.00782
  48. Language models are unsupervised multitask learners (2019)
  49. Language models are few-shot learners (2020)
  50. Large language models are zero-shot reasoners (2022)
  51. Chain-of-thought prompting elicits reasoning in large language models (2022)
  52. Harnessing LLMs for temporal data-a study on explainable financial time series forecasting (2023)
  53. LLMFactor: Extracting Profitable Factors through Prompts for Explainable Stock Movement Prediction (2024) — link
  54. Financial statement analysis with large language models — → 2407.17866
  55. Vader: A parsimonious rule-based model for sentiment analysis of social media text (2014)
  56. FinVADER: Financial Sentiment Analysis (2023) — link
  57. FinBERT: Financial Sentiment Analysis with Pre-trained Language Models (2019) — link
  58. Finbert: A pretrained language model for financial communications (2020) — link
  59. Finbert: A pre-trained financial language representation model for financial text mining (2021)
  60. FinBERT-FOMC: Fine-Tuned FinBERT Model with sentiment focus method for enhancing sentiment analysis of FOMC minutes (2023)
  61. Hybrid neural networks for learning the trend in time series (2017)
  62. Bitcoin volatility forecasting with a glimpse into buy and sell orders (2018)
  63. Sensing Social Media Signals for Cryptocurrency News (2019)
  64. Esg2risk: A deep learning framework from esg news to stock volatility prediction (2020) — link
  65. Simple and scalable predictive uncertainty estimation using deep ensembles (2017)
  66. A simple baseline for Bayesian uncertainty in deep learning (2019)
  67. Preconditioned stochastic gradient Langevin dynamics for deep neural networks (2016)
  68. Exploring interpretable LSTM neural networks over multi-variable data (2019)
  69. The benefits of socially responsible investing: An active manager’s perspective (2015)
  70. Can ESG add alpha? An analysis of ESG tilt and momentum strategies (2016)
  71. Impact of ESG factors on firm risk in Europe (2016)
  72. Every little helps? ESG news and stock market reaction (2019)

4 of 80 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.