Paper: arXiv 2602.00086

Authors: Walid Siala, Ahmed Khanfir, Mike Papadakis

Abstract

This paper addresses stock price movement prediction by leveraging LLM-based news sentiment analysis. Earlier works have largely focused on proposing and assessing sentiment analysis models and stock movement prediction methods, however, separately. Although promising results have been achieved, a clear and in-depth understanding of the benefit of the news sentiment to this task, as well as a comprehensive assessment of different architecture types in this context, is still lacking. Herein, we conduct an evaluation study that compares 3 different LLMs, namely, DeBERTa, RoBERTa and FinBERT, for sentiment-driven stock prediction. Our results suggest that DeBERTa outperforms the other two models with an accuracy of 75% and that an ensemble model that combines the three models can increase the accuracy to about 80%. Also, we see that sentiment news features can benefit (slightly) some stock market prediction models, i.e., LSTM-, PatchTST- and tPatchGNN-based classifiers and PatchTST- and TimesNet-based regression tasks models.

Complexity vs Empirical Score

  • Math Complexity: 2.5/10
  • Empirical Rigor: 6.5/10
  • Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: The paper presents an empirical evaluation comparing LLM sentiment outputs and ensemble methods combined with advanced time-series architectures for stock prediction. It lacks mathematical depth, but offers solid empirical execution with open-source code, ablation studies, and clear comparisons across multiple model families.

Research Flowchart

  flowchart TD
    A[Goal: Evaluate LLM News Sentiment for Stock Prediction] --> B[Input: Financial News & Stock Data]
    B --> C[Sentiment Analysis via 3 LLMs<br/>DeBERTa, RoBERTa, FinBERT]
    C --> D[Predictive Modeling<br/>Classification & Regression Models]
    D --> E[Ensemble Integration<br/>Combine 3 LLM Sentiments]
    E --> F[Outcomes:<br/>1. DeBERTa best single model 75% acc<br/>2. Ensemble boosts acc to ~80%<br/>3. Sentiment slightly improves LSTM, PatchTST, etc.]

Cited References (64)

  1. A Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong (2000) — link
  2. Large-Scale Manual Validation of Bug Fixing Commits: A Fine-grained Analysis of Tangling (2020) — link
  3. The SmartSHARK Repository Mining Data (2021) — link
  4. GitHub Copilot
  5. Amazon CodeWhisperer
  6. PyTorch
  7. Evaluating large language models trained on code.(2021) (2021) — link
  8. CodeBERT: A Pre-Trained Model for Programming and Natural Languages (2020) — link
and 56 more
  1. Attention is all you need (2017)
  2. Bert: Pre-training of deep bidirectional transformers for language understanding (2018) — link
  3. Natural Software Revisited (2019) — link
  4. On the “Naturalness” of Buggy Code (2016) — link
  5. On the Impact of Refactoring Operations on Code Naturalness (2019) — link
  6. Toward Refactoring Evaluation with Code Naturalness (2018) — link
  7. How Do Contributors Impact Code Naturalness? An Exploratory Study of 50 Python Projects (2019) — link
  8. Improving Machine Translation Systems via Isotopic Replacement (2022)
  9. Are Mutants Really Natural? A Study on How “Naturalness” Helps Mutant Selection (2018) — link
  10. TUNA: TUning Naturalness-Based Analysis (2018) — link
  11. On the impact of tokenizer and parameters on n-gram based code analysis (2018)
  12. The Importance of Accounting for Real-World Labelling When Predicting Software Vulnerabilities (2019) — link
  13. On the Naturalness of Software (2012)
  14. Prediction and entropy of printed English (1951) — link
  15. A mathematical theory of communication (1948) — link
  16. An empirical study of smoothing techniques for language modeling (1999) — link
  17. Improved backing-off for M-gram language modeling (1995) — link
  18. Mining source code repositories at massive scale using language modeling (2013)
  19. Survey on software defect prediction techniques (2020) — link
  20. A Survey of Machine Learning for Big Code and Naturalness (2017) — link
  21. Bugram: Bug Detection with n-Gram Language Models (2016) — link
  22. Code smell detection by deep direct-learning and transfer-learning (2021) — link
  23. Reflections on: A Simpler Model of Software Readability (2021) — link
  24. Will They Like This? Evaluating Code Contributions with Language Models (2015) — link
  25. PMD applied (2005)
  26. Finding more null pointer bugs, but not too many (2007)
  27. The Common Weakness Enumeration (CWE) Initiative (2022)
  28. JavaParser
  29. Selecting fault revealing mutants (2020) — link
  30. Classification and evaluation of defects in a project retrospective (2002) — link
  31. Language Models Can Prioritize Patches for Practical Program Patching (2022) — link
  32. Predictive Mutation Analysis via Natural Language Channel in Source Code (2021) — link
  33. Giving content to investor sentiment: The role of media in the stock market (2007)
  34. Text mining for market prediction: A systematic review (2014)
  35. Investor sentiment in the stock market (2007)
  36. Irrational Exuberance (2017)
  37. When is a liability not a liability? Textual analysis, dictionaries, and 10-Ks (2011)
  38. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019)
  39. RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019) — link
  40. DeBERTa: Decoding-enhanced BERT with Disentangled Attention (2021)
  41. Domain-specific pretraining for financial sentiment analysis (2023)
  42. PatchTST: A Patch-based Time Series Transformer (2023)
  43. tPatchGNN: Temporal Graph Neural Networks with Patch-based Learning for Multivariate Time-series Forecasting (2023)
  44. TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis (2023)
  45. Long short-term memory (1997)
  46. Deep Learning for Event-Driven Stock Prediction (2015)
  47. Sentiment analysis on social media for stock movement prediction (2015)
  48. Time Series Analysis: Forecasting and Control (2015)
  49. SEntFiN 1.0: Entity-aware sentiment analysis for financial news (2022) — link
  50. Good debt or bad debt: Detecting semantic orientations in economic texts (2014)
  51. TweetEval:Unified Benchmark and Comparative Evaluation for Tweet Classification (2020)
  52. FinBERT: Financial Sentiment Analysis with Pre-trained Language Models (2019) — link
  53. DeBERTa-v3-base-finetuned-finance-text-classification (2023)
  54. TimeLMs: Diachronic Language Models from Twitter (2022) — link
  55. Fine-grained argument understanding with bert ensemble techniques: A deep dive into financial sentiment analysis (2023)
  56. HuggingFace’s Transformers: State-of-the-art Natural Language Processing (2019) — link

0 of 64 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.