Paper: arXiv 2402.06698

Authors: Zihan Dong, Xinyu Fan, Zhiyuan Peng

Abstract

Financial market predictions utilize historical data to anticipate future stock prices and market trends. Traditionally, these predictions have focused on the statistical analysis of quantitative factors, such as stock prices, trading volumes, inflation rates, and changes in industrial production. Recent advancements in large language models motivate the integrated financial analysis of both sentiment data, particularly market news, and numerical factors. Nonetheless, this methodology frequently encounters constraints due to the paucity of extensive datasets that amalgamate both quantitative and qualitative sentiment analyses. To address this challenge, we introduce a large-scale financial dataset, namely, Financial News and Stock Price Integration Dataset (FNSPID). It comprises 29.7 million stock prices and 15.7 million time-aligned financial news records for 4,775 S&P500 companies, covering the period from 1999 to 2023, sourced from 4 stock market news websites. We demonstrate that FNSPID excels existing stock market datasets in scale and diversity while uniquely incorporating sentiment information. Through financial analysis experiments on FNSPID, we propose: (1) the dataset’s size and quality significantly boost market prediction accuracy; (2) adding sentiment scores modestly enhances performance on the transformer-based model; (3) a reproducible procedure that can update the dataset. Completed work, code, documentation, and examples are available at github.com/Zdong104/FNSPID. FNSPID offers unprecedented opportunities for the financial research community to advance predictive modeling and analysis.

Complexity vs Empirical Score

  • Math Complexity: 2.0/10
  • Empirical Rigor: 8.5/10
  • Quadrant: Street Traders — empirical and applied, lower math complexity

Why this score: The paper focuses primarily on dataset construction, empirical experiments, and reproducible code, with minimal advanced mathematical derivations beyond standard regression and model descriptions.

Research Flowchart

  flowchart TD
  A["Research Goal:<br>Integrate Sentiment & Quantitative Data<br>for Stock Prediction"] --> B["Data Acquisition:<br>Collect 15.7M News Records<br>& 29.7M Stock Prices"]
  B --> C["Data Construction:<br>Time-Align Sentiment & Price Data<br>(1999-2023, S&P 500)"]
  C --> D["Computational Process:<br>Train Transformer Models<br>with/without Sentiment Scores"]
  D --> E["Outcome 1:<br>Dataset Scale/Quality<br>Significantly Boosts Accuracy"]
  D --> F["Outcome 2:<br>Sentiment Scores<br>Modestly Enhance Performance"]
  D --> G["Outcome 3:<br>Reproducible Procedure<br>for Future Updates"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper introduces a large-scale financial dataset, FNSPID, to improve market prediction with integrated sentiment and price data.

Source sentence
we introduce a large-scale financial dataset, namely, Financial News and Stock Price Integration Dataset (FNSPID).

Assumes

No explicit assumptions found in the text.

Data

  • Financial News and Stock Price Integration Dataset (FNSPID) · 4 stock market news websites market-prices public
    1999-2023 · 4,775 S&P500 companies
    Source sentence
    It comprises 29.7 million stock prices and 15.7 million time-aligned financial news records for 4,775 S&P500 companies, covering the period from 1999 to 2023, sourced from 4 stock market news websites.

Recognised sources: FNSPID Bloomberg Yahoo Finance

Costs

Not applicable The paper reports no trading or execution result.

Finds 5

  • The dataset’s size and quality significantly boost market prediction accuracy. predictive-accuracy
    Source sentence
    the dataset’s size and quality significantly boost market prediction accuracy
  • Adding sentiment scores modestly enhances performance on the transformer-based model. performance
    Source sentence
    adding sentiment scores modestly enhances performance on the transformer-based model
  • Experimental results show an average 6.29% improvement in R² when scaling from 5 to 25 stocks across six models. statistical
    Source sentence
    The experimental result demonstrated on average 6.29 percent improvement of 𝑅2 from 5 stocks of training to 25 stocks of training among all 6 models we conducted.
  • The Transformer model achieved the highest accuracy with R² = 0.988. performance
    Source sentence
    The Transformer based model has the highest accuracy 𝑅2 = 0.988
  • Sentiment integration yields a 0.2% R² gain for FNSPID Task 3, while Textblob sentiment reduces performance by 1.16%. predictive-accuracy
    Source sentence
    while FNSPID Task 3 has a 0.2% improvement, the Textblob sentiment has a -1.16% impact on the overall stock price prediction.

Checked by others

Supported by

  • Understanding the Impact of News Articles on the Movement of Market Index: A Case on Nifty 50 (2024)
    On the question: Does adding sentiment scores improve model performance on financial time‑series predictions?
    Compare the findings

    This paper: Adding sentiment scores modestly enhances performance on the transformer-based model.

    That paper: Using VADER sentiment scores with up to 3 lags marginally improves RMSE from 134.85 to 134.15 compared to a model without sentiment.

    Both report modest gains: A notes a modest enhancement, B lowers RMSE from 134.85 to 134.15 (Δ0.70).

  • Integrating market sentiments for stock price prediction: A comparative study of Bi-LSTM and multilayer perceptions (2026)
    On the question: Does incorporating sentiment scores improve stock price prediction performance?
    Compare the findings

    This paper: Adding sentiment scores modestly enhances performance on the transformer-based model.

    That paper: TextBlob sentiment scores yield the lowest MAE, MSE, and RMSE among all model configurations.

    Finding A reports a modest performance boost for a transformer model, and Finding B shows TextBlob sentiment yields the lowest MAE, MSE, and RMSE, supporting A.

Cited by 12 papers in the archive (12 by independent authors): Conformal Risk Certificates for Portfolio Optimization Under … (uses its method or data); Enhancing Risk Assessment in Transformers with Loss-at-Risk … (uses its method or data); Can LLM-based Financial Investing Strategies Outperform the … (uses its method or data); Fusing Narrative Semantics for Financial Volatility … (uses its method or data); Quantifying Semantic Shift in Financial NLP: Robust Metrics … (uses its method or data)

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (54)

  1. The cross-section of expected stock returns (1992)
  2. Some Empirical Tests of the Theory of Arbitrage Pricing (1983)
  3. Capital Asset Prices: A Theory of Market Equilibrium under Conditions of Risk (1964)
  4. Stock price prediction using reinforcement learning (2001) — link
  5. Adaptive stock trading strategies with deep reinforcement learning methods (2020) — link
  6. Stock price prediction: comparison of different moving average techniques using deep learning model (2024) — link
  7. Predicting stock market using machine learning: best and accurate way to know future stock prices (2023) — link
  8. A Comprehensive Comparative Study of Artificial Neural Network (ANN) and Support Vector Machines (SVM) on Stock Forecasting (2023) — link
and 46 more
  1. Deep Reinforcement Learning for Automated Stock Trading: An Ensemble Strategy (2021) — link
  2. Stock Market Prediction Using Machine Learning Classifiers and Social Media, News (2022) — link
  3. Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Models — → 2304.07619
  4. Stock Prediction Based on Deep Learning and its Application in Pairs Trading (2022) — link
  5. Design a neural network for time series financial forecasting: Accuracy and robustness analysis (2008)
  6. Predicting stock market using machine learning: best and accurate way to know future stock prices (2023)
  7. A stock price prediction method based on deep learning technology (2021)
  8. News sentiment and stock market volatility (2021)
  9. Stock Price Prediction using Sentiment Analysis and Deep Learning for Indian Markets (2022) — link
  10. ChatGPT (2023) — link
  11. Sentiment Analysis for Stock Price Prediction (2020) — link
  12. Stock Price Prediction Using News Sentiment Analysis (2019) — link
  13. One Fits All:Power General Time Series Analysis by Pretrained LM (2023) — link
  14. FinGPT: Instruction Tuning Benchmark for Open-Source Large Language Models in Financial Datasets — → 2310.04793
  15. Dynamic Datasets and Market Environments for Financial Reinforcement Learning (2024)
  16. The legacy of modern portfolio theory (2002)
  17. A network and machine learning approach to factor, asset, and blended allocation (2020)
  18. Dynamic connectedness between investors’ sentiment and asset prices: A comparison between major markets in Europe and USA (2023) — link
  19. Data Augmentation for Sentiment Analysis Using Sentence Compression-Based SeqGAN With Data Screening (2021) — link
  20. Effect of training set size on SVM and Naive Bayes for Twitter sentiment analysis (2015) — link
  21. GAN-BElectra: Enhanced Multi-class Sentiment Analysis with Limited Labeled Data (2022) — link
  22. The impact of different training data set on the accuracy of sentiment classification of Naïve Bayes technique (2017) — link
  23. Financial News Dataset from Bloomberg and Reuters (2015)
  24. Corpus of news articles of Lenta.Ru (2019)
  25. Finnhub Stock APIs
  26. Sentence-Level Sentiment Analysis of Financial News Using Distributed Text Representations and Multi-Instance Learning (2018) — link
  27. Leveraging Latent Economic Concepts and Sentiments in the News for Market Prediction (2021) — link
  28. A novel ensemble deep learning model for stock prediction based on stock prices and news (2020) — link
  29. Fine-Grained Sentiment Analysis on Financial Microblogs and News Headlines (2017) — link
  30. News Sentiment Impact Analysis (NSIA) Framework (2016) — link
  31. SEntFiN 1.0: Entity‐aware sentiment analysis for financial news (2022) — link
  32. Predicting Mergers and Acquisitions using Graph-based Deep Learning (2021)
  33. Exploring mutual information-based sentimental analysis with kernel-based extreme learning machine for stock prediction (2017) — link
  34. Layout guide for Journal of Physics: conference series using Microsoft Word (2023) — link
  35. Investigating Deep Stock Market Forecasting with Sentiment Analysis (2023) — link
  36. SemEval-2017 Task 5: Fine-Grained Sentiment Analysis on Financial Microblogs and News (2017) — link
  37. Text summarization and singular value decomposition (2005)
  38. Transforming sentiment analysis in the financial domain with ChatGPT (2023)
  39. ChatGPT: Jack of all trades, master of none (2023)
  40. Instruct-FinGPT: Financial Sentiment Analysis by Instruction Tuning of General-Purpose Large Language Models — → 2306.12659
  41. Algorithm and modeling of stock prices forecasting based on long short-term memory (LSTM) (2018)
  42. Financial time-series data analysis using deep convolutional neural networks (2016)
  43. Deep learning with gated recurrent unit networks for financial sequence predictions (2018)
  44. TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis (2023) — link
  45. Attention is all you need (2017)
  46. Daily market news sentiment and stock prices (2019) — link

3 of 54 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.