Paper: arXiv 2402.06698
Authors: Zihan Dong, Xinyu Fan, Zhiyuan Peng
Abstract
Financial market predictions utilize historical data to anticipate future stock prices and market trends. Traditionally, these predictions have focused on the statistical analysis of quantitative factors, such as stock prices, trading volumes, inflation rates, and changes in industrial production. Recent advancements in large language models motivate the integrated financial analysis of both sentiment data, particularly market news, and numerical factors. Nonetheless, this methodology frequently encounters constraints due to the paucity of extensive datasets that amalgamate both quantitative and qualitative sentiment analyses. To address this challenge, we introduce a large-scale financial dataset, namely, Financial News and Stock Price Integration Dataset (FNSPID). It comprises 29.7 million stock prices and 15.7 million time-aligned financial news records for 4,775 S&P500 companies, covering the period from 1999 to 2023, sourced from 4 stock market news websites. We demonstrate that FNSPID excels existing stock market datasets in scale and diversity while uniquely incorporating sentiment information. Through financial analysis experiments on FNSPID, we propose: (1) the dataset’s size and quality significantly boost market prediction accuracy; (2) adding sentiment scores modestly enhances performance on the transformer-based model; (3) a reproducible procedure that can update the dataset. Completed work, code, documentation, and examples are available at github.com/Zdong104/FNSPID. FNSPID offers unprecedented opportunities for the financial research community to advance predictive modeling and analysis.
Complexity vs Empirical Score
- Math Complexity: 2.0/10
- Empirical Rigor: 8.5/10
- Quadrant: Street Traders — empirical and applied, lower math complexity
Why this score: The paper focuses primarily on dataset construction, empirical experiments, and reproducible code, with minimal advanced mathematical derivations beyond standard regression and model descriptions.
Research Flowchart
flowchart TD A["Research Goal:<br>Integrate Sentiment & Quantitative Data<br>for Stock Prediction"] --> B["Data Acquisition:<br>Collect 15.7M News Records<br>& 29.7M Stock Prices"] B --> C["Data Construction:<br>Time-Align Sentiment & Price Data<br>(1999-2023, S&P 500)"] C --> D["Computational Process:<br>Train Transformer Models<br>with/without Sentiment Scores"] D --> E["Outcome 1:<br>Dataset Scale/Quality<br>Significantly Boosts Accuracy"] D --> F["Outcome 2:<br>Sentiment Scores<br>Modestly Enhance Performance"] D --> G["Outcome 3:<br>Reproducible Procedure<br>for Future Updates"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper introduces a large-scale financial dataset, FNSPID, to improve market prediction with integrated sentiment and price data.
Source sentence
we introduce a large-scale financial dataset, namely, Financial News and Stock Price Integration Dataset (FNSPID).
Assumes
No explicit assumptions found in the text.
Data
- Financial News and Stock Price Integration Dataset (FNSPID) market-prices public
Source sentence
It comprises 29.7 million stock prices and 15.7 million time-aligned financial news records for 4,775 S&P500 companies, covering the period from 1999 to 2023, sourced from 4 stock market news websites.
Costs
Not applicable
Finds 5
- The dataset’s size and quality significantly boost market prediction accuracy. predictive-accuracy
Source sentence
the dataset’s size and quality significantly boost market prediction accuracy
- Adding sentiment scores modestly enhances performance on the transformer-based model. performance
Source sentence
adding sentiment scores modestly enhances performance on the transformer-based model
- Experimental results show an average 6.29% improvement in R² when scaling from 5 to 25 stocks across six models. statistical
Source sentence
The experimental result demonstrated on average 6.29 percent improvement of 𝑅2 from 5 stocks of training to 25 stocks of training among all 6 models we conducted.
- The Transformer model achieved the highest accuracy with R² = 0.988. performance
Source sentence
The Transformer based model has the highest accuracy 𝑅2 = 0.988
- Sentiment integration yields a 0.2% R² gain for FNSPID Task 3, while Textblob sentiment reduces performance by 1.16%. predictive-accuracy
Source sentence
while FNSPID Task 3 has a 0.2% improvement, the Textblob sentiment has a -1.16% impact on the overall stock price prediction.
Checked by others
Supported by
- Understanding the Impact of News Articles on the Movement of Market Index: A Case on Nifty 50
Compare the findings
This paper: Adding sentiment scores modestly enhances performance on the transformer-based model.
That paper: Using VADER sentiment scores with up to 3 lags marginally improves RMSE from 134.85 to 134.15 compared to a model without sentiment.
- Integrating market sentiments for stock price prediction: A comparative study of Bi-LSTM and multilayer perceptions
Compare the findings
This paper: Adding sentiment scores modestly enhances performance on the transformer-based model.
That paper: TextBlob sentiment scores yield the lowest MAE, MSE, and RMSE among all model configurations.
12 papers in the archive : Conformal Risk Certificates for Portfolio Optimization Under … ; Enhancing Risk Assessment in Transformers with Loss-at-Risk … ; Can LLM-based Financial Investing Strategies Outperform the … ; Fusing Narrative Semantics for Financial Volatility … ; Quantifying Semantic Shift in Financial NLP: Robust Metrics …
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (54)
- The cross-section of expected stock returns (1992)
- Some Empirical Tests of the Theory of Arbitrage Pricing (1983)
- Capital Asset Prices: A Theory of Market Equilibrium under Conditions of Risk (1964)
- Stock price prediction using reinforcement learning (2001) — link
- Adaptive stock trading strategies with deep reinforcement learning methods (2020) — link
- Stock price prediction: comparison of different moving average techniques using deep learning model (2024) — link
- Predicting stock market using machine learning: best and accurate way to know future stock prices (2023) — link
- A Comprehensive Comparative Study of Artificial Neural Network (ANN) and Support Vector Machines (SVM) on Stock Forecasting (2023) — link
and 46 more
- Deep Reinforcement Learning for Automated Stock Trading: An Ensemble Strategy (2021) — link
- Stock Market Prediction Using Machine Learning Classifiers and Social Media, News (2022) — link
- Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Models — → 2304.07619
- Stock Prediction Based on Deep Learning and its Application in Pairs Trading (2022) — link
- Design a neural network for time series financial forecasting: Accuracy and robustness analysis (2008)
- Predicting stock market using machine learning: best and accurate way to know future stock prices (2023)
- A stock price prediction method based on deep learning technology (2021)
- News sentiment and stock market volatility (2021)
- Stock Price Prediction using Sentiment Analysis and Deep Learning for Indian Markets (2022) — link
- ChatGPT (2023) — link
- Sentiment Analysis for Stock Price Prediction (2020) — link
- Stock Price Prediction Using News Sentiment Analysis (2019) — link
- One Fits All:Power General Time Series Analysis by Pretrained LM (2023) — link
- FinGPT: Instruction Tuning Benchmark for Open-Source Large Language Models in Financial Datasets — → 2310.04793
- Dynamic Datasets and Market Environments for Financial Reinforcement Learning (2024)
- The legacy of modern portfolio theory (2002)
- A network and machine learning approach to factor, asset, and blended allocation (2020)
- Dynamic connectedness between investors’ sentiment and asset prices: A comparison between major markets in Europe and USA (2023) — link
- Data Augmentation for Sentiment Analysis Using Sentence Compression-Based SeqGAN With Data Screening (2021) — link
- Effect of training set size on SVM and Naive Bayes for Twitter sentiment analysis (2015) — link
- GAN-BElectra: Enhanced Multi-class Sentiment Analysis with Limited Labeled Data (2022) — link
- The impact of different training data set on the accuracy of sentiment classification of Naïve Bayes technique (2017) — link
- Financial News Dataset from Bloomberg and Reuters (2015)
- Corpus of news articles of Lenta.Ru (2019)
- Finnhub Stock APIs
- Sentence-Level Sentiment Analysis of Financial News Using Distributed Text Representations and Multi-Instance Learning (2018) — link
- Leveraging Latent Economic Concepts and Sentiments in the News for Market Prediction (2021) — link
- A novel ensemble deep learning model for stock prediction based on stock prices and news (2020) — link
- Fine-Grained Sentiment Analysis on Financial Microblogs and News Headlines (2017) — link
- News Sentiment Impact Analysis (NSIA) Framework (2016) — link
- SEntFiN 1.0: Entity‐aware sentiment analysis for financial news (2022) — link
- Predicting Mergers and Acquisitions using Graph-based Deep Learning (2021)
- Exploring mutual information-based sentimental analysis with kernel-based extreme learning machine for stock prediction (2017) — link
- Layout guide for Journal of Physics: conference series using Microsoft Word (2023) — link
- Investigating Deep Stock Market Forecasting with Sentiment Analysis (2023) — link
- SemEval-2017 Task 5: Fine-Grained Sentiment Analysis on Financial Microblogs and News (2017) — link
- Text summarization and singular value decomposition (2005)
- Transforming sentiment analysis in the financial domain with ChatGPT (2023)
- ChatGPT: Jack of all trades, master of none (2023)
- Instruct-FinGPT: Financial Sentiment Analysis by Instruction Tuning of General-Purpose Large Language Models — → 2306.12659
- Algorithm and modeling of stock prices forecasting based on long short-term memory (LSTM) (2018)
- Financial time-series data analysis using deep convolutional neural networks (2016)
- Deep learning with gated recurrent unit networks for financial sequence predictions (2018)
- TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis (2023) — link
- Attention is all you need (2017)
- Daily market news sentiment and stock prices (2019) — link
3 of 54 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.