Paper: arXiv 2510.00205
Authors: Zhongtian Sun, Chenghao Xiao, Anoushka Harit, Jongmin Yu
Abstract
Financial news is essential for accurate market prediction, but evolving narratives across macroeconomic regimes introduce semantic and causal drift that weaken model reliability. We present an evaluation framework to quantify robustness in financial NLP under regime shifts. The framework defines four metrics: (1) Financial Causal Attribution Score (FCAS) for alignment with causal cues, (2) Patent Cliff Sensitivity (PCS) for sensitivity to semantic perturbations, (3) Temporal Semantic Volatility (TSV) for drift in latent text representations, and (4) NLI-based Logical Consistency Score (NLICS) for entailment coherence. Applied to LSTM and Transformer models across four economic periods (pre-COVID, COVID, post-COVID, and rate hike), the metrics reveal performance degradation during crises. Semantic volatility and Jensen-Shannon divergence correlate with prediction error. Transformers are more affected by drift, while feature-enhanced variants improve generalisation. A GPT-4 case study confirms that alignment-aware models better preserve causal and logical consistency. The framework supports auditability, stress testing, and adaptive retraining in financial AI systems.
Complexity vs Empirical Score
- Math Complexity: 7.5/10
- Empirical Rigor: 6.8/10
- Quadrant: Holy Grail — high math complexity, high empirical rigor
Why this score: The paper introduces formal mathematical constructs (e.g., distributional shifts, functional mapping for metrics) and uses statistical measures like Jensen-Shannon divergence, showing significant mathematical density. It employs rigorous empirical methods, including multi-period regime testing, model comparison (LSTM vs. Transformer), and a GPT-4 case study, though it lacks full backtest-ready implementation details.
Research Flowchart
flowchart TD A["Research Goal: Quantify robustness<br>of Financial NLP models<br>under regime shifts"] --> B["Methodology: Evaluation Framework"] B --> C["Define Metrics:<br>FCAS, PCS, TSV, NLICS"] B --> D["Data: Financial News across<br>4 Economic Regimes"] C --> E["Computational Process:<br>Apply Metrics to Models<br>LSTM vs Transformer"] D --> E E --> F["Key Findings:<br>Performance degradation during crises"] E --> G["Key Findings:<br>Transformers more affected by drift"] F --> H["Outcome:<br>Framework for stress testing,<br>auditability, & adaptive retraining"] G --> H
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes an evaluation framework for financial NLP robustness under macroeconomic regime shifts, introducing four diagnostic metrics (FCAS, PCS, TSV, NLICS).
Source sentence
We present an evaluation framework to quantify robustness in financial NLP under regime shifts. The framework defines four metrics: (1) Financial Causal Attribution Score (FCAS) for alignment with causal cues, (2) Patent Cliff Sensitivity (PCS) for sensitivity to semantic perturbations, (3) Temporal Semantic Volatility
Assumes 2
- Financial data are non‑stationary, with distributions changing across macroeconomic regimes. data
Source sentence
In practice, financial data are non-stationary, with distribution P changing across macroeconomic regimes.
- News articles are aligned to next‑day stock returns for prediction. market
Source sentence
Articles are timestamp-matched to next-day returns.
Data
- FNSPID news dataset text-news public
Source sentence
News: Financial articles from the FNSPID dataset [4].
- Yahoo Finance daily stock prices market-prices public
Source sentence
Returns: Daily stock prices from Yahoo Finance (2018–2023).
Costs
Not applicable
Finds 5
- Semantic drift measured by Jensen‑Shannon divergence (peak 0.24 between COVID and rate‑hike) is strongly associated with higher prediction error. statistical
Source sentence
Our findings show that semantic drift, measured via Jensen-Shannon divergence (with a peak of 0.24 between COVID and the rate hike period), is strongly associated with elevated prediction error.
- LSTM models exhibit greater stability across regimes, whereas Transformer models are more sensitive to narrative shifts. performance
Source sentence
LSTM models exhibit greater stability across regimes, while Transformers display heightened sensitivity to narrative shifts.
- Feature‑enhanced Transformers reduce semantic volatility and improve generalisation, especially in post‑crisis periods. performance
Source sentence
Feature‑enhanced Transformers reduce semantic volatility and improve generalisation.
- GPT‑4 achieves higher human agreement (85.6%) than BART‑NLI (72.1%) for NLICS, indicating better entailment alignment. performance
Source sentence
GPT‑4 achieves higher agreement with human judgment and better captures nuanced entailment.
- The proposed metrics provide complementary insights beyond accuracy, capturing causal, logical, perturbation, and drift aspects of robustness. other
Source sentence
The proposed metrics offer complementary insights: FCAS and NLICS capture alignment with causal and logical content, while PCS and TSV reflect sensitivity to perturbation and semantic drift.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 31 related papers). Coverage is partial, so this is neither replication nor refutation.
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (133)
- The patent holder’s dilemma: Buy, sell, or troll? (2007) — link
- Deciding equivalances among conjunctive aggregate queries (2007) — link
- Special issue: Digital Libraries (1996)
- Understanding Policy-Based Networking (2001)
- First-Order Dynamic Logic (1979) — link
- The title of book one (2007) — link
- The title of book two (2008) — link
- Achieving application requirements (1990) — link
and 125 more
- Statecarts in use: structured analysis and object-orientation (1998) — link
- The Art of Computer Programming, Vol. 1: Fundamental Algorithms (3rd. ed.) (1997)
- The Art of Computer Programming (1998)
- An experiment in bibliographic mark-up: Parsing metadata for XML export (2010)
- Catch me, if you can: Evading network signatures with web-based polymorphic worms (2007)
- Catch me, if you can: Evading network signatures with web-based polymorphic worms (2008)
- Catch me, if you can: Evading network signatures with web-based polymorphic worms (2009)
- Predicate Path expressions (1979) — link
- LOGICS of Programs: AXIOMATICS and DESCRIPTIVE POWER (1978)
- Optimal Motion Control of a Ground Vehicle (2003)
- Algorithms for Closest-Point Problems (Computational Geometry) (1985)
- Introduction to Bayesian Statistics (2001) — link
- CLIFFORD: a Maple 11 Package for Clifford Algebra Computations, version 11 (2007) — link
- Stats and Analysis (2006) — link
- A more perfect union (2008) — link
- The fountain of youth (2009)
- Solder man (2003) — link
- Interview with Bill Kinder: January 13, 2005 (2005) — link
- The Enabling of Digital Libraries (2008)
- (old) Finding minimum congestion spanning trees (2000) — link
- (new) Finding minimum congestion spanning trees (2000) — link
- (old) Distributed data source verification in wireless sensor networks (2009) — link
- (new) Distributed data source verification in wireless sensor networks (2009) — link
- Portalis: using competitive online interactions to support aid initiatives for the homeless (2008) — link
- Visual Basic 6: Design, Specification, and Objects with Other (1999)
- The Latex Web Companion: Integrating TEX, HTML, and XML (1999)
- Vertex Types in Book-Embeddings (1987)
- Vertex Types in Book-Embeddings (1987)
- CHI ‘08: CHI ‘08 extended abstracts on Human factors in computing systems (2008)
- Algorithms for Closest-Point Problems (Computational Geometry) (1985)
- IEEE TCSC Executive Committee (2004) — link
- Distributed systems (2nd Ed.) (1993)
- New Algorithms for Dependency-Directed Backtracking (Master’s thesis) (1986)
- New Algorithms for Dependency-Directed Backtracking (Master’s thesis) (1986)
- Seminumerical Algorithms (1981)
- E-commerce and cultural values (2001) — link
- Chapter 9 (2002) — link
- The implementation of electronic commerce in SMEs in Singapore (Incoll) (2003) — link
- E-commerce and cultural values - (InBook-num-in-chap) (2004) — link
- E-commerce and cultural values (Inbook-text-in-chap) (2005) — link
- E-commerce and cultural values (Inbook-num chap) (2006) — link
- A library-based synthesis methodology for reversible logic (2010)
- Synthesis of Reversible Circuit Using Cycle-Based Approach (2010)
- Algorithmic Enumeration of Ideal Classes for Quaternion Orders (2010) — link
- Chapter II: Notes on data structuring (1972) — link
- Transcript of question and answer session (1981) — link
- Go to statement considered harmful (1979) — link
- Three-dimensional virtual acoustic displays (1992) — link
- Managerial expert systems and organizational change: some critical research issues (1987) — link
- Simplified Structured COBOL with Microsoft/MicroFocus COBOL (1990)
- A multi-radio unification protocol for IEEE 802.11 wireless networks (2004)
- Wireless Sensor Networks: A Survey (2002)
- A Survey on Wireless Multimedia Sensor Networks (2007)
- SSCH : Slotted Seeded Channel Hopping for Capacity Improvement in IEEE 802.11 Ad-Hoc Wireless Networks (2004)
- XBOW Sensor Motes Specifications (2008)
- Overview of Sensor Networks (2004)
- CodeBlue : Sensor Networks for Medical Care (2008)
- Investigating Network Architectures for Body Sensor Networks (2007)
- Channel-Hopping Multiple Access (2000)
- Body Sensor Networks (2008)
- Mapping Powerlists onto Hypercubes (1994)
- Automatic Parallelization for Distributed-Memory Multiprocessing Systems (1989)
- User recovery and reversal in interactive systems (1984)
- Generalizing specifications for uniformly implemented loops (1985)
- Towards monolingual programming environments (1985)
- The \TeX book (1984)
- Distributed algorithms for finding centers and medians in networks (1984)
- \it \LaTeX : A Document Preparation System (1986)
- Program transformations in a denotational setting (1985)
- Seminumerical Algorithms (1981)
- A high-level approach to computer document formatting (1980)
- A multifrequency MAC specially designed for wireless sensor network applications (2010) — link
- Institutional members of the \TeX Users Group (2017) — link
- acmart— Class for typesetting publications of ACM (2017) — link
- The \textsl doclicense package (2022) — link
- Reasoning About Naming Systems (1993) — link
- Babel, a Multilingual Style-Option System for Use with LaTeX’s Standard Document Styles (1991)
- A Methodology for Implementing Highly Concurrent Data Objects (1993) — link
- Publication quality tables in \LaTeX (2005)
- Using the amsthm Package (2015)
- R: A Language and Environment for Statistical Computing (2019) — link
- UMass Citation Field Extraction Dataset (2013) — link
- Citation concept analysis (CCA) —A new form of citation analysis revealing the usefulness of concepts for other researchers illustrated by two exemplary case studies including classic books by Thomas S.~Kuhn and Karl R.~Popper (2019) — link
- Learning Soft Linear Constraints with Application to Citation Field Extraction (2014) — link
- Maintaining Discrete Probability Distributions Optimally (1993)
- Designing coherent storylines aligned with NGSS for the K-12 classroom (2014) — link
- Fluidity in the Phased Framework of Technology Acceptance: Case Study to Gain a Holistic Understanding of (Older Adult) Participant Advancement Through Acceptance Phases with Mobile Telepresence Robots (2025)
- Dynamic graph neural networks for enhanced volatility prediction in financial markets — → 2410.16858
- Yara Parser: A Fast and Accurate Dependency Parser (2015) — link
- FinBERT: A large language model for extracting information from financial text (2023)
- Deep learning for event-driven stock prediction (2015)
- Financial sentiment analysis: Techniques and applications (2024)
- Fnspid: A comprehensive financial news dataset in time series — → 2402.06698
- Learning under concept drift: A review (2018)
- FinBERT: Financial Sentiment Analysis with Pre-trained Language Models (2019) — link
- Bloomberggpt: A large language model for finance (2023) — link
- On the impact of temporal concept drift on model explanations (2022) — link
- Domain specific concept drift detectors for predicting financial time series (2021) — link
- SEntFiN 1.0: Entity-aware sentiment analysis for financial news (2022)
- Don’t stop pretraining: Adapt language models to domains and tasks (2020) — link
- Logical Consistency of Large Language Models in Fact-checking (2024) — link
- Sentiment and emotion in financial journalism: a corpus-based, cross-linguistic analysis of the effects of COVID (2023)
- The outbreak of COVID-19: An overview (2020)
- A comprehensive capability analysis of gpt-3 and gpt-3.5 series models (2023) — link
- Attention is all you need (2017)
- Long Short-term Memory (1997)
- DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter (2019) — link
- Gpt-4 technical report (2023) — link
- Advanced financial sentiment analysis using FinBERT to explore sentiment dynamics (2025)
- Comparative Advances in Financial Sentiment Analysis: A Review of BERT, FinBert, and Large Language Models (2025)
- Comparative investigation of gpt and finbert’s sentiment analysis performance in news across different sectors (2025)
- Predicting stock prices with finbert-lstm: Integrating news sentiment analysis (2024)
- Enhancing tinybert for financial sentiment analysis using gpt-augmented finbert distillation (2024) — link
- An early prediction model on systemic risk under global risk: Using FinBERT and temporal fusion transformer to multimodal data fusion framework (2025)
- Is domain adaptation worth your investment? comparing BERT and FinBERT on financial tasks (2021)
- Neural text classification for digital transformation in the financial regulatory domain (2022)
- BERT vs GPT for financial engineering — → 2405.12990
- Innovative sentiment analysis and prediction of stock price using FinBERT, GPT-4 and logistic regression: A data-driven approach — → 2412.06837
- Finbert-fomc: Fine-tuned finbert model with sentiment focus method for enhancing sentiment analysis of fomc minutes (2023)
- RicciFlowRec: A Geometric Root Cause Recommender Using Ricci Curvature on Financial Graphs (2025)
- GLANCE: Graph Logic Attention Network with Cluster Enhancement for Heterophilous Graph Representation Learning (2025) — link
- Breaking down financial news impact: A novel ai approach with geometric hypergraphs (2024) — link
- Money: Ensemble learning for stock price movement prediction via a convolutional network with adversarial hypergraph model (2023)
- TextFold: A Geometric Hypergraph Framework for Protein Structure Prediction with Scientific Literature Integration (2025)
- Contrastive learning with heterogeneous graph attention networks on short text classification (2022)
4 of 133 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.