{"paper_id":"2510.00205","title":"Quantifying Semantic Shift in Financial NLP: Robust Metrics for Market Prediction Stability","authors":["Zhongtian Sun","Chenghao Xiao","Anoushka Harit","Jongmin Yu"],"paper_date":"2025-09-30","paper_type":"empirical","paper_url":"https://arxiv.org/pdf/2510.00205v1","page_url":"https://thequant.space/flowcharts/quantifying-semantic-shift-in-financial-nlp-robust-metrics/","scores":{"empirical_rigor":6.8,"math_complexity":7.5,"quadrant":"Holy Grail"},"proposes":{"status":"available","statement":"The paper proposes an evaluation framework for financial NLP robustness under macroeconomic regime shifts, introducing four diagnostic metrics (FCAS, PCS, TSV, NLICS).","quote":"We present an evaluation framework to quantify robustness in financial NLP under regime shifts. The framework defines four metrics: (1) Financial Causal Attribution Score (FCAS) for alignment with causal cues, (2) Patent Cliff Sensitivity (PCS) for sensitivity to semantic perturbations, (3) Temporal Semantic Volatility","summary":"Financial news is essential for accurate market prediction, but evolving narratives across macroeconomic regimes introduce semantic and causal drift that weaken model reliability. We present an evaluation framework to quantify robustness in financial NLP under regime shifts. The framework defines…","strategy":null},"assumptions":{"status":"available","items":[{"assumption":"Financial data are non‑stationary, with distributions changing across macroeconomic regimes.","type":"data","stated":"explicit","quote":"In practice, financial data are non-stationary, with distribution P changing across macroeconomic regimes."},{"assumption":"News articles are aligned to next‑day stock returns for prediction.","type":"market","stated":"explicit","quote":"Articles are timestamp-matched to next-day returns."}]},"datasets":{"status":"available","items":[{"name":"FNSPID news dataset","provider":"FNSPID","kind":"text-news","access":"public","period":"2018–2023","universe":"110 S&P 500 companies across 11 GICS sectors","quote":"News: Financial articles from the FNSPID dataset [4].","known_id":"fnspid"},{"name":"Yahoo Finance daily stock prices","provider":"Yahoo Finance","kind":"market-prices","access":"public","period":"2018–2023","universe":"110 S&P 500 companies","quote":"Returns: Daily stock prices from Yahoo Finance (2018–2023).","known_id":"yahoo-finance"}],"known":[{"id":"fnspid","name":"FNSPID","provider":"Dong et al.","kind":"text-news","access":"public","quote":"Data Sources. • News: Financial articles from the FNSPID dataset [4]. • Returns: Daily stock prices from Yahoo Finance (2018–2023). • Models: Implemented in PyTorch (LSTM, Transformer vari-ants). • NLI Evaluation: GPT-4 and BART-NLI (HuggingFace Trans-formers).","mentions":3,"used_in":2},{"id":"yahoo-finance","name":"Yahoo Finance","provider":"Yahoo","kind":"market-prices","access":"public","quote":"Data Sources. • News: Financial articles from the FNSPID dataset [4]. • Returns: Daily stock prices from Yahoo Finance (2018–2023). • Models: Implemented in PyTorch (LSTM, Transformer vari-ants). • NLI Evaluation: GPT-4 and BART-NLI (HuggingFace Trans-formers).","mentions":1,"used_in":1}],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"daily"},"transaction_costs":{"status":"not_applicable","modeled":"n/a","headline_net_of_costs":null,"detail":null,"quote":null,"source":"front_matter","rule":"R3","candidates":{"front_matter":"n/a","evidence_card":"n/a","evidence_card_quoted":false},"meaning":"the paper reports no trading or execution result"},"results":{"status":"available","evaluation_type":"forecast-accuracy","findings":[{"finding":"Semantic drift measured by Jensen‑Shannon divergence (peak 0.24 between COVID and rate‑hike) is strongly associated with higher prediction error.","kind":"statistical","quote":"Our findings show that semantic drift, measured via Jensen-Shannon divergence (with a peak of 0.24 between COVID and the rate hike period), is strongly associated with elevated prediction error."},{"finding":"LSTM models exhibit greater stability across regimes, whereas Transformer models are more sensitive to narrative shifts.","kind":"performance","quote":"LSTM models exhibit greater stability across regimes, while Transformers display heightened sensitivity to narrative shifts."},{"finding":"Feature‑enhanced Transformers reduce semantic volatility and improve generalisation, especially in post‑crisis periods.","kind":"performance","quote":"Feature‑enhanced Transformers reduce semantic volatility and improve generalisation."},{"finding":"GPT‑4 achieves higher human agreement (85.6%) than BART‑NLI (72.1%) for NLICS, indicating better entailment alignment.","kind":"performance","quote":"GPT‑4 achieves higher agreement with human judgment and better captures nuanced entailment."},{"finding":"The proposed metrics provide complementary insights beyond accuracy, capturing causal, logical, perturbation, and drift aspects of robustness.","kind":"other","quote":"The proposed metrics offer complementary insights: FCAS and NLICS capture alignment with causal and logical content, while PCS and TSV reflect sensitivity to perturbation and semantic drift."}],"reported_performance":null,"out_of_sample":{"method":"holdout","headline_out_of_sample":null}},"independent_evidence":{"status":"none_found","supports":[],"contradicts":[],"qualifies":[],"compared_with":31,"citations":{"cited_by_in_corpus":0,"by_independent_authors":0,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":0,"mentions":0,"unclassified":0},"independent_papers":[],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":false,"code_url":null,"note":"No repository presented by the authors as their own was found in the paper. Absence here is not proof that no code exists."},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":1,"extracted_at":"2026-11-14T16:29:09Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}