{"paper_id":"2601.04160","title":"All That Glisters Is Not Gold: A Benchmark for Reference-Free Counterfactual Financial Misinformation Detection","authors":["Yuechen Jiang","Zhiwei Liu","Yupeng Cao","Yueru He","Ziyang Xu","Chen Xu","Zhiyang Deng","Prayag Tiwari","Xi Chen","Alejandro Lopez-Lira","Jimin Huang","Junichi Tsujii","Sophia Ananiadou"],"paper_date":"2026-01-07","paper_type":"dataset-benchmark","paper_url":"https://arxiv.org/pdf/2601.04160v2","page_url":"https://thequant.space/flowcharts/all-that-glisters-is-not-gold-a-benchmark-for-reference-fre/","scores":{"empirical_rigor":7,"math_complexity":2.5,"quadrant":"Street Traders"},"proposes":{"status":"available","statement":"The paper proposes RFC-BENCH, a paragraph-level benchmark for reference-free financial misinformation detection using plausibility-preserving counterfactual perturbations.","quote":"We propose RFC-BENCH, a benchmark that operationalizes paragraph-level financial misinformation as plausibility-preserving counterfactual perturbations that shift what a paragraph warrants, enabling evaluation without external evidence.","summary":"We introduce RFC Bench, a benchmark for evaluating large language models on financial misinformation under realistic news. RFC Bench operates at the paragraph level and captures the contextual complexity of financial news where meaning emerges from dispersed cues. The benchmark defines two…","strategy":null},"assumptions":{"status":"available","items":[{"assumption":"The benchmark only covers English-language financial news and U.S. stocks.","type":"data","stated":"explicit","quote":"First, RFC-BENCH includes only English-language financial news and focuses on stocks from the U.S. market, which may limit its applicability to other languages, regions, or financial systems"},{"assumption":"Evaluation considers text‑only inputs without multimodal information.","type":"data","stated":"explicit","quote":"Second, the dataset and evaluation consider text-only inputs and do not incorporate multimodal financial information such as tables, figures, earnings slides, audio, or video"},{"assumption":"Perturbations preserve surface plausibility while shifting the paragraph's warrant.","type":"other","stated":"explicit","quote":"operationalizes paragraph-level financial misinformation as plausibility-preserving counterfactual perturbations that shift what a paragraph warrants"}]},"datasets":{"status":"available","items":[{"name":"RFC-BENCH","provider":null,"kind":"benchmark","access":"public","period":null,"universe":null,"quote":"In total, the final retained set contains 1,845 paragraph-level financial misinformation instances."}],"known":[{"id":"yahoo-finance","name":"Yahoo Finance","provider":"Yahoo","kind":"market-prices","access":"public","quote":"2.2.1 Data Acquisition We collect 1,404 unique financial news articles, each consisting of a title and a summary, from Yahoo Finance 1.","mentions":5,"used_in":1}],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"none"},"transaction_costs":{"status":"not_applicable","modeled":"n/a","headline_net_of_costs":null,"detail":null,"quote":null,"source":"front_matter","rule":"R3","candidates":{"front_matter":"n/a","evidence_card":"n/a","evidence_card_quoted":false},"meaning":"the paper reports no trading or execution result"},"results":{"status":"available","evaluation_type":"forecast-accuracy","findings":[{"finding":"Reference‑free detection (Task 1) performs near chance, with Macro‑F1 below 0.53 and best zero‑shot accuracy of 53.0% (Qwen3‑8B) and 52.7% (GPT‑4.1).","kind":"performance","quote":"Task 1: Reference-free Detection is near chance. When given only one paragraph, all models remain close to chance-level performance, with Macro-F1 below 0.53 and MCC near zero. The best zero-shot accuracy peaks at 53.0% (Qwen3-8B, non-thinking), with GPT-4.1 at 52.7% (Table 5)."},{"finding":"Comparative diagnosis (Task 2) yields high accuracy (0.85–0.97) and substantially higher Macro‑F1 and MCC when original and perturbed paragraphs are provided together.","kind":"performance","quote":"... performance rises sharply: strong models reach 0.85–0.97 accuracy with substantially higher Macro-F1 and MCC (Table 5)."}],"reported_performance":null,"out_of_sample":{"method":"unclear","headline_out_of_sample":null}},"independent_evidence":{"status":"none_found","supports":[],"contradicts":[],"qualifies":[],"compared_with":24,"citations":{"cited_by_in_corpus":0,"by_independent_authors":0,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":0,"mentions":0,"unclassified":0},"independent_papers":[],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":true,"code_url":"https://github.com/lzw108/FMD","note":null},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":1,"extracted_at":"2026-11-14T17:17:03Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}