{"paper_id":"2512.03107","title":"Detecting AI Hallucinations in Finance: An Information-Theoretic Method Cuts Hallucination Rate by 92%","authors":["Mainak Singha"],"paper_date":"2025-12-02","paper_type":"empirical","paper_url":"https://arxiv.org/pdf/2512.03107v1","page_url":"https://thequant.space/flowcharts/detecting-ai-hallucinations-in-finance-an-information-theor/","scores":{"empirical_rigor":4,"math_complexity":7.5,"quadrant":"Lab Rats"},"proposes":{"status":"available","statement":"The paper proposes ECLIPSE, a logprob‑native framework that detects hallucinations by modeling the trade‑off between semantic entropy and evidence capacity and by decomposing perplexity to measure evidence utilization.","quote":"We propose ECLIPSE, a framework that treats hallucination as a mismatch between a model’s semantic entropy and the capacity of available evidence.","summary":"Large language models (LLMs) produce fluent but unsupported answers - hallucinations - limiting safe deployment in high-stakes domains. We propose ECLIPSE, a framework that treats hallucination as a mismatch between a model’s semantic entropy and the capacity of available evidence. We combine…","strategy":null},"assumptions":{"status":"available","items":[{"assumption":"The convexity proof holds under mild conditions on the entropy‑capacity objective.","type":"model","stated":"explicit","quote":"We prove that under mild conditions, the resulting entropy–capacity objective is strictly convex with a unique stable optimum."},{"assumption":"Synthetic hallucinations are generated by perturbing correct answers in the financial QA dataset.","type":"data","stated":"explicit","quote":"We evaluate on a controlled financial question answering dataset with GPT-3.5‑turbo (n = 200 balanced samples w"},{"assumption":"Access to token‑level log probabilities is required for the method to work.","type":"model","stated":"explicit","quote":"We call a method logprob‑native if its core signal relies directly on token‑level log probabilities and degrades substantially when those probabilities are replaced by uninformative proxies."},{"assumption":"The financial QA dataset is small (200 samples) and balanced between clean and hallucinated examples.","type":"data","stated":"explicit","quote":"On our controlled financial QA dataset (n = 200), ECLIPSE achieves 0.89 AUC"},{"assumption":"Bootstrap resampling (1000 iterations) provides reliable confidence intervals for AUC.","type":"statistical","stated":"explicit","quote":"For statistical significance, we use a bootstrap test (1000 iterations with replacement) over the full dataset to estimate AUC confidence intervals."},{"assumption":"The capacity feature (Ceff) includes a contradiction penalty aligned with the labeling protocol.","type":"model","stated":"explicit","quote":"Our capacity feature (Ceff) includes an explicit contradiction penalty (wcons) that is aligned with our labeling protocol (identifying synthetic contradictions)."}]},"datasets":{"status":"available","items":[{"name":"controlled financial question answering dataset","provider":null,"kind":"synthetic","access":"unknown","period":null,"universe":null,"quote":"On our controlled financial QA dataset (n = 200), ECLIPSE achieves 0.89 AUC"}],"known":[],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"synthetic"},"transaction_costs":{"status":"not_applicable","modeled":"n/a","headline_net_of_costs":null,"detail":null,"quote":null,"source":"front_matter","rule":"R3","candidates":{"front_matter":"n/a","evidence_card":"n/a","evidence_card_quoted":false},"meaning":"the paper reports no trading or execution result"},"results":{"status":"available","evaluation_type":"forecast-accuracy","findings":[{"finding":"ECLIPSE attains ROC AUC of 0.891 and average precision of 0.89, far surpassing the entropy‑only baseline (AUC ≈0.50).","kind":"performance","quote":"ECLIPSE achieves ROC AUC of 0.891 and average precision of 0.89."},{"finding":"At 30% coverage, ECLIPSE reduces hallucination rate to 3.3% versus 43.3% for entropy‑only, a 92% relative reduction.","kind":"performance","quote":"At 30% coverage (accepting only the most confident 30% of predictions), ECLIPSE achieves 3.3% hallucination rate compared to entropy‑only’s 43.3%—a 92% relative reduction."},{"finding":"Ablation shows entropy alone yields AUC 0.50; adding capacity raises it to 0.68; adding perplexity decomposition reaches 0.89.","kind":"performance","quote":"Entropy alone achieves AUC 0.50; adding capacity improves to 0.68 (+0.18); adding perplexity decomposition reaches 0.89 (+0.21)."},{"finding":"Perplexity decomposition features (LQE, ΔL, ratio) have the largest learned coefficients, confirming they drive discrimination.","kind":"statistical","quote":"The perplexity decomposition features (LQE: +1.73, ∆L: −1.60, ratio: −1.76) exhibit the largest magnitudes, confirming they drive discrimination."},{"finding":"When real log probabilities are unavailable (Claude‑3‑Haiku ablation), performance drops from 0.89 to 0.59 AUC, validating the logprob‑native nature of ECLIPSE.","kind":"negative","quote":"The controlled Claude ablation validates ECLIPSE as logprob‑native: without real log probabilities, performance drops from 0.89 to 0.59 AUC."}],"reported_performance":null,"out_of_sample":{"method":"unclear","headline_out_of_sample":null}},"independent_evidence":{"status":"none_found","supports":[],"contradicts":[],"qualifies":[],"compared_with":39,"citations":{"cited_by_in_corpus":0,"by_independent_authors":0,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":0,"mentions":0,"unclassified":0},"independent_papers":[],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":false,"code_url":null,"note":"No repository presented by the authors as their own was found in the paper. Absence here is not proof that no code exists."},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":1,"extracted_at":"2026-11-14T17:20:18Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}