Detecting AI Hallucinations in Finance: An Information-Theoretic Method Cuts Hallucination Rate by 92%

ArXiv ID: 2512.03107 “View on arXiv”

Authors: Mainak Singha

Abstract

Large language models (LLMs) produce fluent but unsupported answers - hallucinations - limiting safe deployment in high-stakes domains. We propose ECLIPSE, a framework that treats hallucination as a mismatch between a model’s semantic entropy and the capacity of available evidence. We combine entropy estimation via multi-sample clustering with a novel perplexity decomposition that measures how models use retrieved evidence. We prove that under mild conditions, the resulting entropy-capacity objective is strictly convex with a unique stable optimum. We evaluate on a controlled financial question answering dataset with GPT-3.5-turbo (n=200 balanced samples with synthetic hallucinations), where ECLIPSE achieves ROC AUC of 0.89 and average precision of 0.90, substantially outperforming a semantic entropy-only baseline (AUC 0.50). A controlled ablation with Claude-3-Haiku, which lacks token-level log probabilities, shows AUC dropping to 0.59 with coefficient magnitudes decreasing by 95% - demonstrating that ECLIPSE is a logprob-native mechanism whose effectiveness depends on calibrated token-level uncertainties. The perplexity decomposition features exhibit the largest learned coefficients, confirming that evidence utilization is central to hallucination detection. We position this work as a controlled mechanism study; broader validation across domains and naturally occurring hallucinations remains future work.

Keywords: Large Language Models, hallucination detection, semantic entropy, perplexity decomposition, convex optimization, Financial Natural Language Processing

Complexity vs Empirical Score

• Math Complexity: 7.5/10
• Empirical Rigor: 4.0/10
• Quadrant: Lab Rats
• Why: The paper introduces a novel information-theoretic framework with proven convexity and a structured perplexity decomposition, indicating high mathematical complexity. However, the empirical validation is limited to a small, synthetic dataset (n=200) without backtests or real-world implementation details, placing it in the exploratory ‘Lab Rats’ quadrant.

  flowchart TD
    A["Research Goal:<br>Detect AI Hallucinations in Finance"] --> B["Methodology:<br>ECLIPSE Framework"]
    B --> C["Input Data:<br>Financial QA Dataset<br>(n=200 samples)"]
    C --> D["Computational Process:<br>Multi-sample Clustering<br>+ Perplexity Decomposition"]
    D --> E["Key Finding 1:<br>ROC AUC 0.89<br>(vs 0.50 baseline)"]
    D --> F["Key Finding 2:<br>92% Hallucination Rate Cut"]
    D --> G["Key Finding 3:<br>Logprob Dependency Confirmed<br>(AUC drops to 0.59 w/o tokens)"]