Paper: arXiv 2512.03107

Authors: Mainak Singha

Abstract

Large language models (LLMs) produce fluent but unsupported answers - hallucinations - limiting safe deployment in high-stakes domains. We propose ECLIPSE, a framework that treats hallucination as a mismatch between a model’s semantic entropy and the capacity of available evidence. We combine entropy estimation via multi-sample clustering with a novel perplexity decomposition that measures how models use retrieved evidence. We prove that under mild conditions, the resulting entropy-capacity objective is strictly convex with a unique stable optimum. We evaluate on a controlled financial question answering dataset with GPT-3.5-turbo (n=200 balanced samples with synthetic hallucinations), where ECLIPSE achieves ROC AUC of 0.89 and average precision of 0.90, substantially outperforming a semantic entropy-only baseline (AUC 0.50). A controlled ablation with Claude-3-Haiku, which lacks token-level log probabilities, shows AUC dropping to 0.59 with coefficient magnitudes decreasing by 95% - demonstrating that ECLIPSE is a logprob-native mechanism whose effectiveness depends on calibrated token-level uncertainties. The perplexity decomposition features exhibit the largest learned coefficients, confirming that evidence utilization is central to hallucination detection. We position this work as a controlled mechanism study; broader validation across domains and naturally occurring hallucinations remains future work.

Complexity vs Empirical Score

  • Math Complexity: 7.5/10
  • Empirical Rigor: 4.0/10
  • Quadrant: Lab Rats — high math complexity, limited empirical testing

Why this score: The paper introduces a novel information-theoretic framework with proven convexity and a structured perplexity decomposition, indicating high mathematical complexity. However, the empirical validation is limited to a small, synthetic dataset (n=200) without backtests or real-world implementation details, placing it in the exploratory ‘Lab Rats’ quadrant.

Research Flowchart

  flowchart TD
  A["Research Goal:<br>Detect AI Hallucinations in Finance"] --> B["Methodology:<br>ECLIPSE Framework"]
  B --> C["Input Data:<br>Financial QA Dataset<br>(n=200 samples)"]
  C --> D["Computational Process:<br>Multi-sample Clustering<br>+ Perplexity Decomposition"]
  D --> E["Key Finding 1:<br>ROC AUC 0.89<br>(vs 0.50 baseline)"]
  D --> F["Key Finding 2:<br>92% Hallucination Rate Cut"]
  D --> G["Key Finding 3:<br>Logprob Dependency Confirmed<br>(AUC drops to 0.59 w/o tokens)"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes ECLIPSE, a logprob‑native framework that detects hallucinations by modeling the trade‑off between semantic entropy and evidence capacity and by decomposing perplexity to measure evidence utilization.

Source sentence
We propose ECLIPSE, a framework that treats hallucination as a mismatch between a model’s semantic entropy and the capacity of available evidence.

Assumes 6

  • The convexity proof holds under mild conditions on the entropy‑capacity objective. model
    Source sentence
    We prove that under mild conditions, the resulting entropy–capacity objective is strictly convex with a unique stable optimum.
  • Synthetic hallucinations are generated by perturbing correct answers in the financial QA dataset. data
    Source sentence
    We evaluate on a controlled financial question answering dataset with GPT-3.5‑turbo (n = 200 balanced samples w
  • Access to token‑level log probabilities is required for the method to work. model
    Source sentence
    We call a method logprob‑native if its core signal relies directly on token‑level log probabilities and degrades substantially when those probabilities are replaced by uninformative proxies.
  • The financial QA dataset is small (200 samples) and balanced between clean and hallucinated examples. data
    Source sentence
    On our controlled financial QA dataset (n = 200), ECLIPSE achieves 0.89 AUC
Show 2 more
  • Bootstrap resampling (1000 iterations) provides reliable confidence intervals for AUC. statistical
    Source sentence
    For statistical significance, we use a bootstrap test (1000 iterations with replacement) over the full dataset to estimate AUC confidence intervals.
  • The capacity feature (Ceff) includes a contradiction penalty aligned with the labeling protocol. model
    Source sentence
    Our capacity feature (Ceff) includes an explicit contradiction penalty (wcons) that is aligned with our labeling protocol (identifying synthetic contradictions).

Data

  • controlled financial question answering dataset synthetic unknown
    Source sentence
    On our controlled financial QA dataset (n = 200), ECLIPSE achieves 0.89 AUC

Costs

Not applicable The paper reports no trading or execution result.

Finds 5

  • ECLIPSE attains ROC AUC of 0.891 and average precision of 0.89, far surpassing the entropy‑only baseline (AUC ≈0.50). performance
    Source sentence
    ECLIPSE achieves ROC AUC of 0.891 and average precision of 0.89.
  • At 30% coverage, ECLIPSE reduces hallucination rate to 3.3% versus 43.3% for entropy‑only, a 92% relative reduction. performance
    Source sentence
    At 30% coverage (accepting only the most confident 30% of predictions), ECLIPSE achieves 3.3% hallucination rate compared to entropy‑only’s 43.3%—a 92% relative reduction.
  • Ablation shows entropy alone yields AUC 0.50; adding capacity raises it to 0.68; adding perplexity decomposition reaches 0.89. performance
    Source sentence
    Entropy alone achieves AUC 0.50; adding capacity improves to 0.68 (+0.18); adding perplexity decomposition reaches 0.89 (+0.21).
  • Perplexity decomposition features (LQE, ΔL, ratio) have the largest learned coefficients, confirming they drive discrimination. statistical
    Source sentence
    The perplexity decomposition features (LQE: +1.73, ∆L: −1.60, ratio: −1.76) exhibit the largest magnitudes, confirming they drive discrimination.
  • When real log probabilities are unavailable (Claude‑3‑Haiku ablation), performance drops from 0.89 to 0.59 AUC, validating the logprob‑native nature of ECLIPSE. negative
    Source sentence
    The controlled Claude ablation validates ECLIPSE as logprob‑native: without real log probabilities, performance drops from 0.89 to 0.59 AUC.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 39 related papers). Coverage is partial, so this is neither replication nor refutation.

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (31)

  1. The Internal State of an LLM Knows When It’s Lying (2023)
  2. Chat L aw: Open-Source Legal Large Language Model with Integrated External Knowledge Bases (2023) — link
  3. On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models? (2022)
  4. On the Foundations of Noise-Free Selective Classification (2010)
  5. Dropout as a B ayesian Approximation: Representing Model Uncertainty in Deep Learning (2016)
  6. Selective Classification for Deep Neural Networks (2017)
  7. On Calibration of Modern Neural Networks (2017)
  8. REALM : Retrieval-Augmented Language Model Pre-Training (2020)
and 23 more
  1. The Curious Case of Neural Text Degeneration (2020)
  2. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023) — link
  3. Atlas: Few-shot Learning with Retrieval Augmented Language Models (2023)
  4. Survey of Hallucination in Natural Language Generation (2023)
  5. Language Models (Mostly) Know What They Know (2022) — link
  6. Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs (2024) — link
  7. Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation (2023)
  8. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020)
  9. Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models (2024)
  10. Evaluating Verifiability in Generative Search Engines (2023)
  11. Self C heck GPT : Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models (2023)
  12. On Faithfulness and Factuality in Abstractive Summarization (2020)
  13. FA ct S core: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023)
  14. Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods (1999)
  15. Large Language Models Can Be Easily Distracted by Irrelevant Context (2023)
  16. Retrieval Augmentation Reduces Hallucination in Conversation (2021)
  17. Large Language Models Encode Clinical Knowledge (2023)
  18. Investigating Selective Prediction Approaches Across Several Tasks in IID , OOD , and Adversarial Settings (2022)
  19. Bloomberg GPT : A Large Language Model for Finance (2023) — link
  20. Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs (2024)
  21. A Theory of Usable Information Under Computational Constraints (2020) — link
  22. Making Retrieval-Augmented Language Models Robust to Irrelevant Context (2024)
  23. Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models (2023) — link

0 of 31 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.