Paper: arXiv 2508.10192

Authors: Igor Halperin

Abstract

The proliferation of Large Language Models (LLMs) is challenged by hallucinations, critical failure modes where models generate non-factual, nonsensical or unfaithful text. This paper introduces Semantic Divergence Metrics (SDM), a novel lightweight framework for detecting Faithfulness Hallucinations – events of severe deviations of LLMs responses from input contexts. We focus on a specific implementation of these LLM errors, {“confabulations, defined as responses that are arbitrary and semantically misaligned with the user’s query. Existing methods like Semantic Entropy test for arbitrariness by measuring the diversity of answers to a single, fixed prompt. Our SDM framework improves upon this by being more prompt-aware: we test for a deeper form of arbitrariness by measuring response consistency not only across multiple answers but also across multiple, semantically-equivalent paraphrases of the original prompt. Methodologically, our approach uses joint clustering on sentence embeddings to create a shared topic space for prompts and answers. A heatmap of topic co-occurances between prompts and responses can be viewed as a quantified two-dimensional visualization of the user-machine dialogue. We then compute a suite of information-theoretic metrics to measure the semantic divergence between prompts and responses. Our practical score, \(\mathcal{S”}_H\), combines the Jensen-Shannon divergence and Wasserstein distance to quantify this divergence, with a high score indicating a Faithfulness hallucination. Furthermore, we identify the KL divergence KL(Answer \(||\) Prompt) as a powerful indicator of \textbf{“Semantic Exploration”}, a key signal for distinguishing different generative behaviors. These metrics are further combined into the Semantic Box, a diagnostic framework for classifying LLM response types, including the dangerous, confident confabulation.

Complexity vs Empirical Score

  • Math Complexity: 6.5/10
  • Empirical Rigor: 3.0/10
  • Quadrant: Lab Rats — high math complexity, limited empirical testing

Why this score: The paper introduces several information-theoretic metrics (e.g., Jensen-Shannon, Wasserstein, KL divergence) and uses joint clustering on embeddings, requiring dense math, but lacks backtest-ready datasets, code, or heavy statistical validation for real-world trading applications.

Research Flowchart

  flowchart TD
  A["Research Goal<br>Detect Faithfulness Hallucinations<br>& Confabulations"] --> B["Data Input<br>Dataset of LLM Responses<br>to Original & Paraphrased Prompts"]
  B --> C["Core Methodology<br>Joint Clustering of Sentence Embeddings<br>to Create Shared Topic Space"]
  C --> D["Computation: Heatmap<br>Visualize Topic Co-occurrences<br>between Prompts & Responses"]
  D --> E["Computation: Metrics<br>Calculate JSD, Wasserstein Distance<br>& KL(Answer || Prompt)"]
  E --> F["Outcome: Score S_H<br>Quantifies Semantic Divergence<br>High S_H = Faithfulness Hallucination"]
  E --> G["Outcome: Semantic Box<br>Diagnostic Framework<br>Classifies Response Types<br>e.g., Confident Confabulation"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes Semantic Divergence Metrics (SDM), a lightweight prompt‑aware framework that quantifies semantic alignment between prompts and LLM responses using joint clustering of sentence embeddings and information‑theoretic scores, including a combined Jensen‑Shannon/Wasserstein hallucinatio

Source sentence
This paper introduces Semantic Divergence Metrics (SDM), a novel lightweight framework for detecting Faithfulness Hallucinations -- events of severe deviations of LLMs responses from input contexts. We propose a novel lightweight framework, Semantic Divergence Metrics (SDM), to directly quantify this alignment in a more robust, prompt-aware manner.

Assumes 5

  • Sentence‑level embeddings from a pre‑trained transformer reliably capture the semantic content of prompts and answers. model
    Source sentence
    Methodologically, our approach uses joint clustering on sentence embeddings to create a shared topic space for prompts and answers.
  • Paraphrasing the original prompt yields semantically equivalent inputs that can be used to test prompt‑aware arbitrariness. data
    Source sentence
    Our SDM framework improves upon this by being more prompt-aware: we test for a deeper form of arbitrariness by measuring response consistency not only across multiple answers but also across multiple, semantically-equivalent paraphrases of the original prompt.
  • Joint hierarchical agglomerative clustering of prompt and answer embeddings produces meaningful shared topic clusters without needing a predefined number of clusters. model
    Source sentence
    4.5 Joint Semantic Clustering and Topic Estimation A key innovation of our method is the use of joint clustering. Rather than analyzing prompts and answers in isolation, we pool their sentence embeddings into a single dataset, S = P ∪A. We then apply hierarchical agglomerative clustering with Ward’s linkage to this combined set to identify shared semantic topics.
  • KL(Answer || Prompt) reflects the amount of semantic generative exploration required by the model. statistical
    Source sentence
    Furthermore, we identify the KL divergence KL(Answer $||$ Prompt) as a powerful indicator of \textbf{Semantic Exploration}, a key signal for distinguishing different generative behaviors. Furthermore, we identify the KL divergence KL(Answer || Prompt) as a powerful measure of Semantic Exploration, a key signal for distinguishing between different generative behaviors.
Show 1 more
  • The synthetic prompts and generated responses used in experiments are representative of factual, interpretive, and creative tasks. other
    Source sentence
    The first set was specifically designed to test the framework’s sensitivity across a controlled ”stability gradient,” from highly factual to purely creative tasks. The second set used a more diverse range of prompt types to identify different modes of model behavior, including a ”confident hallucination” state.

Data

No dataset named in the text.

Costs

Not applicable The paper reports no trading or execution result.

Finds 5

  • The SH score increases monotonically with prompt ambiguity (0.2918 → 0.3297 → 0.5919), confirming it tracks the stability gradient across factual, interpretive, and creative prompts. performance
    Source sentence
    As hypothesized, the final, complexity-scaled SDM score SH increases monotonically with the prompt’s ambiguity: 0.2918 →0.3297 →0.5919.
  • The low‑stability prompt’s high SH score is driven primarily by a large Ensemble JSD (0.6626), while the moderate‑stability prompt shows the highest Wasserstein distance (0.8782). statistical
    Source sentence
    The Low Stability prompt’s high score is driven primarily by its large Ensemble JSD (0.6626). Conversely, the Moderate Stability prompt’s score is characterized by the highest absolute Wasserstein Distance (0.8782) in the set, suggesting that while the model’s topic structure was somewhat constrained, the specific semantic content it generated varied significantly—the signature of an interpretive task. • KL
  • Ensemble KL(Answer || Prompt) provides a strong signal of semantic exploration, exploding to 19.5591 for the highly creative ‘AGI Dilemma’ prompt. statistical
    Source sentence
    For the highly creative ‘AGI Dilemma‘ prompt, the value explodes to 19.5591, confirming that the model must invent a vast amount of new semantic content to answer.
  • The Semantic Box framework combines SH and KL metrics to classify response behaviors, including the dangerous confident confabulation. theoretical
    Source sentence
    Combining our metrics, we introduce the Semantic Box (Section 7), a novel diagnostic framework that uses our metrics to classify response behaviors—from faithful alignment to the most dangerous, confidently misaligned falsehoods. Figure 3: The Semantic Box framework for classifying LLM response behavior.
  • Baseline Semantic Entropy (SE) fails to correlate with the expected stability gradient, often mis‑ranking complex factual answers as higher risk. negative
    Source sentence
    As shown in our experiments (e.g., Table 1), the SE score did not correlate with the expected stability gradient of the prompts. In fact, it often assigned a higher entropy (suggesting higher risk) to complex, factually-grounded responses than to interpretive ones.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 18 related papers). Coverage is partial, so this is neither replication nor refutation.

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.