Kullback-Leibler Divergence

Semantic Faithfulness and Entropy Production Measures to Tame Your LLM Demons and Manage Hallucinations ArXiv ID: 2512.05156 “View on arXiv” Authors: Igor Halperin Abstract Evaluating faithfulness of Large Language Models (LLMs) to a given task is a complex challenge. We propose two new unsupervised metrics for faithfulness evaluation using insights from information theory and thermodynamics. Our approach treats an LLM as a bipartite information engine where hidden layers act as a Maxwell demon controlling transformations of context $C $ into answer $A$ via prompt $Q$. We model Question-Context-Answer (QCA) triplets as probability distributions over shared topics. Topic transformations from $C$ to $Q$ and $A$ are modeled as transition matrices ${"\bf Q"}$ and ${"\bf A"}$ encoding the query goal and actual result, respectively. Our semantic faithfulness (SF) metric quantifies faithfulness for any given QCA triplet by the Kullback-Leibler (KL) divergence between these matrices. Both matrices are inferred simultaneously via convex optimization of this KL divergence, and the final SF metric is obtained by mapping the minimal divergence onto the unit interval [“0,1”], where higher scores indicate greater faithfulness. Furthermore, we propose a thermodynamics-based semantic entropy production (SEP) metric in answer generation, and show that high faithfulness generally implies low entropy production. The SF and SEP metrics can be used jointly or separately for LLM evaluation and hallucination control. We demonstrate our framework on LLM summarization of corporate SEC 10-K filings. ...

Financial Information Theory ArXiv ID: 2511.16339 “View on arXiv” Authors: Miquel Noguer i Alonso Abstract This paper introduces a comprehensive framework for Financial Information Theory by applying information-theoretic concepts such as entropy, Kullback-Leibler divergence, mutual information, normalized mutual information, and transfer entropy to financial time series. We systematically derive these measures with complete mathematical proofs, establish their theoretical properties, and propose practical algorithms for estimation. Using S&P 500 data from 2000 to 2025, we demonstrate empirical usefulness for regime detection, market efficiency testing, and portfolio construction. We show that normalized mutual information (NMI) behaves as a powerful, bounded, and interpretable measure of temporal dependence, highlighting periods of structural change such as the 2008 financial crisis and the COVID-19 shock. Our entropy-adjusted Value at Risk, information-theoretic diversification criterion, and NMI-based market efficiency test provide actionable tools for risk management and asset allocation. We interpret NMI as a quantitative diagnostic of the Efficient Market Hypothesis and demonstrate that information-theoretic methods offer superior regime detection compared to traditional autocorrelation- or volatility-based approaches. All theoretical results include rigorous proofs, and empirical findings are validated across multiple market regimes spanning 25 years of daily returns. ...

Entropy-Guided Multiplicative Updates: KL Projections for Multi-Factor Target Exposures ArXiv ID: 2510.24607 “View on arXiv” Authors: Yimeng Qiu Abstract We introduce Entropy-Guided Multiplicative Updates (EGMU), a convex optimization framework for constructing multi-factor target-exposure portfolios by minimizing Kullback-Leibler divergence from a benchmark under linear factor constraints. We establish feasibility and uniqueness of strictly positive solutions when the benchmark and targets satisfy convex-hull conditions. We derive the dual concave formulation with explicit gradient, Hessian, and sensitivity expressions, and provide two provably convergent solvers: a damped dual Newton method with global convergence and local quadratic rate, and a KL-projection scheme based on iterative proportional fitting and Bregman-Dykstra projections. We further generalize EGMU to handle elastic targets and robust target sets, and introduce a path-following ordinary differential equation for tracing solution trajectories. Stable and scalable implementations are provided using LogSumExp stabilization, covariance regularization, and half-space KL projections. Our focus is on theory and reproducible algorithms; empirical benchmarking is optional. ...

Through the Looking Glass: Bitcoin Treasury Companies ArXiv ID: 2507.14910 “View on arXiv” Authors: B K Meister Abstract Bitcoin treasury companies have taken stock markets by storm amassing billions of dollars worth of tokens in hundreds of entities. The paper discusses, how leverage - whether created through corporate debt or investors using stock as loan collateral - fuels this trend. The extension of the binary-choice Kelly criterion to incorporate uncertainty in the form of the Kullback-Leibler divergence or more generally Bregman divergence is also briefly discussed. ...

Application of the Kelly Criterion to Prediction Markets ArXiv ID: 2412.14144 “View on arXiv” Authors: Unknown Abstract Betting markets are gaining in popularity. Mean beliefs generally differ from prices in prediction markets. Logarithmic utility is employed to study the risk and return adjustments to prices. Some consequences are described. A modified payout structure is proposed. A simple asset price model based on flipping biased coins is investigated. It is shown using the Kullback-Leibler divergence how the misjudgment of the bias and the miscalculation of the investment fraction influence the portfolio growth rate. ...

Price predictability at ultra-high frequency: Entropy-based randomness test ArXiv ID: 2312.16637 “View on arXiv” Authors: Unknown Abstract We use the statistical properties of Shannon entropy estimator and Kullback-Leibler divergence to study the predictability of ultra-high frequency financial data. We develop a statistical test for the predictability of a sequence based on empirical frequencies. We show that the degree of randomness grows with the increase of aggregation level in transaction time. We also find that predictable days are usually characterized by high trading activity, i.e., days with unusually high trading volumes and the number of price changes. We find a group of stocks for which predictability is caused by a frequent change of price direction. We study stylized facts that cause price predictability such as persistence of order signs, autocorrelation of returns, and volatility clustering. We perform multiple testing for sub-intervals of days to identify whether there is predictability at a specific time period during the day. ...

Trend patterns statistics for assessing irreversibility in cryptocurrencies: time-asymmetry versus inefficiency ArXiv ID: 2307.08612 “View on arXiv” Authors: Unknown Abstract In this paper, we present a measure of time irreversibility using trend pattern statistics. We define the irreversibility index as the Kullback-Leibler divergence between the distribution of uptrends subsequences (increasing trends) and the corresponding downtrends subsequences distribution (decreasing trends) in a time series. We use this index to analyze the degree of irreversibility in log return series over time, specifically focusing on five cryptocurrencies: Bitcoin, Ethereum, Ripple, Litecoin, and Bitcoin Cash. Our analysis reveals a strong indication of irreversibility in all these cryptocurrencies and the characteristic evolves over time. We additionally evaluate the market efficiency for these cryptocurrencies based on a recently proposed information-theoretic measure. By comparing inefficiency and irreversibility, we explore the relationship between these statistical features. This comparison provides insight into the non-trivial relationship between inefficiency and irreversibility. ...