Paper: arXiv 2312.16637
Authors: Andrey Shternshis, Stefano Marmi
Abstract
We use the statistical properties of Shannon entropy estimator and Kullback-Leibler divergence to study the predictability of ultra-high frequency financial data. We develop a statistical test for the predictability of a sequence based on empirical frequencies. We show that the degree of randomness grows with the increase of aggregation level in transaction time. We also find that predictable days are usually characterized by high trading activity, i.e., days with unusually high trading volumes and the number of price changes. We find a group of stocks for which predictability is caused by a frequent change of price direction. We study stylized facts that cause price predictability such as persistence of order signs, autocorrelation of returns, and volatility clustering. We perform multiple testing for sub-intervals of days to identify whether there is predictability at a specific time period during the day.
Complexity vs Empirical Score
- Math Complexity: 7.5/10
- Empirical Rigor: 4.0/10
- Quadrant: Lab Rats — high math complexity, limited empirical testing
Why this score: The paper is highly mathematical, featuring rigorous statistical derivations of entropy estimators, Neyman-Pearson tests, and asymptotic distributions (e.g., χ²), but it lacks implementation details, backtest results, or transaction cost analysis, making it more theoretical than backtest-ready.
Research Flowchart
flowchart TD A["Research Goal: Test for price predictability<br>in ultra-high frequency data"] --> B["Methodology: Shannon Entropy &<br>Kullback-Leibler Divergence"] B --> C["Data Input: Ultra-High Frequency<br>Equity Transaction Data"] C --> D["Computational Process:<br>1. Calculate Entropy at various<br>aggregation levels<br>2. Perform statistical tests<br>3. Analyze stylized facts<br>(order signs, autocorrelation, volatility)"] D --> E["Key Findings:<br>- Randomness increases with aggregation<br>- Predictable days = high trading activity<br>- Persistence in order flow causes predictability<br>- Predictability varies by time of day"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes a statistical test for randomness of a symbolic sequence that can be applied to ultra‑high‑frequency price data.
Source sentence
First, we propose the test for randomness of a symbolic sequence. It can be applied with varying numbers of distinct symbols, presuming the sequence length is sufficiently large.
Assumes 6
- The price direction series is a realization of a stationary random process with symbols from a finite alphabet. statistical
Source sentence
We introduce a statistical test designed to evaluate the predictability of a sequence. The input for a statistical test is a realization X of a stationary random process X with symbols from a finite alphabet A = {0, 1, · · · , s −1}.
- Under the null hypothesis, symbols occur independently of the past history. model
Source sentence
H0: The occurrence of a new symbol in sequence X is independent of the sequence’s prior history.
- The sequence length is sufficiently large for the asymptotic chi‑square distribution to hold. statistical
Source sentence
It can be applied with varying numbers of distinct symbols, presuming the sequence length is sufficiently large.
- Probabilities of symbols need not be equal; the test remains valid. model
Source sentence
We show that the proposed test statistics is valid even if the probabilities of appearing symbols are not equal.
Show 2 more
- Transactions occurring at the same nanosecond are aggregated to a single price to avoid spurious predictability. execution
Source sentence
To mitigate the impact of such events on the analysis, we aggregate volumes and consider the final available price for each nanosecond showcasing trading activity.
- Predictability does not generate arbitrage because it is offset by transaction‑cost fluctuations. market
Source sentence
According to [16, 17], such type of predictability does not lead to arbitrage opportunities because it is compensated by fluctuations in transaction costs and liquidity
Data
- LOBSTER limit order book data order-book public
Source sentence
... The timeframe under consideration spans from 01.08.2022 to 21.11.2022, encompassing a total of 80 trading days. Table 1 presents the tickers
Costs
Not applicable
Finds 5
- The degree of randomness of price direction increases as the aggregation level in transaction time grows. statistical
Source sentence
We show that the degree of randomness grows with the increase of aggregation level in transaction time.
- Predictable days are associated with unusually high trading volumes and a larger number of price changes. statistical
Source sentence
We also find that predictable days are usually characterized by high trading activity, i.e., days with unusually high trading volumes and the number of price changes.
- Excluding simultaneous nanosecond‑level transactions reduces the fraction of predictable days. statistical
Source sentence
The exclusion of simultaneous transactions leads to a reduction in the predictability of price return signs.
- For TSLA, predictability decreases sharply with higher aggregation; only a few days remain predictable at aggregation a = 65. predictive-accuracy
Source sentence
Increasing the aggregation level by 5, we observe that the probability of repeating a symbol is significantly high even when a = 60. However, by further increasing the aggregation level to a = 65, only 4 predictable days remained
- Predictable intervals generally do not occur consecutively, except for SNAP where several predictable intervals appear in a row. statistical
Source sentence
However, considering all transactions of the SNAP stock (Snap Inc.), we are able to detect several predictable time intervals going in a row.
Checked by others
Supported by
- Market Intraday Momentum
Compare the findings
This paper: Predictable days are associated with unusually high trading volumes and a larger number of price changes.
That paper: Predictability is stronger on more volatile days, higher volume days, recession days, and major macro‑economic news release days.
1 paper in the archive
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (65)
- The subtle nature of financial random walks (2005) — link
- CHAPTER 2 - How Markets Slowly Digest Changes in Supply and Demand (2009) — link
- The Econometrics of Ultra-High-Frequency Data (2000) — link
- The Long Memory of the Efficient Market (2004) — link
- Statistical properties of stock order books: empirical results and models (2002) — link
- Autoregressive Conditional Duration: A New Model for Irregularly Spaced Transaction Data (1998) — link
- The Stochastic Dependence of Security Price Changes and Transaction Volumes: Implications for the Mixture-of-Distributions Hypothesis (1976) — link
- Fluctuations and response in financial markets: the subtle nature of `random’ price changes (2003) — link
and 57 more
- Leverage Effect in Financial Markets: The Retarded Volatility Model (2001) — link
- An Empirical Analysis of the Limit Order Book and the Order Flow in the Paris Bourse (1995) — link
- A New Approach for the Dynamics of Ultra-High-Frequency Data: The Model with Uncertainty Zones (2010) — link
- Limit Order Book Markets (2013) — link
- Limit order books (2013) — link
- Limit Distributions for a Statistical Estimate of the Entropy (1974) — link
- A statistical test of market efficiency based on information theory (2023) — link
- Rectangular Confidence Regions for the Means of Multivariate Normal Distributions (1967) — link
- Sampling returns for realized variance calculations: tick time or transaction time? (2008) — link
- Properties of Realized Variance Under Alternative Sampling Schemes (2006) — link
- Entropy and the Consistent Estimation of Joint Distributions (1994) — link
- A mathematical theory of communication (1948) — link
- Statistical Methods in Markov Chains (1961)
- The frequency goodness of fit test for probability chains (1951)
- The Likelihood Test of Independence in Contingency Tables (1935)
- The Ergodic Theory of Discrete Sample Paths (1996)
- On Information and Sufficiency (1951) — link
- Entropy estimation of symbol sequences (1996) — link
- On a Statistical Estimate for the Entropy of a Sequence of Independent Random Variables (1959) — link
- Jumps in Financial Markets: A New Nonparametric Test and Jump Dynamics (2007) — link
- Fact or friction: Jumps at ultra high frequency (2014) — link
- Event studies: A methodology review (2011) — link
- Local order, entropy and predictability of financial time series (2000) — link
- Stock market uncertainty and economic fundamentals: an entropy-based approach (2019) — link
- Entropy and Efficiency of the ETF Market (2020) — link
- The Dynamics of Market Efficiency (2016) — link
- The adaptive market hypothesis in the high frequency cryptocurrency market (2019) — link
- High frequency trading, price discovery and market efficiency in the FTSE100 (2019) — link
- Technical trading and cryptocurrencies (2021) — link
- Technical trading: Is it still beating the foreign exchange market? (2016) — link
- Forecasting high-frequency stock returns: A comparison of alternative methods (2022) — link
- Automatic trading method based on piecewise aggregate approximation and multi-swarm of improved self-adaptive particle swarm optimization with validation (2017) — link
- Does Academic Research Destroy Stock Return Predictability? (2016) — link
- Efficient Capital Markets: A Review of Theory and Empirical Work (1970) — link
- A non-parametric test for independence based on symbolic dynamics (2007) — link
- Irregularity, volatility, risk, and financial market time series (2004) — link
- Market efficiency in foreign exchange markets (2007) — link
- A singular value decomposition entropy approach for testing stock market efficiency (2021) — link
- On the use and interpretation of certain test criteria for purposes of statistical inference (1928) — link
- The valuation effects of stock splits and stock dividends (1984) — link
- Stock price reaction to news and no-news: drift and reversal after headlines (2003) — link
- Stock Price Reaction to News: The Joint Effect of Tone and Attention on Momentum (2017) — link
- Market efficiency in real time (2002) — link
- The Behavior of Stock Prices Around Institutional Trades (1995) — link
- Long memory and data frequency in financial markets (2019) — link
- Theory for long memory in supply and demand (2005)
- Estimating a Structural Model of Herd Behavior in Financial Markets (2014)
- The impact of institutional trading on stock prices (1992)
- A simulation analysis of the microstructure of double auction markets (2002)
- Long-memory in an order-driven market (2007)
- Foundations of technical analysis: Computational algorithms, statistical inference, and empirical implementation (2000)
- What do we know about the profitability of technical analysis? (2007)
- The adaptive markets hypothesis: Market efficiency from an evolutionary perspective (2004)
- Quantitative trading: algorithms, analytics, data, models, optimization (2017)
- Variance of entropy for testing time-varying regimes with an application to meme stocks (2024)
- Continuous auctions and insider trading (1985)
- Causes of fragile stock market stability (2022) — link
0 of 65 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.