Paper: arXiv 2312.16637

Authors: Andrey Shternshis, Stefano Marmi

Abstract

We use the statistical properties of Shannon entropy estimator and Kullback-Leibler divergence to study the predictability of ultra-high frequency financial data. We develop a statistical test for the predictability of a sequence based on empirical frequencies. We show that the degree of randomness grows with the increase of aggregation level in transaction time. We also find that predictable days are usually characterized by high trading activity, i.e., days with unusually high trading volumes and the number of price changes. We find a group of stocks for which predictability is caused by a frequent change of price direction. We study stylized facts that cause price predictability such as persistence of order signs, autocorrelation of returns, and volatility clustering. We perform multiple testing for sub-intervals of days to identify whether there is predictability at a specific time period during the day.

Complexity vs Empirical Score

  • Math Complexity: 7.5/10
  • Empirical Rigor: 4.0/10
  • Quadrant: Lab Rats — high math complexity, limited empirical testing

Why this score: The paper is highly mathematical, featuring rigorous statistical derivations of entropy estimators, Neyman-Pearson tests, and asymptotic distributions (e.g., χ²), but it lacks implementation details, backtest results, or transaction cost analysis, making it more theoretical than backtest-ready.

Research Flowchart

  flowchart TD
  A["Research Goal: Test for price predictability<br>in ultra-high frequency data"] --> B["Methodology: Shannon Entropy &<br>Kullback-Leibler Divergence"]
  B --> C["Data Input: Ultra-High Frequency<br>Equity Transaction Data"]
  C --> D["Computational Process:<br>1. Calculate Entropy at various<br>aggregation levels<br>2. Perform statistical tests<br>3. Analyze stylized facts<br>(order signs, autocorrelation, volatility)"]
  D --> E["Key Findings:<br>- Randomness increases with aggregation<br>- Predictable days = high trading activity<br>- Persistence in order flow causes predictability<br>- Predictability varies by time of day"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes a statistical test for randomness of a symbolic sequence that can be applied to ultra‑high‑frequency price data.

Source sentence
First, we propose the test for randomness of a symbolic sequence. It can be applied with varying numbers of distinct symbols, presuming the sequence length is sufficiently large.

Assumes 6

  • The price direction series is a realization of a stationary random process with symbols from a finite alphabet. statistical
    Source sentence
    We introduce a statistical test designed to evaluate the predictability of a sequence. The input for a statistical test is a realization X of a stationary random process X with symbols from a finite alphabet A = {0, 1, · · · , s −1}.
  • Under the null hypothesis, symbols occur independently of the past history. model
    Source sentence
    H0: The occurrence of a new symbol in sequence X is independent of the sequence’s prior history.
  • The sequence length is sufficiently large for the asymptotic chi‑square distribution to hold. statistical
    Source sentence
    It can be applied with varying numbers of distinct symbols, presuming the sequence length is sufficiently large.
  • Probabilities of symbols need not be equal; the test remains valid. model
    Source sentence
    We show that the proposed test statistics is valid even if the probabilities of appearing symbols are not equal.
Show 2 more
  • Transactions occurring at the same nanosecond are aggregated to a single price to avoid spurious predictability. execution
    Source sentence
    To mitigate the impact of such events on the analysis, we aggregate volumes and consider the final available price for each nanosecond showcasing trading activity.
  • Predictability does not generate arbitrage because it is offset by transaction‑cost fluctuations. market
    Source sentence
    According to [16, 17], such type of predictability does not lead to arbitrage opportunities because it is compensated by fluctuations in transaction costs and liquidity

Data

  • LOBSTER limit order book data · LOBSTER (www.lobsterdata.com) order-book public
    01.08.2022 to 21.11.2022 · Apple, Microsoft, Tesla, Intel, Eli Lilly, Snap, Ford, Carnival, SPY (80 trading days)
    Source sentence
    ... The timeframe under consideration spans from 01.08.2022 to 21.11.2022, encompassing a total of 80 trading days. Table 1 presents the tickers

Recognised sources: LOBSTER

Costs

Not applicable The paper reports no trading or execution result; the cost the extractor found is a model parameter.

The paper-text extraction found a cost parameter inside a model (Predictability is said to be offset by fluctuations in transaction costs and liquidity, but no explicit cost model is incorporated into the headline results.); the classifier found no reported trading result, so costs are not applicable.

Finds 5

  • The degree of randomness of price direction increases as the aggregation level in transaction time grows. statistical
    Source sentence
    We show that the degree of randomness grows with the increase of aggregation level in transaction time.
  • Predictable days are associated with unusually high trading volumes and a larger number of price changes. statistical
    Source sentence
    We also find that predictable days are usually characterized by high trading activity, i.e., days with unusually high trading volumes and the number of price changes.
  • Excluding simultaneous nanosecond‑level transactions reduces the fraction of predictable days. statistical
    Source sentence
    The exclusion of simultaneous transactions leads to a reduction in the predictability of price return signs.
  • For TSLA, predictability decreases sharply with higher aggregation; only a few days remain predictable at aggregation a = 65. predictive-accuracy
    Source sentence
    Increasing the aggregation level by 5, we observe that the probability of repeating a symbol is significantly high even when a = 60. However, by further increasing the aggregation level to a = 65, only 4 predictable days remained
  • Predictable intervals generally do not occur consecutively, except for SNAP where several predictable intervals appear in a row. statistical
    Source sentence
    However, considering all transactions of the SNAP stock (Snap Inc.), we are able to detect several predictable time intervals going in a row.

Checked by others

Supported by

  • Market Intraday Momentum (2014)
    On the question: Does price predictability increase on days with high trading volume and volatility?
    Compare the findings

    This paper: Predictable days are associated with unusually high trading volumes and a larger number of price changes.

    That paper: Predictability is stronger on more volatile days, higher volume days, recession days, and major macro‑economic news release days.

    Both findings report higher predictability on unusually high‑volume days; B adds that volatility and macro news also strengthen the effect, confirming A's volume link.

Cited by 1 paper in the archive

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (65)

  1. The subtle nature of financial random walks (2005) — link
  2. CHAPTER 2 - How Markets Slowly Digest Changes in Supply and Demand (2009) — link
  3. The Econometrics of Ultra-High-Frequency Data (2000) — link
  4. The Long Memory of the Efficient Market (2004) — link
  5. Statistical properties of stock order books: empirical results and models (2002) — link
  6. Autoregressive Conditional Duration: A New Model for Irregularly Spaced Transaction Data (1998) — link
  7. The Stochastic Dependence of Security Price Changes and Transaction Volumes: Implications for the Mixture-of-Distributions Hypothesis (1976) — link
  8. Fluctuations and response in financial markets: the subtle nature of `random’ price changes (2003) — link
and 57 more
  1. Leverage Effect in Financial Markets: The Retarded Volatility Model (2001) — link
  2. An Empirical Analysis of the Limit Order Book and the Order Flow in the Paris Bourse (1995) — link
  3. A New Approach for the Dynamics of Ultra-High-Frequency Data: The Model with Uncertainty Zones (2010) — link
  4. Limit Order Book Markets (2013) — link
  5. Limit order books (2013) — link
  6. Limit Distributions for a Statistical Estimate of the Entropy (1974) — link
  7. A statistical test of market efficiency based on information theory (2023) — link
  8. Rectangular Confidence Regions for the Means of Multivariate Normal Distributions (1967) — link
  9. Sampling returns for realized variance calculations: tick time or transaction time? (2008) — link
  10. Properties of Realized Variance Under Alternative Sampling Schemes (2006) — link
  11. Entropy and the Consistent Estimation of Joint Distributions (1994) — link
  12. A mathematical theory of communication (1948) — link
  13. Statistical Methods in Markov Chains (1961)
  14. The frequency goodness of fit test for probability chains (1951)
  15. The Likelihood Test of Independence in Contingency Tables (1935)
  16. The Ergodic Theory of Discrete Sample Paths (1996)
  17. On Information and Sufficiency (1951) — link
  18. Entropy estimation of symbol sequences (1996) — link
  19. On a Statistical Estimate for the Entropy of a Sequence of Independent Random Variables (1959) — link
  20. Jumps in Financial Markets: A New Nonparametric Test and Jump Dynamics (2007) — link
  21. Fact or friction: Jumps at ultra high frequency (2014) — link
  22. Event studies: A methodology review (2011) — link
  23. Local order, entropy and predictability of financial time series (2000) — link
  24. Stock market uncertainty and economic fundamentals: an entropy-based approach (2019) — link
  25. Entropy and Efficiency of the ETF Market (2020) — link
  26. The Dynamics of Market Efficiency (2016) — link
  27. The adaptive market hypothesis in the high frequency cryptocurrency market (2019) — link
  28. High frequency trading, price discovery and market efficiency in the FTSE100 (2019) — link
  29. Technical trading and cryptocurrencies (2021) — link
  30. Technical trading: Is it still beating the foreign exchange market? (2016) — link
  31. Forecasting high-frequency stock returns: A comparison of alternative methods (2022) — link
  32. Automatic trading method based on piecewise aggregate approximation and multi-swarm of improved self-adaptive particle swarm optimization with validation (2017) — link
  33. Does Academic Research Destroy Stock Return Predictability? (2016) — link
  34. Efficient Capital Markets: A Review of Theory and Empirical Work (1970) — link
  35. A non-parametric test for independence based on symbolic dynamics (2007) — link
  36. Irregularity, volatility, risk, and financial market time series (2004) — link
  37. Market efficiency in foreign exchange markets (2007) — link
  38. A singular value decomposition entropy approach for testing stock market efficiency (2021) — link
  39. On the use and interpretation of certain test criteria for purposes of statistical inference (1928) — link
  40. The valuation effects of stock splits and stock dividends (1984) — link
  41. Stock price reaction to news and no-news: drift and reversal after headlines (2003) — link
  42. Stock Price Reaction to News: The Joint Effect of Tone and Attention on Momentum (2017) — link
  43. Market efficiency in real time (2002) — link
  44. The Behavior of Stock Prices Around Institutional Trades (1995) — link
  45. Long memory and data frequency in financial markets (2019) — link
  46. Theory for long memory in supply and demand (2005)
  47. Estimating a Structural Model of Herd Behavior in Financial Markets (2014)
  48. The impact of institutional trading on stock prices (1992)
  49. A simulation analysis of the microstructure of double auction markets (2002)
  50. Long-memory in an order-driven market (2007)
  51. Foundations of technical analysis: Computational algorithms, statistical inference, and empirical implementation (2000)
  52. What do we know about the profitability of technical analysis? (2007)
  53. The adaptive markets hypothesis: Market efficiency from an evolutionary perspective (2004)
  54. Quantitative trading: algorithms, analytics, data, models, optimization (2017)
  55. Variance of entropy for testing time-varying regimes with an application to meme stocks (2024)
  56. Continuous auctions and insider trading (1985)
  57. Causes of fragile stock market stability (2022) — link

0 of 65 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.