Market data is a ladder, and every research question has a minimum rung: try to answer it from below and you get artifacts, not findings. This guide maps rung → capabilities → cost, so the data budget matches the question.
The ladder
1. Daily OHLCV (~KB/instrument/year). Answers: cross-sectional strategies at daily+ horizons, factor research, trend/vol-targeting at portfolio level. Cannot answer: anything intraday, true execution costs, and even “daily volatility” beyond crude estimators. Hidden subtleties this rung still carries: which close, which adjustments, and consolidated-vs-primary volume.
2. Intraday bars (1s–1m; ~MB/instrument/year). Adds: realized volatility at sane sampling, intraday seasonality, execution-window analysis, event studies at honest timestamps. Still hides: everything inside the bar — bars are summaries, and bar-construction choices (time vs tick vs volume bars) are themselves methodology.
3. Trade prints (tick data) (~10–100MB/instrument/year liquid). Adds: actual transaction analysis, volume profiles, trade-size distributions, signature plots. The classic ceiling: trades alone can’t separate buyer- from seller-initiated flow reliably (inference rules like Lee-Ready guess, with known error rates) — and can’t see the liquidity that didn’t trade.
4. Quotes / top-of-book (L1) (~100MB–1GB/instrument/year). Adds: spreads as they stood (true cost measurement), mid-quote returns (killing bounce artifacts), trade classification against prevailing quotes, basic imbalance. This rung is where most execution research becomes possible — and where timestamp discipline becomes the binding constraint.
5. Order-book snapshots / depth (L2) (~1–10GB/instrument/year). Adds: depth-aware impact and capacity estimates, book-shape features, fill simulation with queue approximations. Snapshot frequency matters: 1-second snapshots miss the intra-second life where HFT ecology lives.
6. Full event feeds (L3/MBO) (~10–100GB+/instrument/year). Adds: exact book reconstruction, queue-position modeling, order-lifecycle analysis (cancellations, spoofing-adjacent patterns), honest market-making backtests. The research rung of record for microstructure claims — and the storage rung where database choices stop being optional.
The two classic mismatches
Working a rung too low — the artifact factory: “mean reversion” from OHLCV that’s bid-ask bounce; execution costs assumed because quotes weren’t available; capacity claims with no depth data behind them. If the claim mentions spreads, fills, or impact, it needs rung 4+; if it mentions queues, rung 6.
Working a rung too high — the budget bonfire: buying L3 feeds to research monthly factor tilts burns storage, money, and engineering time on precision the question can’t use. The notebook-to-production stack principle applies: buy the rung the bottleneck demands.
The self-collection option
For rungs 4–6, one under-used path: record your own — exchange websockets (crypto natively; equities via vendor feeds) into your raw zone. Storage is cheap; history is unbuyable. The prediction-market rule generalizes: start recording before you need the history, because rung 5–6 historical data is the hardest to purchase after the fact.
Questions to ask (of a paper, or your own plan)
- What’s the minimum rung for this claim — and is the paper on it?
- Bars: which construction, and does the result survive an alternative?
- Trades: how is direction classified, and what’s the stated error rate?
- Books: snapshots or events, at what frequency, single-venue or consolidated?
What our scoring captures
Rung-appropriateness is scored implicitly: rigor rewards data descriptions precise enough to check, and the microstructure hub’s ranking correlates strongly with rung honesty — L3-based measurement work at the top, OHLCV-based “microstructure” at the bottom. State your rung; the readers who matter check.
Storage for the upper rungs: tick data efficiently → · database comparison → · The budget: free vs paid →