Tick data’s volume problem is real — rungs 4–6 of the data ladder run gigabytes per instrument-year — but it’s a layout problem, not a hardware problem: the same data stored well vs badly differs by 5–20× in size and 100× in scan speed. This is the practical layer under the database comparison; most single-researcher setups need only this guide’s file-based approach.
The four decisions that matter
1. Partitioning: date first, then maybe symbol. The universal access pattern is time-ranged; partition Parquet by date=YYYY-MM-DD/ so every backtest scan touches only its window. Add symbol= partitioning only if single-name queries dominate — thousands of small files per day (the many-symbols × daily-partition explosion) is the classic self-inflicted wound; for broad-universe daily work, one file per date with a symbol column beats a directory zoo. Rule of thumb: target individual files of 100MB–1GB; below ~10MB, partitioning overhead dominates.
2. Format: columnar, standard, boring. Parquet remains the vendor-neutral answer: columnar (scans read only needed fields), self-describing, readable by every engine (DuckDB, Polars, Spark, ClickHouse ingest). Keep the raw vendor payloads too — compressed, append-only, in the raw zone — because normalization decisions are revisable only if the pre-normalized truth survives.
3. Compression: zstd, and let sorting do the real work. Zstd at moderate levels is the current default (better ratios than snappy at similar speed). The bigger lever is sort order before writing: ticks sorted by (symbol, timestamp) compress dramatically better than arrival-ordered ones, because deltas shrink — prices become runs of near-identical values, timestamps become small increments. Same data, same codec, 2–5× size difference from sort order alone.
4. Types: the discipline that compounds. Timestamps as int64 nanoseconds UTC (never strings, never local time); prices as int64 in ticks/pips or fixed-point decimal — not float64 (floats waste bits on impossible values, break exact equality on price levels, and accumulate rounding in P&L reconciliation); sizes as integers; categorical columns (symbol, venue, side) dictionary-encoded — automatic in Parquet if the types are right. Schema-drift protection: version your schema, and validate on ingest — the feed will change under you.
The reference layout
data/
├── raw/ # vendor payloads, append-only, zstd
│ └── source=polygon/date=2026-09-05/*.jsonl.zst
├── ticks/ # normalized, sorted, typed
│ └── date=2026-09-05/trades.parquet (symbol, ts, px, sz, venue, cond)
├── quotes/
│ └── date=2026-09-05/nbbo.parquet
└── bars/ # derived, regenerable
└── freq=1m/date=2026-09-05/bars.parquet
Derived layers (bars, features) are regenerable — mark them as such and never let irreplaceable and rebuildable data share a backup policy. Version the transformations, not just the outputs.
Sizing reality check
Order-of-magnitude for a liquid US equity, per year: trades ~50–200MB raw → 10–40MB as sorted zstd Parquet; NBBO quotes 5–10× that; full depth 10–100× again. A 500-name universe’s trades+quotes for a decade fits in low single-digit TB well-stored — one NVMe drive, DuckDB-queryable, no cluster. The cluster conversation starts with full-depth multi-venue data or team concurrency, not before.
Common ways tick storage fails
- The small-files explosion: per-symbol-per-day files across thousands of symbols → filesystem metadata dominates, scans crawl.
- Float prices: exact price-level queries (
px == 100.00) silently miss; order-book reconstruction breaks on representation error. - Arrival-order writes: unsorted files that compress badly and scan worse; sort at ingest, always.
- Timezone soup: mixed local/UTC timestamps discovered during a cross-venue study — the archaeology nobody budgets.
- No ingest validation: schema drift, duplicate days, and gap days enter silently; checks at write time cost minutes and save weeks.
Questions to ask (yourself, quarterly)
- Can I scan a year of one symbol in seconds, and a day of all symbols in seconds? (If not, layout — not hardware.)
- Is anything irreplaceable stored only in a derived format?
- Would a restore from backup reproduce byte-identical files?
Where this connects
This layout is what makes the stack guide’s Parquet+DuckDB default actually fast, what the database engines ingest when you outgrow files, and the substrate the PIT architecture builds its event-sourced layers on.
Engines on top: TimescaleDB vs ClickHouse vs DuckDB → · Parquet vs Postgres → · The correctness layer: PIT architecture → · Size it: tick data storage sizer →