Definition. An RL trading paper trains an agent to act — allocate, execute, quote, hedge — by interacting with a market environment and maximizing cumulative reward. The framing is seductive because it skips the forecast-then-optimize pipeline; the evaluation problem is that markets are close to the hardest possible RL setting — non-stationary, near-random-walk, weak reward signal — so our RL hub contains both genuine advances and the field’s most spectacular simulator victories.
The worked example: winning a game nobody plays
A DQN portfolio agent reports 3× buy-and-hold on five years of daily crypto data. Deconstruct the environment: the agent observes today’s OHLCV including the close, acts “at” that close (same-bar execution), pays zero spread in an asset class where taker costs are 2–10 bps+, and its reward includes unrealized mark-to-market on positions sized without impact. Each simplification is defensible in an RL-methods paper; jointly they define a game whose optimal policy has no market counterpart. The agent won the game. The game was the leak.
The RL-paper checklist
Everything on the ML checklist applies. Additionally:
- ☐ Environment audit before results: observation timing vs action timing, cost and impact model, and whether rewards use information the agent couldn’t have. In RL papers the environment is the methodology — read it the way you’d read a backtest engine.
- ☐ Dumb-baseline comparison: buy-and-hold, equal-weight, TWAP (for execution). RL papers benchmarking only against other RL agents are comparing losers of different games.
- ☐ Train/test regime separation: an agent trained through 2020–21 crypto and tested on 2022 tells you something; trained and tested inside one bull regime tells you nothing. Regime dependence is fatal here because policies encode regime implicitly.
- ☐ Seeds, and more than ML needs: RL variance across seeds is notoriously large; fewer than 5 seeds with dispersion is a screenshot.
- ☐ Sample-efficiency honesty: how many environment steps did training take, and how many independent market periods does that actually represent? A million steps over one replayed year is one year, replayed.
- ☐ Reward-hacking check: does the reward function admit degenerate optima (e.g., volatility-pumping unrealized P&L, or churning for a per-trade bonus)?
Where RL claims deserve real attention
Calibrate skepticism by sub-domain. Execution and market making — dense feedback, clear cost structure, stationary-enough microstructure — is where RL results replicate and deploy; the optimal-execution literature meets RL credibly. Hedging (deep hedging under transaction costs) is the second defensible island. End-to-end portfolio alpha from prices is where the sample-efficiency arithmetic is unforgiving; hold those papers to the maximum standard.
Questions to ask while reading
- Could the environment’s optimal policy be profitable in a real market, even in principle?
- What fraction of the reported gain survives next-bar execution and realistic costs?
- How many truly independent market regimes did the agent train across?
- Is the comparison seed-matched, cost-matched, and correctly trained on the baseline side?
- Does the paper report policy behavior (turnover, exposure, inventory) or only returns? Behavior is where hacking shows.
What our scoring captures — and misses
The rigor axis treats environment realism as empirical rigor, so RL papers with honest cost/impact modelling and multi-seed reporting surface at the top of the hub. The blind spot is symmetric with ML’s: a described environment can differ from the implemented one, and reward hacking is often only visible in training curves and logged behavior a paper omits. When an RL result matters to you, the environment code is the paper — reproduce it or don’t rely on it.
Companions: ML trading papers → · Regime dependence → · Backtest checklist → · From the corpus: Monte Carlo policy gradient (MCPG) in trading →