Introducing Deep Reinforcement Learning for Trading for after-work quants

Takeaway. Do not forecast then invent a rule. Train the agent to output a position in [−1, 0, +1] (or continuous [−1, 1] for A2C). Volatility scaling sizes the trade by σ_tgt / σ_t so a quiet contract and a wild one sit on the same reward scale, and fees sit inside the reward.

Deep Reinforcement Learning for Trading (Zihao Zhang, Stefan Zohren, Stephen Roberts, 2019; arXiv:1911.10107). Pinnacle 50 ratio-adjusted futures (commodities, equity indices, rates, FX); train up to each 5-year block, test 2011–2019.

Fit for learners: DQN/PG/A2C on futures with volatility scaling — concrete RL trading loop to code after work.

This Hub note is a plain-language intro. Not investment advice.

The problem

Supervised nets predict a short-horizon return; large trends last weeks. Mapping that number to a held position is the missing step. Sign-of-12-month-return (TSMOM) and MACD rules work in trending books (rates) and bleed in choppy FX. Fully long or short with no size control is painful in high-vol regimes.

The idea / core intuition

• State: 60 days of normalised close, multi-horizon returns (1m–1y, vol-scaled), MACD, 30-day RSI.
• Actions: DQN and PG use discrete {−1, 0, +1} target positions (same position → no extra fee). A2C uses continuous [−1, 1].
• Reward: additive PnL with vol targeting minus a bp × price × |Δ scaled position| cost. Linear utility ↔ maximise expected cumulative trade returns.
• LSTMs (64 then 32) per asset class; Adam; γ = 0.3; train-time cost 20 bp as a turnover regulariser.
• Vs long-only, Sign(R), MACD: after portfolio-level vol targeting, DQN leads the all-contract book (paper Sharpe ~1.3 vs ~0.4 for Sign(R)); equity-index bull years still favour long-only. DQN/A2C stay positive out to ~25 bp costs in their plots.

A research futures study—not a CTA product.

Why after-work Python learners care

You see a full loop: features → position → vol-scaled PnL minus cost. That is closer to a notebook you can write than a weight-softmax crypto paper.

Tiny evening exercise: position +1, r = 0.01, σ = 0.02, σ_tgt = 0.15. Scaled PnL ≈ (σ_tgt/σ) × r. Compute it. Then subtract 2 bp × |Δposition| if you flipped from 0. Learning only.

One-evening reading checklist

1. Why the output is a position, not a predicted return.
2. What volatility scaling does to reward and size.
3. Discrete DQN/PG vs continuous A2C.
4. Which asset class still likes buy-and-hold in 2011–2019.

Download PDF

Download PDF

Landing: arXiv abs

Important disclaimer

Not investment advice. This Hub note is educational only. Zhang–Zohren–Roberts is academic RL on historical futures—not a recommendation to run a vol-targeted futures book.

What this is not

• Not CEF / subsidy framing
• Not Quant or Macro course CTA
• Not a CTA or broker pitch

Next step

Single course CTA: SoloQuant Academy’s Introduction to Python Programming—multiplying a return by a scale factor is the first line of a vol-targeted PnL.