Introducing Model-based Reinforcement Learning for Predictions and Control for Limit Order Books for after-work quants

Wei, Wang, Mangu and Decker (2019) train a CNN autoencoder, RNN-MDN transition and reward head on HKEX LOB+prints, then PG/A2C/Double DQN entirely inside the world model. Policy transfers back to replay. Plain-language intro. Not investment advice.

Introducing Model-based Reinforcement Learning for Predictions and Control for Limit Order Books for after-work quants

Takeaway. Model-free trading RL hammers a replay tape. This paper first learns a world model (latent LOB + next-state mix + reward), then the agent only talks to the simulator. They report the policy still prints on held-out April 2018 days.

Model-based Reinforcement Learning for Predictions and Control for Limit Order Books (Haoran Wei, Yuanbo Wang, Lidia Mangu, Keith Decker, 2019; arXiv:1910.03743). Delaware + J.P. Morgan. One HKEX name; ~0.17s LOB updates; 2018 Q1 train / April test; ~6M / 2M transitions.

Fit for learners: Model-based RL on LOB dynamics — alternative to model-free FinRL when you want a learned market simulator.

This Hub note is a plain-language intro. Not investment advice.

The problem

Atari-style DQN needs millions of real steps. In a market those steps cost money and do not reset. A hand-coded simulator stacks approximation error; a hand-coded policy stacks corner cases. World Models (Ha & Schmidhuber) showed you can dream in latent space—this is that idea on books.

The idea / core intuition

State = CNN autoencoder of a (T=40)-tick × 3-level LOB window (latent dim 16) + up to 10 recent trade prints + inventory in ([-po_{max}, po_{max}]).
Transitions: sequence-to-sequence RNN-MDN (5 Gaussians) maps ((z, u, a)) to a distribution over next (z'). Historical prints are the exploratory actions (like other cars in a driving log).
Reward head predicts (\Delta)mid; mark-to-market (r = \Delta\mathrm{mid}\times po). Position clips at capacity. Fees 2% of unit PnL in the demo plots.
Actions discretized to 21 sizes ({-1000,\ldots,0,\ldots,1000}) at mid. Train horizon 500 states (~1 hour of market).
Agents: Double DQN, policy gradient, A2C inside the frozen world. A2C is the most stable. Vs momentum band, CNN mid-move classifier + rule, and a greedy-one-step oracle: RL sits near the oracle and ~10–30% above the average of the two heuristics on four random test days. Weak in oscillating regimes (~16% of train).

Transfer: same weights, five random days, world vs real tape—cumulative reward tracks. Still observational impact (prints already happened).

Why after-work Python learners care

You separate “learn the market” from “learn the policy.” FinRL-style model-free gyms skip the first box; this paper is the first box.

Tiny evening exercise: 21 lots from −1000 to +1000. If (po_{max}=2000) and (po_t=1800), which buy actions clip? Learning only.

One-evening reading checklist

Why an autoencoder before the RNN, not raw 3-level books.
Why an MDN (mixture), not a point next-state.
Reward uses predicted (\Delta)mid × position—not a broker fill file.
Why oscillating days fail (data share).

Download PDF

Download PDF

Landing: arXiv abs

Important disclaimer

Not investment advice. This Hub note is educational only. Wei–Wang–Mangu–Decker is a 2018 HKEX world-model study—not a recommendation to trade latent-space policies with real capital.

What this is not

Not CEF / subsidy framing
Not Quant or Macro course CTA
Not a trading-bot pitch

Next step

Single course CTA: SoloQuant Academy’s Introduction to Python Programming—normalizing a (T \times 4L) window before an encoder is a nested loop you already meet in week one.