Introducing Market Making with Deep Reinforcement Learning from Limit Order Books for after-work quants

Takeaway. Hand-crafted spreads and Avellaneda–Stoikov closed forms assume a price process. This paper feeds raw LOB snapshots through Attn-LOB (CNN + inception + multi-head attention), then a market maker quotes a reservation price and a spread with two numbers in ([0,1]).

Market Making with Deep Reinforcement Learning from Limit Order Books (Hong Guo, Jianwu Lin, Fanlin Huang, 2023; arXiv:2305.15821). Tsinghua SIGS + Microsoft. Nov 2019 SZSE event data; Ping An Bank, Wuliangye, Hikvision.

Fit for learners: Attn-LOB + continuous reservation-price quotes — a readable DRL market-making loop on reconstructed China A-share books.

This Hub note is a plain-language intro. Not investment advice.

The problem

Stoll’s split of MM costs (adverse selection / processing / inventory) is still the job. Classic AS needs (\sigma,\gamma,\kappa). Prior RL MM papers often used inventory + time or a few LOB gadgets and a discrete 8-action grid. Neither uses the book as a tensor.

The idea / core intuition

• State: (T=50) snapshots × 10 levels × (ask px, ask qty, bid px, bid qty), plus OSI (order-strength on new limits / markets / cancels over 10s/60s/300s), realized vol, RSI, inventory, remaining time.
• Pre-train Attn-LOB to classify next mid move (up / flat / down) with Ntakaris-style labels; then freeze it as the MM encoder. F1 ~0.73 vs DeepLOB ~0.71 on Ping An Bank.
• Continuous action (C-PPO): (A_1) shifts reservation (p_r = p_m - \mathrm{sign}(q)\delta); (A_2) sets quoted spread. Inventory sign forces you to bid nearer when short. Cap (\omega=10) lots per side.
• Reward = dampened PnL (clip upside from holding) + trading PnL (edge vs mid at fill) − (\zeta q^2).
• Simulator: fill only when a historical opposing order arrives (zero impact, zero fees). Episodes ~2000 events (~3–5 min), flatten with a market order at the end.
• C-PPO beats AS, random, fixed-level quotes, inventory-only RL and handcrafted-LOB RL on ND-PnL and PnL-per-inventory; more robust to added latency than fixed rules.

A historical replay back-test. Tiny quotes, so they ignore own-book impact.

Why after-work Python learners care

You connect DeepLOB-style encoders to an AS-shaped action, and you see why holding PnL as a reward creates directional gambling.

Tiny evening exercise: given mid 10.00, inventory (q=+3), (A_1=0.4), (\mathrm{max_bias}=0.05), (A_2=0.5), (\mathrm{max_spread}=0.10), write (p_r, p_a, p_b). Learning only.

One-evening reading checklist

1. Why pre-train mid direction, then RL the quotes.
2. How inventory sign enters (p_r).
3. Dampened PnL vs trading PnL vs (q^2).
4. Replay fills vs a reactive ABM (see JAX-LOB).

Download PDF

Download PDF

Landing: arXiv abs

Important disclaimer

Not investment advice. This Hub note is educational only. Guo–Lin–Huang is academic MM on 2019 SZSE prints—not a recommendation to quote two-sided size in production.

What this is not

• Not CEF / subsidy framing
• Not Quant or Macro course CTA
• Not a trading-bot pitch

Next step

Single course CTA: SoloQuant Academy’s Introduction to Python Programming—rolling windows over (T, 4n) LOB arrays is the state you will actually code.