Introducing Practical Deep Reinforcement Learning Approach for Stock Trading for after-work quants

Takeaway. Markowitz is two-step: forecast means/cov, then allocate. Revising every day with costs is messy. This paper writes trading as an MDP and uses DDPG so the action can be a vector of buy/sell sizes instead of a gigantic discrete grid.

Practical Deep Reinforcement Learning Approach for Stock Trading (Xiao-Yang Liu, Zhuoran Xiong, Shan Zhong, Hongyang Yang, Anwar Walid, 2018; arXiv:1811.07522). Columbia + Nokia Bell Labs. DJIA 30 as of 1 Jan 2016; Compustat daily 2009-01-01 to 2018-09-30.

Fit for learners: DDPG on DJIA 30 daily bars — the Columbia paper that later grew into FinRL; a concrete actor-critic trading MDP.

This Hub note is a plain-language intro. Not investment advice.

The problem

Dynamic programming on a 30-name book explodes: holdings are integers, prices are continuous. DQN’s max-over-actions is hopeless. You want a deterministic actor (\mu(s)) that outputs the whole book’s trades at once.

The idea / core intuition

• State (s=[p,h,b]): 30 prices, 30 integer lots, cash. Action per name: sell (k\in[1,h_d]), hold, or buy (k) lots, subject to cash (\ge 0) after sells-then-buys.
• Reward = change in portfolio value (p^\top h + b).
• DDPG: actor + critic, target nets, replay, exploration noise on (\mu(s)). Soft updates with (\tau).
• Split: train 2009–2014, validate 2015, trade 2016-01-01 to 2018-09-30 while continuing to train. Initial $10,000.
• Reported test: DDPG final ~$19,791 vs min-var ~$14,369 vs DJIA ~$15,428; annualized return ~25.9% / 15.9% / 16.4%; Sharpe ~1.79 / 1.45 / 1.27. (Table 1 vs the 22.24% sentence in the text disagree slightly—read the table.)

No transaction cost in the published MDP; daily bars only; DJIA 30 survivorship as of 2016. A research back-test.

Why after-work Python learners care

This is the Columbia DDPG trading note that the FinRL line cites. Once you can write state = concat(price, lots, cash), the rest is library PPO/DDPG.

Tiny evening exercise: two names, (p=(10, 20)), (h=(3, 0)), (b=25). Is buying 2 shares of name 2 feasible? After a sell of 1 share of name 1 first? Learning only.

One-evening reading checklist

1. Why DDPG, not DQN, for 30 names.
2. What is in (s)—and what is not (no volume, no news).
3. Train / validate / trade split, and online updates in the test window.
4. Sharpe vs raw wealth: they beat both baselines on both.

Download PDF

Download PDF

Landing: arXiv abs

Important disclaimer

Not investment advice. This Hub note is educational only. Liu–Xiong–Zhong–Yang–Walid is an academic DDPG on 30 US names—not a recommendation to run actor-critic size with real capital.

What this is not

• Not CEF / subsidy framing
• Not Quant or Macro course CTA
• Not a trading-bot pitch

Next step

Single course CTA: SoloQuant Academy’s Introduction to Python Programming—concatenating three arrays into one state vector is list-and-loop work, not magic.