Introducing An intelligent financial portfolio trading strategy using deep Q-learning for after-work quants

Takeaway. Skip continuous portfolio weights. This paper uses discrete actions: for each asset, buy, hold, or sell a fixed lot (encoded −1 / 0 / +1). A mapping function turns cash-short or inventory-short orders into the nearest feasible neighbour so the agent does not flip from “sell both” to “buy both.”

An intelligent financial portfolio trading strategy using deep Q-learning (Hyungjun Park, Min Kyu Sim, Dong Gu Choi, 2019; arXiv:1907.03665). Long-only cash + three assets; US ETFs (SPY / Russell 1000 Value / microcap) and Korean KOSPI size indices; 0.25% fees; 2017 test year.

Fit for learners: Discrete DQN portfolio actions — approachable RL starter if continuous PPO feels heavy.

This Hub note is a plain-language intro. Not investment advice.

The problem

Most DRL trading papers either trade one name or output target weights. Weights are not an order ticket: you still decide how to get there, and unbounded Δw can explode turnover. Discrete lots look “less optimal” on paper, but they cap daily size the way many real desks already do.

The idea / core intuition

• Action vector (a_t \in {-1,0,1}^I): sell / hold / buy a fixed δ of each of I assets. Space size (3^I).
• Infeasible actions (no cash to buy, no shares to sell) are masked, then mapped to a similar feasible vector by largest Q among nearest neighbours—not by “next-best Q in the whole remaining set.”
• Q-net: shared LSTM autoencoder on a 20-day window of five technical features per asset (close/open/high/low/volume changes), concatenated with current weights, then a small DNN.
• Simulate every feasible action each day and store the experience list—one year of bars is not enough joint (state, multi-asset action) data otherwise.
• Reward is excess vs do-nothing next-day value, so a bull market does not automatically praise every buy.
• 2017 back-test vs buy-and-hold, random, momentum, reversion: DQN wins cumulative return and Sharpe on both books, with lower average turnover than the active heuristics (still not zero; B&H has none). Mapping cut direction-flip counts ~35–60% and lifted cumulative return ~7–11 percentage points vs unmapped DQN.

A research back-test—not a live book.

Why after-work Python learners care

You meet discrete action combinatorics, feasibility, and why fees + turnover belong in the design, not a footnote. Continuous PPO can wait until this loop is clear.

Tiny evening exercise: two assets, δ = 1, cash = 0.5, holdings = (0.3, 0.2). Which of the nine (−1/0/+1)² actions are infeasible? List them. Learning only.

One-evening reading checklist

1. Why the action is ±δ lots, not a target weight vector.
2. What the mapping function does when cash or inventory is short.
3. Why they simulate all feasible actions instead of one ε-greedy step.
4. Reward = value vs static (no-trade) next day—not raw portfolio return.

Download PDF

Download PDF

Landing: arXiv abs

Important disclaimer

Not investment advice. This Hub note is educational only. Park–Sim–Choi is academic DQN on 2017 index/ETF prints—not a recommendation to run a discrete-lot agent with real capital.

What this is not

• Not CEF / subsidy framing
• Not Quant or Macro course CTA
• Not a trading-bot pitch

Next step

Single course CTA: SoloQuant Academy’s Introduction to Python Programming—nested loops over three choices per asset is how you enumerate a discrete action grid.