Introducing A Time Series is Worth 64 Words: Long-term Forecasting with Transformers (PatchTST) for after-work quants

Takeaway. Point-wise tokens make a Transformer attend ticks, not motifs. PatchTST groups (P) steps into one token (stride (S)) and runs channel-independent series through a shared vanilla encoder—ViT, but for univariate streams.

A Time Series is Worth 64 Words: Long-term Forecasting with Transformers (Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, Jayant Kalagnanam, 2022; arXiv:2211.14730 / ICLR 2023). Princeton + IBM Research. Weather, Traffic, Electricity, ILI, four ETT sets.

Fit for learners: Patch + channel-independence transformer that often beats fancy TS models — easy to port to equity return panels.

This Hub note is a plain-language intro. Not investment advice.

The problem

Informer / Autoformer / FEDformer invented sparse attention because (N=L) is quadratic. Zeng et al. (DLinear) then beat them with linear layers, which made people ask whether Transformers even belong in LTSF. PatchTST’s answer: the architecture was fine; the tokenization was not.

The idea / core intuition

• Split (M) variates; each length-(L) series → patches of length (P), stride (S). Number of tokens (N \approx L/S) (they pad the last value). Default (P=16), (S=8); PatchTST/42 uses (L=336) ((N=42)); /64 uses (L=512).
• Instance-norm per series (zero mean, unit var; add back after the head) to fight train/test shift.
• Vanilla encoder, BatchNorm inside blocks, flatten + linear head to horizon (T). Loss = MSE averaged over channels.
• Complexity of attention drops (\sim S^2). Traffic (L=336): ~22× faster with patches vs tokens-per-tick.
• Supervised: ~20–21% MSE cut vs the best prior Transformer; wins especially on large panels (Traffic 862, Electricity 321, Weather 21). Often beats DLinear there too.
• Masked pre-train: non-overlap patches, 40% zeroed, reconstruct. Fine-tune (linear probe then full) can beat train-from-scratch on large sets. Electricity-pretrained weights transfer to other sets.

They drop Exchange-rate from the main claim: random-walk last-value is already a brutal benchmark (Fama / Rossi). Equity returns are closer to that warning than electricity load.

Why after-work Python learners care

You can forecast a panel of names without mixing them in the same token—and you get a reason DLinear won the last round.

Tiny evening exercise: (L=336), (P=16), (S=8). How many patches if you pad one extra step of the last value? (Paper: (N = \lfloor (L-P)/S \rfloor + 2).) Learning only.

One-evening reading checklist

1. Patch vs point token: what local semantics you keep.
2. Channel-independence vs mixing all (M) features in one vector.
3. Why longer look-back finally helps (it did not for Informer-style models).
4. Masked patches, not masked ticks (ticks interpolate; patches must be understood).

Download PDF

Download PDF

Landing: arXiv abs

Important disclaimer

Not investment advice. This Hub note is educational only. PatchTST is an academic LTSF architecture on ETT/Traffic/load/weather—not a recommendation to trade Transformer forecasts.

What this is not

• Not CEF / subsidy framing
• Not Quant or Macro course CTA
• Not a trading-bot pitch

Next step

Single course CTA: SoloQuant Academy’s Introduction to Python Programming—slicing a 1-D array into overlapping windows of length (P) is the patch operator.