Introducing iTransformer: Inverted Transformers Are Effective for Time Series Forecasting for after-work quants

Takeaway. Standard TS Transformers embed all tickers at one timestamp as a token. That glues delayed sensors and different units into one vector, then attends across time (permutation-invariant on a series that cares about order). iTransformer does not invent a new attention. It transposes: one token = one variate’s whole look-back. Attention = who correlates with whom. FFN = temporal encoder.

iTransformer: Inverted Transformers Are Effective for Time Series Forecasting (Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, Mingsheng Long, 2023; ICLR 2024; arXiv:2310.06625). Tsinghua / Ant Group.

Fit for learners: Invert tokens to variables — strong multivariate forecaster for multi-asset return panels in PyTorch.

This Hub note is a plain-language intro. Not investment advice.

The problem

DLinear-style linear models beat many LTSF Transformers. The authors’ diagnosis: the tokenization is wrong. A timestamp token (i) mixes physically different measurements, (ii) may not be the same event if series are lagged (Traffic sensors), (iii) has a tiny receptive field, (iv) wastes permutation-invariant attention on the time axis that actually needs order. PatchTST enlarges the field but still treats channels independently.

The idea / core intuition

• X is T by N. Transpose to N tokens of length T. MLP embedding: R^T -> R^D. No positional encoding — order lives in the FFN’s neuron layout.
• Encoder-only stack: LayerNorm per token (stabilizes mixed units; related to RevIN/stationarization), self-attention over N, shared FFN on each token’s D features, project R^D -> R^S independently per variate.
• Attention scores N by N are readable as variate correlation. Shallow maps look like look-back Pearson; deeper maps look more like future correlation (Solar-Energy case).
• Plug-in: any efficient attention (Reformer, Informer, Flowformer, FlashAttention) becomes an i- variant because N (assets) is the sequence length, not T.
• Look-back T=96, horizons {96,192,336,720} (PEMS {12,24,36,48}). Avg MSE: ECL 0.178 vs PatchTST 0.205 / TimesNet 0.192; Traffic 0.428 vs PatchTST 0.481; Weather 0.258; Solar 0.233; PEMS avg 0.119. High-N panels are the sweet spot.