Introducing Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting for after-work quants

Takeaway. Do not STL-decompose only the history and hope. Autoformer puts moving-average decomposition inside every block, so trend is peeled off while you forecast. Attention becomes Auto-Correlation: FFT finds lags (\tau), then roll sub-series and mix the top-(k) periods — series-wise, not point-wise.

Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting (Haixu Wu, Jiehui Xu, Jianmin Wang, Mingsheng Long, 2021; arXiv:2106.13008). Tsinghua / BNRist. NeurIPS 2021. Code: thuml/Autoformer.

Fit for learners: Seasonal-trend decomposition inside attention — practical for equity/index series with strong cycles.

This Hub note is a plain-language intro. Not investment advice.

The problem

Long horizon (O \gg 1) mixes trend and many periods. Vanilla attention is (O(L^2)) and point-wise; sparse cousins (Informer, LogTrans, Reformer) drop connections and waste the rest. Pre-decomposition of the past only cannot model how seasonal and trend interact in the future.

The idea / core intuition

• Inner block: SeriesDecomp = padding + AvgPool (window 25) → trend; residual → seasonal. Encoder throws trend away and models season; decoder accumulates extracted trend through the stack.
• Decoder seed: last (I/2) of the encoder input, split into seasonal + trend; future slots are zeros (season) and mean (trend).
• Auto-Correlation: Wiener–Khinchin — FFT of Q, conjugate-K, IFFT → (R(\tau)). Keep top (k = \lfloor c \log L \rfloor) lags ((c \in [1,3])). Softmax weights; Roll value by each (\tau) and sum. Complexity (O(L \log L)).
• Encoder–decoder Auto-Correlation uses encoder K/V (resized) and decoder Q.
• Default: 2 encoder layers, 1 decoder, (d=512), L2 loss, Adam (10^{-4}). Input 96 (ILI 36); predict 96/192/336/720.
• Multivariate: ~38% average MSE cut vs prior SOTA on ETT, Electricity, Exchange, Traffic, Weather, ILI. Still competitive on aperiodic Exchange. Lags recovered on Traffic: 24 h and 168 h.

Later DLinear (paper 86) shows a linear + the same MA kernel can beat this architecture on the same tables — read both.