Introducing FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting for after-work quants

Takeaway. Pointwise Transformer forecasts can drift in distribution (trend/frequency of the path looks wrong even if MSE is OK). FEDformer (1) decomposes trend vs seasonal with a mixture-of-experts moving average, (2) runs self/cross-attention in the frequency domain on a random subset of modes (keep high and low), complexity (O(L)).

FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting (Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, Rong Jin, 2022; ICML; arXiv:2201.12740). Alibaba Machine Intelligence. Code: MAZiqing/FEDformer.

Fit for learners: Fourier/Wavelet enhanced transformer — good next step after Autoformer for frequency-aware market forecasts.

This Hub note is a plain-language intro. Not investment advice.

The problem

Informer / LogTrans / Reformer made attention cheaper. They still attend in time. On ETTm1 a vanilla Transformer’s histogram of levels fails a KS test vs the history — the forecast “sounds” like a different series. Autoformer already decomposes; FEDformer adds sparse frequency mixing.

The idea / core intuition

• MOEDecomp: several average-pool kernels (7, 12, 14, 24, 48) mixed by a softmax of a linear map — trend is not one fixed window.
• FEB-f: DFT → select (M\approx 64) random modes → multiply by a learned complex kernel (R) → pad → iDFT. Replaces self-attention.
• FEA-f: same select on (Q,K,V) in frequency; (\sigma(\tilde Q \tilde K^\top)\tilde V) then iDFT. Replaces cross-attention. (\sigma) = softmax or tanh.
• FEB-w / FEA-w: Legendre multiwavelets, recursive decompose (L=3); three FEB-f blocks per scale.
• Theory: random Fourier rows are RIP; keeping (s = O(k^2)) modes preserves a rank-(k) Fourier matrix. Do not keep only the lowest frequencies — events live in the high band too.
• Six benchmarks (ETT, Electricity, Exchange, Traffic, Weather, ILI). Input 96, horizons 96–720 (ILI 24–60). vs Autoformer: 14.8% multivariate / 22.6% univariate relative MSE drop. Fourier vs wavelet win on different sets — complementary.
• KS test: FEDformer is the only Transformer they quote whose forecast cannot be rejected as same-distribution as the 96-step input on all ETTm1/m2 horizons (p > 0.01).