Introducing Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting for after-work quants

Takeaway. Vanilla Transformer attention is (O(L^2)), stacked layers blow memory, and autoregressive decode is as slow as an RNN on a 720-step horizon. Informer: (i) ProbSparse — only (u = c \ln L) queries with the most non-uniform attention keep full keys; (ii) distilling halves the time axis each layer; (iii) decoder emits the whole horizon in one forward.

Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting (Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, Wancai Zhang, 2020/2021; AAAI 2021; arXiv:2012.07436). Beihang / UC Berkeley / Rutgers. Code: zhouhaoyi/Informer2020. ETT data: zhouhaoyi/ETDataset.

Fit for learners: Sparse ProbAttention for long horizons — useful when your Python backtest window gets long and slow.

This Hub note is a plain-language intro. Not investment advice.

The problem

LSTF = forecast much longer than the old 48-step RNN papers (policy / capacity planning). LSTM MSE and latency collapse past ~48 points on their transformer-oil example. Full attention cannot eat a week of 15-minute bars on one V100.

The idea / core intuition

• Attention scores are long-tailed: a few query–key pairs dominate. Query sparsity (M(q_i,K) = \mathrm{LSE}(q_i K^\top/\sqrt{d}) - \mathrm{mean}(q_i K^\top/\sqrt{d})). Approximate with max−mean on (U = L \ln L) random pairs, keep Top-(u), (u = c\ln L_Q) (they use (c=5)).
• Other queries get the mean of (V) — cheap uniform fallback.
• Distilling: Conv1d (k=3) + ELU + max-pool stride 2 after each attention block. Pyramid replicas with (L, L/2, L/4) inputs concatenated.
• Decoder: concat a start token (earlier slice of the known series) with a zero placeholder of length (L_y); masked ProbSparse; one forward, not (L_y) steps. Avoids error accumulation.
• Input: scalar conv embed + local PE + learnable global stamps (minute/hour/week/month/holiday).
• Univariate/multivariate on ETTh1/h2, ETTm1, Weather, ECL vs LogTrans, Reformer, LSTMa, DeepAR, ARIMA, Prophet, LSTnet. Informer wins most long horizons (168–720 / 960). Canonical Informer† OOMs where ProbSparse still fits.

This is the 2021 LSTF Transformer that Autoformer / FEDformer / DLinear all argue with. Start here so the later papers’ tables make sense.