Introducing EA-LSTM: Evolutionary Attention-based LSTM for Time Series Prediction for after-work quants

Takeaway. Vanilla LSTM forgets which lags matter. EA-LSTM multiplies the sliding window by a per-timestep attention vector (W = (W^1,\ldots,W^L)), then feeds the weighted series to LSTM gates. Those (W) are not trained with Adam: a competitive random search (binary encode → champion subset → random select / recombine / mutate) ranks candidates by validation error.

EA-LSTM: Evolutionary Attention-based LSTM for Time Series Prediction (Youru Li, Zhenfeng Zhu, Deqiang Kong, Hua Han, Yao Zhao, 2018; arXiv:1811.03760). Beijing Jiaotong University / Microsoft Multimedia / CAS NLPR.

Fit for learners: Open-PDF classic/working paper on time-series — useful for after-work Python quant learners building intuition before coding.

This Hub note is a plain-language intro. Not investment advice.

The problem

LSTM already handles long memory with forget/input/output gates. What it still does poorly is importance sampling inside one window: lag (t-3) and lag (t-18) get the same first-layer treatment. Gradient attention (DA-RNN, Qin 2017) can stick in a local basin. The authors want a global search for the attention vector.

The idea / core intuition

• Window (X_t = (x_t^1,\ldots,x_t^L)). Weighted input (\tilde X_t = (x_t^1 W^1,\ldots,x_t^L W^L)). Then standard LSTM: (i^t, f^t, c^t, o^t, h^t).
• LSTM weights: mini-batch SGD + Adam (BPTT). Attention (W): CRS population size (N=36), 6-bit encoding (weights in (\approx 0.016)–(1)), champion subset size 6, (T=20) epochs.
• Transfer: CRS writes (W) into the attention layer; validation loss ranks the next generation.
• Ablation: “Attention-LSTM” (same layer, gradient on (W)) is worse than CRS on all three sets.
• Regression: Beijing PM2.5 (hourly 2010–2014, 8 sensors; (L=18), 128 units, batch 256) MAE 0.190 / RMSE 0.276 vs LSTM 0.617 / 0.703 and DA-RNN (SML only). SML2010 indoor temperature (16 sensors, (L=24)) MAE 0.010 / RMSE 0.015 vs DA-RNN 0.015 / 0.020.
• Classification: MSR Action3D skeleton — EA-LSTM 95.20% vs HBRNN 94.49% and LSTM 90.67%.
• Heatmaps: learned (W) is not uniform across the window — local structure inside one sample.

A 2018 LSTM + evolutionary attention paper, not a 2022 LTSF Transformer. Useful as the “why not just backprop the attention?” control before Informer / FEDformer.