Introducing MOMENT: A Family of Open Time-series Foundation Models for after-work quants

Takeaway. Most TS nets are fit on one dataset. MOMENT is a T5-style encoder pretrained with masked patches on a public mix (the Time Series Pile), then reused with a tiny head: linear probe for long-horizon forecast, frozen encoder + SVM for classification, zero-shot reconstruction for impute / anomaly.

MOMENT: A Family of Open Time-series Foundation Models (Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, Artur Dubrawski, 2024; ICML; arXiv:2402.03885). CMU Auton Lab / UPenn. Weights: AutonLab/MOMENT-1-large. Pile: AutonLab/Timeseries-PILE.

Fit for learners: Open TS foundation models — try fine-tuning on your HK/US return series instead of training from scratch.

This Hub note is a plain-language intro. Not investment advice.

The problem

No ImageNet for series: public TS sets are scattered, vary in rate / channels / length / amplitude, and often miss values. Multi-dataset pretraining was rare; LLM-reprogrammed forecasters (Time-LLM, GPT4TS) are heavy. Limited-supervision eval (zero-shot, linear probe) was thin.

The idea / core intuition

• Pile: Informer LTSF sets, Monash (~58 short-horizon archives), UCR/UEA classification, TSB-UAD anomalies — train splits only, seed 13, to limit leakage.
• Input: univariate length (T=512) (subsample or left-pad). RevIN. Split into (N=64) patches of length (P=8). Multivariate = channel-independent, batched.
• Mask 30% of patches with a learned [MASK] embedding (not zeros). Reconstruct with a light head; MSE on masked patches.
• Encoder: T5 mods (pre-LN, no bias, relative PE) plus sinusoidal absolute PE. Sizes: Small 40M / Base 125M / Large 385M. Random init (language init lost to random on their loss curves). 2 epochs, batch 2048, AdamW.
• Downstream: swap reconstruction head for a flatten→linear forecast head; keep reconstruction for impute/anomaly; freeze encoder, SVM on pooled embeddings for UCR.
• Long-horizon: linear-probed MOMENT sits next to PatchTST, often ahead of Time-LLM / GPT4TS / FEDformer on the TimesNet table they quote. Short-horizon zero-shot: statistical Theta/ETS still win many M3/M4 slices — be honest.
• Classification: without per-dataset fine-tune, mean accuracy 0.794 on 91 UCR sets < 512. Imputation: LP best MSE on ETT. Anomaly: LP adjusted F1 0.628 on UCR archive subset; (k)-NN still competitive on VUS-ROC.
• PCA of sine embeddings: the model sees trend / amplitude / frequency / phase; not vertical shift (RevIN). Cross-modal: frozen MOMENT does Bit Memory / MNIST / IMDb in the ballpark of GPT-2 / Flan-T5 of similar size.

Open weights you can pip after work. Chronos (paper 89) tokenizes like language; MOMENT reconstructs patches like MAE.

Why after-work Python learners care

Training a 385M transformer from scratch is not an evening. Downloading MOMENT-large and linear-probing 512-bar windows is. Compare it to a from-scratch PatchTST on one HK equity before you buy a bigger GPU story.

Tiny evening exercise: chunk a length-512 array into 64 patches of 8, zero 30% of them, reconstruct each patch by the mean of visible neighbours. Plot. That is the masked-modelling cartoon. Learning only.

One-evening reading checklist

1. Why patches of 8, context 512, channel independence.
2. Mask token vs fill-with-zero.
3. Linear probe vs end-to-end vs zero-shot reconstruction.
4. Time Series Pile split hygiene (no test leakage).

Download PDF

Download PDF

Landing: arXiv abs

Important disclaimer

Not investment advice. This Hub note is educational only. MOMENT’s public-benchmark scores are not a recommendation to fine-tune a foundation model on live orders.

What this is not

• Not CEF / subsidy framing
• Not Quant or Macro course CTA
• Not a trading-bot pitch

Next step

Single course CTA: SoloQuant Academy’s Introduction to Python Programming—functions, arrays, and file I/O are what you need before you load a Hugging Face weight file.