xLSTM

xLSTM (Extended Long Short-Term Memory) asks how far recurrent models can go when scaled with modern LLM techniques. From Sepp Hochreiter's team at NXAI and JKU Linz, it introduces exponential gating with normalization plus two new memory structures: sLSTM with scalar memory and memory mixing, and mLSTM with a fully parallelizable matrix memory and covariance update rule, stacked as residual xLSTM blocks (arXiv 2405.04517, May 2024).

The paper reports xLSTM performing favorably against state-of-the-art transformers and state space models on language modeling, both in performance and scaling. NXAI later trained xLSTM-7B, a 7-billion parameter recurrent LLM on 2.3 trillion tokens aimed at fast, efficient inference. The PyTorch implementation is Apache 2.0; note that ports like the JAX version use the separate NXAI Community License.