XLNet

XLNet is a generalized autoregressive pretraining method from CMU and Google Brain that overcame the limits of both BERT-style autoencoding and GPT-style autoregression. Its permutation language modeling objective trains the model to predict tokens in random factorization orders, learning bidirectional context without BERT's pretrain-finetune discrepancy from artificial mask tokens, while integrating Transformer-XL's segment recurrence and relative encoding for long context.

Published in 2019 (NeurIPS), XLNet outperformed BERT on 20 NLP tasks spanning question answering, natural language inference, sentiment analysis, and document ranking. The official implementation is open source; it remains a landmark in the pre-LLM era of pretraining objectives.