BART

BART is a denoising autoencoder for pretraining sequence-to-sequence models from Facebook AI. It corrupts input text with a noising function, shuffling sentence order and replacing spans with mask tokens, then trains a standard transformer encoder-decoder to reconstruct the original. The framework generalizes BERT through its bidirectional encoder, GPT through its left-to-right decoder, and several other pretraining schemes.

Published October 29, 2019 (arXiv 1910.13461), BART matched RoBERTa on GLUE and SQuAD with comparable resources and set new state-of-the-art results on abstractive dialogue, question answering, and summarization, with gains up to 6 ROUGE. It became the workhorse backbone for summarization fine-tunes, and its checkpoints are available under Apache 2.0 via fairseq and Hugging Face.