BERT

BERT (Bidirectional Encoder Representations from Transformers) was introduced by Google researchers in October 2018 and released with open weights shortly after. Its masked-language-modeling and next-sentence-prediction objectives let it learn deep bidirectional context, and fine-tuning it set new state-of-the-art results on eleven NLP tasks including GLUE and SQuAD at launch.

BERT established the dominant NLP paradigm of the next several years: pre-train once on unlabeled text, fine-tune cheaply per task. The base (110M) and large (340M) variants, released under Apache 2.0, became the default starting point for thousands of production NLP systems and spawned an entire family of derivatives.