ALBERT

ALBERT (A Lite BERT) tackled BERT's parameter bloat with two techniques: factorized embedding parameterization, which decouples vocabulary size from hidden dimensions, and cross-layer parameter sharing, so all transformer layers reuse one parameter set. An ALBERT configuration matching BERT-large's shape carries about 18 times fewer parameters and trains roughly 1.7 times faster.

The paper (arXiv 1909.11942, September 2019; ICLR 2020) also replaced BERT's Next Sentence Prediction with the harder Sentence Order Prediction task for inter-sentence coherence. ALBERT-xxlarge set new state-of-the-art results on GLUE, SQuAD 1.1 and 2.0, and RACE at publication with fewer parameters than BERT-large. Code and checkpoints are Apache 2.0 via google-research/ALBERT.