BigBird

BigBird, introduced by a Google Research team led by Manzil Zaheer and Guru Guruganesh at NeurIPS 2020 (arXiv:2007.14062), extended Longformer's local-plus-global pattern with a third ingredient: random attention edges. The combination was not just empirical — the paper proved the sparse pattern is a universal approximator of sequence functions and Turing complete, preserving the theoretical properties of full attention while cutting the quadratic cost to linear.

Practically, BigBird handled sequences up to 8x longer than BERT on the same hardware, improved question answering and summarization, and pioneered attention-based modeling of genomics data such as DNA sequences and chromatin profiles. It remains a landmark in the efficient-transformer literature and a standard reference in the Long Range Arena benchmark. Checkpoints (bigbird-roberta-base and others) are openly available.