BLIP-2

BLIP-2, published by Salesforce Research in January 2023, made vision-language models cheap to train. Instead of training an image encoder and language model jointly from scratch, it freezes both a pretrained vision encoder and a pretrained LLM and trains only a lightweight Querying Transformer (Q-Former, ~188M parameters) between them in two stages: vision-language representation learning, then vision-to-language generative learning.

The result outperformed the 80B Flamingo by 8.7% on zero-shot VQAv2 with 54x fewer trainable parameters, and the modular design lets users swap the frozen LLM (OPT, Flan-T5, and Vicuna variants were released). Code and weights are available through Salesforce's LAVIS library. It is a landmark research model that shaped how later multimodal systems bolt vision onto existing language models.