Chameleon

Chameleon is Meta's family of early-fusion, token-based mixed-modal foundation models. Instead of bolting modality-specific encoders onto a language model, it represents images, text, and code as discrete tokens inside one unified transformer trained end-to-end from scratch on trillions of tokens of interleaved mixed-modal data, so it can reason over and generate arbitrary sequences mixing images and text.

The 7B and 34B models (arXiv 2405.09818, May 2024) beat Llama 2 on text-only tasks, competed with Mixtral 8x7B and Gemini Pro, and set state-of-the-art image captioning results at release, with human judges rating its mixed-modal generation at or above Gemini Pro and GPT-4V. Meta never released the weights, so Chameleon remains a research system and an architectural landmark for unified multimodal modeling rather than something you can download.