Kosmos-2

Kosmos-2, from Microsoft Research in June 2023, added explicit spatial grounding to multimodal large language models. Where earlier VLMs could describe an image but not point at what they meant, Kosmos-2 represents referring expressions as Markdown-style links pairing text spans with bounding-box location tokens, so it can both understand image regions you point to and answer with visual boxes.

It was trained on GRIT, a constructed dataset of roughly 91 million images with 137 million bounding boxes paired to referring expressions, alongside standard multimodal corpora. Microsoft released code and pretrained models, and Kosmos-2 became a foundation for referring-expression comprehension, phrase grounding, and grounded dialogue work. It was an early step toward the embodied-AI convergence of language, perception, and action.