Vid2Seq

Vid2Seq (CVPR 2023, arXiv 2302.14115) from Google Research with Inria reframed dense video captioning. Instead of separate proposal and captioning stages, it augments a language model with special time tokens, so the model generates event boundaries and their textual descriptions in one output sequence. The key enabler is scale without annotation cost: pretraining reformulates transcribed-speech sentence boundaries from narrated videos as pseudo event boundaries, using the YT-Temporal-1B collection of millions of narrated videos, with generative plus denoising objectives. The resulting model set the state of the art on ActivityNet-Captions, YouCook2 and ViTT dense-captioning benchmarks at publication (author-reported), and also generalized to video paragraph captioning and few-shot settings. Code is Apache 2.0 in google-research/scenic; the README initially said checkpoints were coming, and later work confirms a public pretrained checkpoint exists and is reused by follow-up research. No hosted API.

• State of the art on ActivityNet-Captions, YouCook2 and ViTT dense event captioning at publication (author-reported)\n- Generalized to video paragraph captioning and few-shot settings (author-reported)\n- A later paper (Streaming Dense Video Captioning) confirms using the publicly released pretrained checkpoint\n- All original figures author-reported; the field has since moved to LLM-based captioners