AuroraCap

AuroraCap (ICLR 2025, arXiv 2410.03051) is a video captioner aimed at a gap in the field: most video-caption benchmarks reward one-sentence summaries, so models trained on them produce thin descriptions. AuroraCap is built on a deliberately simple recipe — a CLIP ViT-H visual encoder plus Vicuna-1.5-7B, with no extra temporal-modeling parameters — and handles long frame sequences with token merging, keeping only about 10% of visual tokens in the final training stage with little quality loss. It was trained in three stages on 20M+ image/video-text pairs, with the training datasets released on Hugging Face. Alongside the model, the authors released VDC, a benchmark of 1,000+ carefully structured human captions, and VDCscore, an LLM-assisted metric that breaks long-caption evaluation into short QA pairs and correlates better with human Elo rankings. On classic captioning benchmarks it reports a Flickr30k CIDEr of 88.9 and MSR-VTT CIDEr of 33.1 (all author-reported). There is no hosted API; you self-host the weights.

• Flickr30k CIDEr 88.9 (author-reported; vs GPT-4V 55.3, Gemini-1.5-Pro 82.2 on the same benchmark)\n- MSR-VTT CIDEr 33.1, VATEX CIDEr 33.8, zero-shot (author-reported)\n- Video QA: ActivityNet-QA accuracy 61.8, iVQA 55.2 (author-reported)\n- Introduced the VDC detailed-captioning benchmark (1,000+ structured human captions) plus the VDCscore metric, validated against human Elo rankings\n- No independent replication of any benchmark figure found