TimeChat

TimeChat (arXiv 2312.02051, December 2023) tackles long-video understanding with two mechanisms. A sliding video Q-Former compresses long frame sequences without dropping the temporal order, and a timestamp-aware frame encoder binds each frame to its position in the video, so the model can say not just what happened but when. The same model handles three jobs: dense video captioning (YouCook2), highlight detection (QVHighlights) and temporal grounding (Charades-STA), where it reported 32.2 R@1 at IoU 0.5 zero-shot — a large jump over general video LLMs of the era, which mostly scored in the single digits on grounding (all author-reported). Its TimeIT instruction-tuning dataset and training recipe fed directly into the follow-up TimeSuite work from OpenGVLab. The repo (RenShuhuai-Andy/TimeChat) is BSD-3-Clause licensed with public checkpoints, and it is still pushed as of 2025. There is no hosted API.

• Charades-STA temporal grounding R@1 (IoU=0.5) 32.2, zero-shot (author-reported; vs ~3 for VideoChat-Embed 7B)\n- YouCook2 dense captioning: SODA_c 1.2, CIDEr 3.4, F1 12.6 (author-reported)\n- QVHighlights highlight detection: mAP 14.5, HIT@1 23.9 (author-reported)\n- Later grounding specialists now beat these numbers; all figures author-reported