VTimeLLM
One of the first LLMs built to grasp video moments: it answers 'when did this happen?' by predicting timestamps and descriptions in a single token stream, with a three-stage training recipe other grounding models still copy.
About
VTimeLLM (arXiv 2311.18445, November 2023) was among the first video LLMs designed specifically for temporal grounding — finding the exact segment of a video that matches a text query — rather than bolting timestamps onto a general video chatbot. The trick is representational: timestamps are discretized into special tokens in the LLM vocabulary, so the model emits event boundaries and captions as one continuous sequence. Training runs in three stages (feature alignment, instruction tuning, then a new LoRA module added for boundary-aware tuning while reusing the stage-2 LoRA), and the ablations show the staged approach is what makes grounding work. The 7B and 13B variants report ActivityNet temporal-grounding R@0.5 of 27.8 and 29.5 and Charades-STA R@0.5 of 27.5 and 34.3 (all author-reported). Later work (TimeChat, VTG-LLM, Time-R1) has beaten those numbers, but VTimeLLM's time-token idea became standard. Code is public at huangb23/VTimeLLM; there is no license file in the repo, so check terms before commercial use.
Specs
Context window
Pricing
Free quota
API access
Weights
Released
Benchmarks
- ActivityNet-Captions temporal grounding: R@0.5 27.8 (7B) / 29.5 (13B), mIoU 30.4 / 31.4 (author-reported)\n- Charades-STA: R@0.5 27.5 (7B) / 34.3 (13B), mIoU 31.2 / 34.6 (author-reported)\n- ActivityNet dense captioning CIDEr 27.6 (7B, author-reported)\n- Superseded by later grounding models (TimeChat, VTG-LLM, Time-R1) on the same benchmarks; all figures author-reported
Provider-reported unless a source says otherwise. Benchmarks are marketing-adjacent by nature — treat them as directional.
Use cases
Trust score — 7/10
Scored 7/10 (TIER verified, metrics-v1). Evidence: published method with public code and checkpoints; the time-token representation became the standard approach for LLM temporal grounding and is cited across the subfield. Strengths: genuinely pioneering for LLM-based grounding, clear staged-training ablations, reproducible recipe. Docked for: (1) no license file in the repo, so commercial terms are unverified; (2) benchmarks are author-reported and the model is now well behind the state of the art it helped create; (3) repo activity stopped in mid-2024. License: none declared — verify before commercial use.
How the trust score is built
Full methodology →- Adoption · stars/downloads, log-scaled vs peers
- 6.0/10
- Maintenance · recency, releases, issue load
- 4.0/10
- Openness · license + docs
- 7.0/10
- Manual review · sanity read + red-flag check
- 7.0/10
- · Among the first video LLMs for temporal grounding; the time-token representation became the subfield standard.
- · ActivityNet R@0.5 27.8 (7B)/29.5 (13B), Charades-STA R@0.5 27.5/34.3 (author-reported).
- · Repo activity stopped mid-2024; superseded by the models it inspired.