PLLaVA
Pooling LLaVA: the parameter-free trick for turning an image LLM into a video model — smooth visual features across time to kill dominant-token bias, and captioning plus video QA jump without training a video encoder.
About
PLLaVA (arXiv 2404.16994, April 2024) from researchers at NUS, NYU and ByteDance asks why naively fine-tuning image LLMs on video frames stalls or even hurts captioning. Their diagnosis: a few high-norm 'dominant' visual patches hijack the representation. The fix is almost embarrassingly simple — adaptive average pooling along the temporal dimension to smooth the feature distribution — and it needs no new parameters. With that single change, PLLaVA reported new state-of-the-art results on the video benchmarks of its day: 3.48 out of 5 averaged across the five VideoChatGPT dimensions (versus 3.17 for the prior GPT-4V-based best) and 58.1% on MVBench across 20 sub-tasks, 14.5 points above the GPT-4V baseline (all author-reported). The official repo is magic-research/PLLaVA with public checkpoints. The repo has no license file, so verify terms before commercial use. Later, stronger video models have passed it, but the pooling insight shows up across later work. No hosted API.
Specs
Context window
Pricing
Free quota
API access
Weights
Released
Benchmarks
- VideoChatGPT benchmark: 3.48/5 average across five dimensions (author-reported; prior SOTA 3.17)\n- MVBench: 58.1% average across 20 sub-tasks, +14.5 over GPT-4V (IG-VLM) baseline (author-reported)\n- Superseded by later video LLMs; all figures author-reported
Provider-reported unless a source says otherwise. Benchmarks are marketing-adjacent by nature — treat them as directional.
Use cases
Trust score — 7/10
Scored 7/10 (TIER verified, metrics-v1). Evidence: published paper with public code and checkpoints (670+ stars); the dominant-patch diagnosis and pooling fix are clearly explained and the idea recurs in later video models. Strengths: a genuinely useful, cheap recipe for video-adapting image LLMs with honest ablations. Docked for: (1) no license file in the repo, so commercial terms are unverified; (2) benchmarks are author-reported and the model is now dated; (3) repo activity stopped in mid-2024. License: none declared — verify before commercial use.
How the trust score is built
Full methodology →- Adoption · stars/downloads, log-scaled vs peers
- 7.0/10
- Maintenance · recency, releases, issue load
- 4.0/10
- Openness · license + docs
- 7.0/10
- Manual review · sanity read + red-flag check
- 7.0/10
- · Parameter-free temporal pooling to fix dominant-patch bias when video-adapting image LLMs.
- · VideoChatGPT 3.48/5, MVBench 58.1% at publication (author-reported).
- · Pooling insight recurs in later video models; repo inactive since mid-2024.