BiomedCLIP
A biomedical CLIP trained on 15 million figure-caption pairs from the scientific literature.
About
BiomedCLIP (Zhang et al., NEJM AI 2024) is a vision-language foundation model for biomedicine, built the way CLIP was built for the general web: by contrastive pretraining on image-text pairs at scale. The team mined PubMed Central for PMC-15M, 15 million figure-caption pairs from 4.4 million open-access articles, two orders of magnitude larger than prior biomedical multimodal datasets. The model pairs a PubMedBERT text encoder with a ViT-B/16 image encoder, with domain-specific adaptations.
The authors report new state-of-the-art results across biomedical retrieval, classification, and visual question-answering benchmarks, and notably it beats radiology-specific models like BioViL on RSNA pneumonia detection despite being a generalist. The model is MIT-licensed on Hugging Face and loads through open_clip, and Microsoft released the PMC-15M data pipeline as open source. Figure captions from papers are a noisy training signal, and it is a research model, not validated for clinical use.
Specs
Context window
Pricing
Free quota
API access
Weights
Released
Benchmarks
Author-reported (Zhang et al., NEJM AI 2024): new state of the art across biomedical retrieval, classification, and VQA benchmarks; outperformed radiology-specific BioViL on RSNA pneumonia detection
Provider-reported unless a source says otherwise. Benchmarks are marketing-adjacent by nature — treat them as directional.
Use cases
Trust score — 8/10
Scored 8/10 (metrics tier, metrics-v1). Evidence: NEJM AI 2024 paper (arXiv 2303.00915), the Hugging Face model card (license: MIT), and the microsoft/biomedclip_data_pipeline repo confirm the PMC-15M dataset, the PubMedBERT+ViT architecture, and the benchmark claims. Strengths: largest open biomedical image-text resource, strong zero-shot results, permissive license, genuinely open access. Docked for: (1) figure-caption pairs from papers are a noisy training signal; (2) no hosted API or maintained product surface; (3) research model, not validated for clinical use. License: MIT.
How the trust score is built
Full methodology →- Adoption · stars/downloads, log-scaled vs peers
- 8.0/10
- Maintenance · recency, releases, issue load
- 6.0/10
- Openness · license + docs
- 9.0/10
- Manual review · sanity read + red-flag check
- 8.0/10
- · NEJM AI 2024; PMC-15M is the largest open biomedical image-text resource; strong zero-shot results.
- · Figure-caption pairs from papers are a noisy training signal.
- · Research model, not validated for clinical use; no hosted API.