Pix2Struct
Google's screenshot-parsing pre-training — learn to turn masked web pages into HTML, and document understanding comes along for free.
About
Pix2Struct's big idea is that the web is a free, infinite source of visually-situated language: every web page pairs a screenshot with its HTML structure. Pre-train an image-to-text model to parse masked screenshots into simplified HTML and, the authors argue, you subsume OCR, language modeling, and image captioning in one objective. Add a variable-resolution input so images are never awkwardly squished, and render text prompts directly onto the image instead of fusing modalities through extra machinery.
The ICML 2023 paper reports that a single pre-trained model reached state-of-the-art on six of nine tasks across four domains: documents, illustrations, user interfaces, and natural images. Google released fine-tuned variants for chart QA, document VQA, AI2D diagrams, and more on Hugging Face, and the model is supported natively in Transformers.
It later became the architectural base for DePlot, which says something about how well the idea held up.
Specs
Context window
Pricing
Free quota
API access
Weights
Released
Benchmarks
- Author-reported (ICML 2023): single pre-trained model reached SOTA on 6 of 9 tasks across documents, illustrations, UIs, and natural images (provider-claimed)
- Fine-tuned variants on Hugging Face (google/pix2struct-chartqa-base, docvqa, ai2d, widget-captioning, textcaps)
- Added to Hugging Face Transformers 2023-03-22; architectural base for DePlot
Provider-reported unless a source says otherwise. Benchmarks are marketing-adjacent by nature — treat them as directional.
Use cases
Trust score — 8/10
Scored 8/10 (TIER verified, metrics-v1). Evidence: peer-reviewed ICML 2023 paper from Google Research (Lee et al.) with a novel and well-motivated pre-training objective, multiple fine-tuned checkpoints on Hugging Face, and native Transformers support. Strengths: fully open, influential (DePlot builds on it), broad multi-domain results. Docked for: (1) headline numbers are author-reported; (2) maintenance is archival — the research line moved on. License: Google research release; weights freely available on Hugging Face.
How the trust score is built
Full methodology →- Adoption · stars/downloads, log-scaled vs peers
- 8.0/10
- Maintenance · recency, releases, issue load
- 6.0/10
- Openness · license + docs
- 9.0/10
- Manual review · sanity read + red-flag check
- 8.0/10
- · Peer-reviewed ICML 2023 paper from Google Research (Lee et al.): screenshot-parsing-as-pretraining (masked screenshots -> simplified HTML), variable-resolution input.
- · Multiple fine-tuned checkpoints on Hugging Face (chartqa, docvqa, ai2d, widget-captioning, textcaps); native Transformers support since 2023-03-22.
- · Author-reported SOTA on 6 of 9 tasks across 4 domains; influential (DePlot builds on it); maintenance archival.