UDOP
Microsoft's unified document foundation model — one Vision-Text-Layout Transformer for understanding and even generating document images.
About
UDOP (Universal Document Processing) unifies text, image, and layout in a single Transformer and, unusually, unifies the tasks too: document understanding and document generation run through the same prompt-based sequence generation scheme. It can answer questions about a document and it can also reconstruct or edit document images from text and layout, via masked image reconstruction with a vision decoder — the first Document AI model to do both at quality, per the authors.
The architecture is a T5-large-based encoder-decoder (794M parameters total) trained with self-supervised objectives on large unlabeled corpora plus supervised tasks like classification, layout analysis, information extraction, QA, and document NLI. The paper (arXiv 2212.02623, CVPR 2023) claims state-of-the-art on 8 Document AI tasks and first place on the Document Understanding Benchmark leaderboard.
It needs OCR output (words plus boxes) as input rather than reading pixels alone, which dates the design, and the headline results are author-reported. Checkpoints are on Hugging Face as microsoft/udop-large variants.
Specs
Context window
Pricing
Free quota
API access
Weights
Released
Benchmarks
- Author-reported (arXiv 2212.02623, CVPR 2023): state-of-the-art on 8 Document AI tasks; first on the Document Understanding Benchmark leaderboard
- 794M parameters; masked image reconstruction enables neural document editing
- No independent replication published — all headline numbers are author-reported
Provider-reported unless a source says otherwise. Benchmarks are marketing-adjacent by nature — treat them as directional.
Use cases
Trust score — 7/10
Scored 7/10 (TIER verified, metrics-v1). Evidence: peer-reviewed CVPR 2023 paper (arXiv 2212.02623) from Microsoft with named authors, and released checkpoints on Hugging Face. Strengths: ambitious unified understanding-plus-generation design, open weights, peer-reviewed. Docked for: (1) all headline numbers are author-reported with no independent replication; (2) OCR-dependent input pipeline dates it and maintenance is archival. License: Microsoft research release; no explicit license field on the HF card — verify before commercial use.
How the trust score is built
Full methodology →- Adoption · stars/downloads, log-scaled vs peers
- 7.0/10
- Maintenance · recency, releases, issue load
- 5.0/10
- Openness · license + docs
- 8.0/10
- Manual review · sanity read + red-flag check
- 8.0/10
- · Peer-reviewed CVPR 2023 paper (arXiv 2212.02623) from Microsoft: unified Vision-Text-Layout Transformer for document understanding and generation (794M params, T5-large-based).
- · Released checkpoints on Hugging Face (microsoft/udop-large-512 variants).
- · Author-reported SOTA on 8 DocAI tasks and first on the DUE leaderboard; OCR-dependent input dates it; no independent replication.