DocFormer
AWS AI's end-to-end multimodal transformer for document understanding — text, vision, and spatial features fused with shared spatial embeddings.
About
DocFormer came out of AWS AI for the unglamorous but valuable problem of reading forms and receipts. Its core idea is a multi-modal self-attention layer that fuses text, vision, and spatial features, with learned spatial embeddings shared across modalities so the model can correlate a word with its visual region and back again. Pre-training is unsupervised, built around three tasks (multi-modal masked language modeling, image reconstruction, and text-image matching) designed to force the modalities to interact.
The ICCV 2021 paper reports state-of-the-art on four document-understanding datasets, in some cases beating models four times its size. That efficiency claim was the point: a compact multi-modal model that punches above its weight on visually-rich documents.
The catch is availability. AWS never released official weights; what exists are community re-implementations (one under MIT-0). Treat DocFormer as a paper plus a recipe rather than a downloadable model.
Specs
Context window
Pricing
Free quota
API access
Weights
Released
Benchmarks
- Author-reported (ICCV 2021): state-of-the-art on four document-understanding benchmarks covering forms and receipts tasks, sometimes beating models 4x its size (provider-claimed)
- No official weights were ever released, so no independent replication of the exact model exists
Provider-reported unless a source says otherwise. Benchmarks are marketing-adjacent by nature — treat them as directional.
Use cases
Trust score — 6/10
Scored 6/10 (TIER verified, metrics-v1). Evidence: peer-reviewed ICCV 2021 paper from AWS AI (Appalaraju et al.) with a clear multi-modal methodology and strong author-reported results. Strengths: peer-reviewed, influential architecture ideas (shared spatial embeddings), efficiency story. Docked for: (1) no official weights or maintained code — it is a paper plus community re-implementations; (2) all numbers are author-reported. License: no official release; community implementations carry their own licenses (e.g. MIT-0).
How the trust score is built
Full methodology →- Adoption · stars/downloads, log-scaled vs peers
- 6.0/10
- Maintenance · recency, releases, issue load
- 4.0/10
- Openness · license + docs
- 5.0/10
- Manual review · sanity read + red-flag check
- 7.0/10
- · Peer-reviewed ICCV 2021 paper from AWS AI (Appalaraju et al.): multi-modal self-attention with shared spatial embeddings across text, vision, layout.
- · Author-reported SOTA on four document-understanding benchmarks (forms/receipts), sometimes beating 4x-larger models.
- · AWS never released official weights; only community re-implementations exist — a paper plus a recipe, not a downloadable model.