trafilatura
Extract clean text from the web: 13M downloads a month.
About
trafilatura is the standard Python library for extracting main text from web pages: precise boilerplate removal, metadata, and language detection — the invisible workhorse behind countless scraping, corpus-building, and RAG-ingestion pipelines. 7k stars (it understates: it's embedded everywhere), Apache-2.0, 12.9M PyPI downloads/month. It extracts from fetched HTML; pair it with a crawler (crawl4ai, Firecrawl) or requests for the full pipeline.
What it can do
- **Precise extraction**: main content, not nav/ads
- **Metadata**: title, author, date, language
- **Deduplication**: built-in near-duplicate detection
- **Formats**: text, XML, JSON output
- **Fast**: efficient enough for large crawls
Setup
```bash
pip install trafilatura
```
```python
import trafilatura
html = trafilatura.fetch_url("https://example.com/article")
text = trafilatura.extract(html)
```
Docs: trafilatura.readthedocs.io.
Prerequisites
Python 3.9+.
Use cases
Works with
Trust score — 8/10
Scored 8/10 (verified tier, metrics-v1). Evidence: 6,962 GitHub stars (log-scaled vs catalog peers); blended with ~12,938,590 monthly package downloads; last push 6d ago; 4 releases/12mo; 75 open issues. Manual review: web-text extraction standard (12.9M PyPI/mo); extraction only — needs a fetcher; star count understates embedded use. License: Apache-2.0.
★ 6,962 GitHub stars
How the trust score is built
Full methodology →- Adoption · stars/downloads, log-scaled vs peers
- 8.1/10
- Maintenance · recency, releases, issue load
- 8.4/10
- Openness · license + docs
- 10.0/10
- Manual review · sanity read + red-flag check
- 8.0/10
- · 6,962 GitHub stars (log-scaled vs catalog peers)
- · blended with ~12,938,590 monthly package downloads
- · last push 6d ago; 4 releases/12mo; 75 open issues
- · manual review: web-text extraction standard (12.9M PyPI/mo); extraction only — needs a fetcher; star count understates embedded use
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.