Scrapy
The open-source Python framework that turned web scraping from throwaway scripts into production pipelines.
About
Scrapy is an open-source web crawling and scraping framework for Python, maintained by Zyte with more than 500 contributors. It has been in production use for over 15 years and is the most widely used open-source data extraction framework in the world. Instead of a single script, Scrapy scaffolds a full project: spiders define what to crawl, item pipelines validate and store the data, and middlewares handle cookies, retries, user-agent rotation, and throttling. Feed exports write straight to JSON, CSV, XML, or S3, and an interactive shell lets you test CSS and XPath selectors against a live page before writing any code.
Scrapy runs on the Twisted async networking engine, so one process keeps many requests in flight, and its AutoThrottle adapts concurrency to how fast the target server responds. It does not render JavaScript natively; the community fills that gap with extensions like scrapy-playwright and scrapy-zyte-api. The scrapy.org homepage reports 64.8k GitHub stars, which is provider-stated rather than independently counted.
What it can do
- CSS and XPath selectors with regex helpers on every response
- Async engine with AutoThrottle and per-domain concurrency controls
- Duplicate-request filtering, automatic retries, and robots.txt handling
- Item pipelines for validation, cleaning, and storage
- Feed exports to JSON, CSV, XML, FTP, S3, and GCS
- Interactive scrapy shell for testing selectors live
- Extensible via middlewares, signals, and community extensions
Setup
pip install scrapy, then scrapy startproject to scaffold a project and scrapy crawl <spider> to run it.
Prerequisites
Python 3.10+; a target site that returns server-rendered HTML, or a browser-rendering extension for JavaScript-heavy pages.
Use cases
Works with
Trust score — 9/10
Scored 9/10 (metrics tier, metrics-v1). Evidence: scrapy.org and the scrapy/scrapy GitHub repo confirm the BSD-3-Clause license, the 15+ year history, 500+ contributors, and the feature set (AutoThrottle, item pipelines, feed exports); the 64.8k star figure is provider-stated on the homepage. Strengths: the reference open-source scraping framework, battle-tested at massive scale, huge extension ecosystem. Docked for: (1) no native JavaScript rendering, browser support comes from third-party extensions; (2) steeper learning curve than script-style libraries, overkill for small one-off jobs. License: BSD-3-Clause.
★ 64,800 GitHub stars
How the trust score is built
Full methodology →- Adoption · stars/downloads, log-scaled vs peers
- 10.0/10
- Maintenance · recency, releases, issue load
- 9.0/10
- Openness · license + docs
- 10.0/10
- Manual review · sanity read + red-flag check
- 9.0/10
- · Open-source Python crawling framework, in production 15+ years
- · scrapy.org reports 64.8k GitHub stars and 12k forks (provider-stated)
- · Async Twisted engine with AutoThrottle, item pipelines, feed exports to JSON/CSV/XML/S3
- · Interactive scrapy shell for testing selectors live
- · No native JavaScript rendering; covered by extensions like scrapy-playwright
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.