Crawlee
Apify's open-source web crawling library for Node.js and Python.
About
Crawlee is Apify's open-source crawling and browser-automation library, available for both Node.js/TypeScript and Python under Apache 2.0. It combines request queueing, proxy rotation, session management, and storage primitives with Playwright, Puppeteer, Cheerio, or plain HTTP crawlers, so you can build anything from a single-site scraper to a large whole-site crawl from one toolkit. A CLI scaffolds new crawlers with sensible defaults.
A community skill registry reports around 22,000 GitHub stars and roughly 72,000 weekly npm downloads. Because the same team builds Apify's hosted platform, a Crawlee scraper can be deployed to Apify as an actor later without rewriting. It is infrastructure rather than a turnkey LLM-extraction service: producing clean Markdown for RAG pipelines requires composing the extraction step yourself.
What it can do
- Queue-based crawling with retries, routing, and persistence
- Playwright, Puppeteer, Cheerio, and raw HTTP crawler types
- Structured dataset export and key-value storage
- Proxy rotation and session management built in
Setup
Install with npm install crawlee (plus playwright or puppeteer for browser crawling) or pip install crawlee. Scaffold with npx crawlee create my-crawler. Node.js 16+ required for the JS version.
Prerequisites
Node.js 16+ or Python 3.9+; Playwright browsers if doing browser crawling.
Use cases
Works with
Trust score — 8/10
Scored 8/10 (verified tier, metrics-v1). Evidence: Verified via vendor site/docs/GitHub and reputable coverage, wave 93; Trending score 78/100 from researcher assessment.
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.