Screaming Frog SEO Spider
Desktop crawler for technical SEO audits
About
A desktop website crawler (Windows, macOS, Linux) that audits technical and on-page SEO in real time: it crawls live pages to find broken links, server errors, redirects, duplicate content, missing or duplicate metadata, and robots.txt issues, then exports bulk data for analysis. The free tier crawls up to 500 URLs; the paid licence removes the limit (bounded only by local hardware) and adds JavaScript rendering, custom extraction via XPath/CSS/regex, and integrations with Google Analytics, Search Console, and PageSpeed Insights. It is a practitioner staple, not an all-in-one suite — there is no keyword research, rank tracking, or backlink index.
What it can do
- desktop crawler (live-page crawling)
- broken link, error and redirect detection
- metadata and duplicate-content analysis
- javascript rendering (paid)
- custom extraction with xpath, css, regex (paid)
- xml sitemap generation
- ga, search console, pagespeed integrations (paid)
- crawl scheduling (paid)
Setup
Download and install the desktop app; point it at a URL and start the crawl. No account required for the free tier.
Prerequisites
A desktop OS (Windows, macOS, or Linux); sufficient RAM/storage for large crawls.
Use cases
Works with
Trust score — 8/10
Scored 8/10 (VERIFIED tier, metrics-v1). Evidence: long-running UK company, desktop tool at v24.x in 2026, consistently cited as the standard technical crawler across independent reviews. Strengths: one-time-per-year licence (£199/$279, or free up to 500 URLs), transparent scope, no lock-in. Docked for: (1) closed-source desktop product — correctness of crawl data is provider-claimed, no public methodology; (2) no keyword, rank, or backlink modules, so it is a single-purpose instrument. License: commercial, proprietary.
How the trust score is built
Full methodology →- Adoption · stars/downloads, log-scaled vs peers
- 8.0/10
- Maintenance · recency, releases, issue load
- 9.0/10
- Openness · license + docs
- 3.0/10
- Manual review · sanity read + red-flag check
- 7.0/10
- · desktop crawler, no cloud lock-in
- · closed-source crawl logic, provider-claimed capabilities
- · no published independent benchmarks
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.