Screaming Frog
Screaming Frog is a desktop crawler that visits every URL it can find on your site and reports what it sees: titles, metas, status codes, redirects, duplicate content, missing alt text, broken links, orphan pages. It's the closest thing to seeing your site through a crawler's eyes without waiting for Google to tell you. Free for up to 500 URLs; the paid version is the workhorse of technical SEO audits.
Why It Matters
Screaming Frog is why technical SEO stopped being guesswork. Instead of 'we think there might be duplicate titles', you get a definitive list: here are the 47 pages with duplicate titles, the 12 with missing meta, the 6 with 404s being linked internally. Every issue is a named, locatable, fixable item.
It's also the early-warning system. Google finds problems on its own schedule - sometimes months later. Screaming Frog finds them now, which means you fix indexation, redirect and content issues before they cost rankings, not after.
In Practice
Crawl the site with the right configuration - include subdomains you care about, set the crawl to match your site's scale, and let it run fully. Export the issues that matter: 4xx and 5xx URLs, redirect chains, duplicate titles and metas, thin pages, orphaned content. Prioritise by impact: broken pages linked from money pages beat cosmetic issues every time.
Run it on a schedule, not as a one-off. The value is in the trend - a new redirect chain, a burst of 404s, pages going missing. The technical layer decays; the crawler is how you catch it.
Common Mistakes
Crawling once, exporting everything and doing nothing with it, or crawling a test environment and assuming it reflects production. The tool is only as useful as the action taken on its output.
Sources & Further Reading
Where to verify this yourself - Google's own documentation, industry reporting, and how we apply it at Underdog. Don't take our word for it; check the source.
Related Terms
Glossary
Crawling
How search engine bots discover and download your pages - the first step to ranking.
Glossary
Technical SEO
Optimising your site's infrastructure so search engines can crawl, render, and index it.
Glossary
Canonical Tag
Tells search engines which version of a page is the original when duplicates exist.
Glossary
Redirect (301 vs 302)
301 is permanent (transfers ranking signals), 302 is temporary - use the right one.
Glossary
Orphan Page
A page with no internal links pointing to it - invisible to search engines and users.
Know the Words.
Now See Them in Action.
Free teardown. No jargon. Just what's broken and how to fix it.
Get The Teardown