Worth knowing
Blocked from crawling does not automatically mean removed from Google.
robots.txt controls fetching. If you need a page out of search, use the appropriate noindex approach on a crawlable page instead of relying on robots.txt alone.
Free · No sign-up · PrivateWebsite & SEO
Test what robots.txt is actually allowing.
Fetch a site's robots.txt and test a path against Googlebot using the longest matching allow/disallow rule.
Free to run. No account. Public URL checks are fetched only to return the result.
The annoying bit
One line in a text file can quietly lock Google out.
robots.txt rarely looks dramatic. That is the problem. A broad Disallow can sit there doing exactly what it was told while everyone wonders why a whole section stopped being crawled.
Next little job
Need a different website check?
Titles, redirects, canonicals, robots, sitemaps and schema all live in the same focused category.
See Website & SEO tools →Got another annoying job?
What’s the annoying thing you need done?
We’ll probably have a monkey for it. If we don’t yet, tell us what keeps wasting your time.