Worth knowing
Indexing and crawling are different things.
A robots.txt block can stop crawling, while noindex controls indexing. Check both when a page is mysteriously absent.
Website & SEO
Check a public URL for meta robots, Googlebot and X-Robots-Tag directives and get one clear answer.
Free to run. No account. Public URL checks are fetched only to return the result.
The annoying bit
A page can load perfectly in your browser while one quiet noindex directive tells search engines to leave it out. That is the kind of problem you usually notice after the traffic disappears.
Worth knowing
A robots.txt block can stop crawling, while noindex controls indexing. Check both when a page is mysteriously absent.
Next little job
Titles, redirects, canonicals, robots, sitemaps and schema all live in the same focused category.
See Website & SEO tools →Got another annoying job?
We’ll probably have a monkey for it. If we don’t yet, tell us what keeps wasting your time.