The three rules people get wrong
The first is precedence. Rules are not applied in order — the most specific one wins, and specificity means the length of the path pattern. So `Disallow: /admin/` beats `Allow: /` no matter which appears first in the file, and moving lines around changes nothing. On an exact tie in length, Allow wins.
The second is groups. A crawler obeys exactly one group: the one whose User-agent matches it most specifically, and it ignores every other group in the file entirely. If you have a Googlebot group and a `*` group, Googlebot reads only the Googlebot group — even if that group is empty and the `*` group blocks everything. This surprises people constantly.
The third is the trailing slash. `Disallow: /admin` blocks `/admin`, `/admin/`, `/administrator` and `/admin-panel`, because robots patterns are prefixes rather than paths. `Disallow: /admin/` blocks only what is inside the directory. One character, and a considerable difference in what disappears.
Crawling is not indexing
This is the single most consequential misunderstanding about robots.txt. Blocking a URL stops crawlers fetching it. It does not stop the URL appearing in search results — if other pages link to it, it can be indexed on the strength of those links alone, and it appears with no description because nothing was ever read.
Worse, a page blocked in robots.txt cannot be de-indexed by a noindex tag, because the crawler is not permitted to fetch the page and read the tag. The correct sequence for removing something from results is to allow crawling, add noindex, wait for it to be recrawled and dropped, and only then block it if you want to.
Note also that `Noindex:` as a robots.txt directive has never been supported by Google and was explicitly dropped in 2019. If a file contains one, it is doing nothing, and this tool says so.
What a missing file means, and what a broken one means
A 404 for robots.txt is completely fine. It means there are no restrictions, and every crawler may fetch everything. Plenty of sites need nothing more than that, and a tool that reports a missing file as an error sends people looking for a problem that does not exist.
A 5xx is a different matter, and it is the one genuine emergency in this area. Google's documented behaviour is that a robots.txt returning a server error is treated as though the entire site were disallowed, because the crawler cannot establish what it is permitted to fetch. A file that intermittently 500s can quietly suppress crawling of a whole site.
The other failure worth watching for is a staging robots.txt reaching production. `User-agent: *` with `Disallow: /` is correct on a staging server and catastrophic on a live one, and it is among the most common causes of a site vanishing from search results after a deploy.