Why does the tool warn that Googlebot will ignore my rules?

Because it will. A crawler follows only its single most specific matching group and never merges in the * rules. If you have a Googlebot group, Googlebot reads that one alone — so anything you disallowed in * no longer applies to it. Repeat the rules inside each named group.

Can robots.txt hide a page from Google?

No. Disallow stops crawling, not indexing — Google can still list a URL based on links to it, showing a bare result with no description. To keep a page out of results use a noindex meta tag, which means the page must stay crawlable.

Where does the file go?

At the root of the domain, exactly as https://yourdomain.com/robots.txt. It cannot live in a subfolder, and each subdomain needs its own — blog.example.com does not inherit example.com's file.

Should I block AI crawlers like GPTBot?

That is a business decision, not an SEO one. Blocking them keeps your content out of training data and out of AI answers that might cite you. Both effects are real and they point in opposite directions. The presets are here; the call is yours.

Does Google respect Crawl-delay?

No. Bing and Yandex do; Googlebot ignores it entirely. Crawl rate for Google is adjusted in Search Console. The tool flags this so you do not add a line believing it does something.

Robots.txt Generator

Builds a robots.txt and then checks it the way a crawler reads it — catching the group-specificity trap that silently switches off your rules for Googlebot.

Runs in your browser — nothing is uploaded

Rules for *

CrawlerBlock these paths (one per line)Allow these, despite the rules above

Sitemap

Full URL, one per line.

robots.txt

Save this as robots.txt at the root of your domain.

User-agent: *
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /*?s=
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap.xml

Checks

No problems found in this file.

The specificity check is the one worth reading twice. A crawler follows only its single most specific matching group and never merges in the * rules — so adding a Googlebot group silently switches off every restriction you wrote for everyone.

Why use it

Built to be genuinely useful

Catches the specificity trap

Adding a Googlebot group makes it ignore every rule in your * group. Almost no generator warns you.

Blocked assets flagged

Disallowing CSS or JS means Google renders your site unstyled and judges it on that.

Real matching rules

Longest match wins, * and $ supported — so the checks match how crawlers actually behave.

Nothing is uploaded

It runs in your browser. Your keyword list and your draft copy are a content plan — they should not have to leave the machine to be analysed.

How it works

Three steps

1
Choose a crawler and list the paths to block.
2
Add crawler-specific groups if you need them.
3
Read the checks, then save the file at your domain root.

The trap almost every robots.txt falls into

A crawler does not merge the rules that apply to it. It finds the single group whose user-agent line most specifically matches its name, follows that group, and ignores every other group in the file — including User-agent: *.

So a file that says "User-agent: * / Disallow: /admin/" and then adds "User-agent: Googlebot / Allow: /" does not do what its author intended.

The author blocked /admin/ for everyone and added a friendly note for Googlebot. What they actually did was exempt Googlebot from the /admin/ rule entirely, because the Googlebot group is more specific and Googlebot therefore never reads the * group at all.

This is the single most common robots.txt mistake, it produces no error anywhere, and the file looks correct to anyone reading it casually. This tool checks for it directly and tells you which paths a named crawler is no longer blocked from.

What robots.txt is not

It is not a security measure. The file is public — anyone can read yours right now — and listing /admin/ in it advertises exactly where your admin panel is. Well-behaved crawlers obey it; nothing else has to.

It is also not a way to keep a page out of search results. Disallow stops crawling, not indexing: Google can still list a URL it has never crawled, based on links pointing at it, showing a result with no description. To keep a page out of results you need a noindex meta tag — which requires the page to be crawlable, so it must not be disallowed here.

That contradiction catches people constantly. Blocking a page in robots.txt to hide it from Google makes it impossible for Google to see the noindex that would actually hide it.

What to block, and what never to

Worth blocking: internal search results pages, faceted-filter URLs that generate infinite combinations, cart and checkout, admin areas, and staging subdomains. These waste crawl budget on pages with no value in search.

Never block CSS, JavaScript or images. Google renders pages before judging them, and with the stylesheet blocked it sees an unstyled page and can conclude the site is unusable on mobile. This was a real and widespread problem when WordPress security guides recommended blocking /wp-content/, and the advice still circulates.

One line deserves its own warning: Disallow: / blocks your entire site. It arrives by accident more often than you would think, copied from a staging configuration during a launch, and it can remove a site from search entirely within days.

FAQ

Questions people ask

Related tools

More seo tools

Available tools are linked; the rest are in development.

Sitemap Generator Slug Generator AI Writer AI Summarizer AI Paraphraser AI Humanizer AI Grammar Checker AI Rewriter AI Translator AI PDF Summarizer Chat With PDF AI Document Analyzer

All SEO Tools

Need a tool like this for your business?

We build internal tools, dashboards and automation that fit how your team works.

See Our Services