Robots.txt generator + tester

What robots.txt does

robots.txt is a plain text file at the root of your domain that tells search engine crawlers which paths they may and may not request. It's the first file Googlebot looks for on a site, and a few lines in it can steer how your entire crawl budget is spent.

The key distinction — the one that trips up most people — is that robots.txt controls crawling, not indexing. Disallowing a URL stops bots from fetching it, but if other pages link to that URL it can still appear in search results, just without a description. To keep a page out of the index entirely, allow crawling and use a noindex meta tag instead.

How to build and test your robots.txt

This generator builds the file from a form and includes a path tester so you can verify a rule before it goes live — every URL on your site is one typo away from being blocked.

• Add a User-agent block — '*' for all crawlers, or a name like Googlebot for one.
• List Disallow paths for sections bots should skip — admin pages, carts, internal search.
• Add Allow rules to carve exceptions out of a broader Disallow.
• Include the full URL of your XML sitemap so crawlers can find it.
• Use the path tester to confirm important URLs still resolve as Allowed.

Common robots.txt mistakes

The most expensive mistake is 'Disallow: /' left in place after a site launch — it blocks the entire site from being crawled, and traffic disappears within days. Always test the homepage and a few key URLs after any change.

Other frequent errors: blocking your CSS or JavaScript folders, which stops Google from rendering the page as a user sees it; using robots.txt to hide pages that should instead carry a noindex tag; and forgetting that each subdomain needs its own file. Keep the file minimal, deliberate, and tested.