Robots.txt Syntax: The 9 Mistakes That Silently Break Your Crawl
Robots.txt looks simple. Then a one-character typo deindexes half your site. Here are the syntax rules every dev should know.
Robots.txt is a plain text file. It has no schema validation, no error messages, and no warnings when you get it wrong. A single misplaced character can silently block Googlebot from crawling your entire site — or do the opposite and open up paths you meant to protect.
These are the 9 syntax mistakes that cause the most damage, how each one plays out, and how to test before you ship.
The technical SEO guides hub has the broader context if you want to understand how robots.txt fits into crawl control alongside noindex and canonicals.
Mistake 1: Case-Sensitive Paths
Robots.txt paths are case-sensitive on case-sensitive filesystems. If your server runs on Linux (most web servers do), /Admin/ and /admin/ are different directories.
Wrong:
User-agent: *
Disallow: /Admin/
If your admin panel is at /admin/ (lowercase), this rule does nothing. Googlebot can crawl /admin/ freely.
Right:
User-agent: *
Disallow: /admin/
Match the exact case of your URL paths. If you have both /Admin/ and /admin/ in use (a Windows-to-Linux migration can cause this), list both.
Mistake 2: Missing Blank Line Between User-Agent Groups
Each group of directives must start with one or more User-agent: lines, followed by Allow: and Disallow: directives, and separated from the next group by a blank line.
Wrong — the two groups merge into one:
User-agent: Googlebot
Disallow: /private/
User-agent: *
Disallow: /staging/
Most parsers interpret this as a single group applying to both Googlebot and *. The directives get merged, and the behavior may not match your intent.
Right:
User-agent: Googlebot
Disallow: /private/
User-agent: *
Disallow: /staging/
One blank line between groups. Always.
Mistake 3: Wildcard Misuse — * and $
Robots.txt supports two wildcard characters in path patterns:
*matches any sequence of characters$anchors the match to the end of the URL
Both are supported by Google and Bing, but not by all crawlers. The syntax looks like regex but isn't — it's a simplified pattern syntax.
Common misuse: * in the User-agent line vs. the path line
# This is correct — * in User-agent means "all bots"
User-agent: *
Disallow: /tmp/
# This is also correct — * in the path acts as a wildcard
User-agent: *
Disallow: /*.pdf$
Wrong — confusing the $ anchor:
User-agent: *
Disallow: /page$
This blocks exactly /page and nothing else. If you want to block all pages matching a pattern and their variations, you probably want /page* or more specific patterns.
Wrong — using .* like a regex:
User-agent: *
Disallow: /search.*q=
This works in Google's implementation but is not standard per the original spec. Bing may interpret it differently. If cross-bot consistency matters, stick to simpler patterns.
Practical pattern examples:
Disallow: /*.pdf$ # All URLs ending in .pdf
Disallow: /search?* # All search URLs with any query string
Disallow: /category/*/page # All paginated category pages
Mistake 4: Allow vs. Disallow Precedence (Longest Match Wins)
When Allow and Disallow rules conflict for the same URL, the most specific rule wins — the one with the longest matching path. This is Google's behavior. Bing uses last-match-wins (see Mistake 8).
Example: allowing a file in a disallowed directory
User-agent: *
Disallow: /private/
Allow: /private/public-file.pdf
For /private/public-file.pdf: the Allow rule has a longer match (/private/public-file.pdf = 24 chars) than the Disallow rule (/private/ = 9 chars). Google honors the Allow. The file is crawlable.
For /private/other-file.pdf: only the Disallow rule matches. Not crawlable.
This is intentional and useful — it lets you carve exceptions into disallowed directories. But it bites you when you don't expect it:
Common footgun:
User-agent: *
Disallow: /
Allow: /public/
This correctly allows /public/ while blocking everything else. But if /public/ contains a link to /public/sitemap.xml and you later add Disallow: /public/*.xml$ — that more specific rule overrides the broader Allow.
Mistake 5: BOM at the Start of the File
A Byte Order Mark (BOM) is a hidden character that some Windows text editors (Notepad, older versions of VS Code with specific settings) prepend to UTF-8 files. It's invisible in most editors but present in the file bytes.
If your robots.txt has a BOM, Google's parser may fail to recognize the first User-agent: directive, treating the file as invalid or ignoring the first block entirely.
How to check:
xxd robots.txt | head -1
If the first bytes are ef bb bf, that's a UTF-8 BOM. Strip it.
How to fix:
sed -i '1s/^\xEF\xBB\xBF//' robots.txt
Or save the file from your editor with "UTF-8 without BOM" encoding.
Mistake 6: Comments on Directive Lines
The # character starts a comment in robots.txt. Comments must be on their own line or at the end of a line with a space. Robots.txt parsers handle inline comments inconsistently — some parsers include the comment text as part of the directive value.
Wrong — comment appended without a space:
Disallow: /admin/#this is the admin section
Some parsers will try to match the literal path /admin/#this is the admin section instead of /admin/.
Right — comment on a separate line:
# Admin section - keep blocked
Disallow: /admin/
Also acceptable:
Disallow: /admin/ # Admin section
But the safest habit is comments on their own lines, full stop.
Mistake 7: Conflicting Directives for * vs. Specific Bots
When you have both a * (all bots) group and a named bot group (e.g., Googlebot), a named bot follows only its specific group and ignores the * group entirely.
User-agent: *
Disallow: /private/
Disallow: /admin/
User-agent: Googlebot
Disallow: /staging/
In this configuration, Googlebot follows only the second group. It can crawl /private/ and /admin/ freely — those rules are in the * group, which Googlebot ignores because it has its own group.
Fix: duplicate rules into the specific group if needed, or remove the named group and use * only.
User-agent: Googlebot
Disallow: /private/
Disallow: /admin/
Disallow: /staging/
This is a significant and common mistake on sites that add bot-specific rules without realizing they've effectively exempted that bot from the general rules.
Mistake 8: Bing vs. Google Conflict Handling
Google uses longest-match-wins for conflicting Allow and Disallow rules. Bing uses last-match-wins (or first-match in some documented versions — Bing's behavior here has been inconsistent). This matters when you have overlapping rules.
User-agent: *
Disallow: /category/
Allow: /category/featured/
Google: /category/featured/ is allowed (longer match wins).
Bing: Result depends on rule order and Bing's current parser version.
If Bing traffic matters to you, prefer non-conflicting rule structures: either use one comprehensive Allow/Disallow pattern, or test explicitly with Bing Webmaster Tools' robots.txt tester.
Mistake 9: Missing Sitemap Line
Robots.txt supports a Sitemap: directive that tells crawlers where your sitemap lives. This isn't just optional housekeeping — it's one of the most reliable ways to ensure your sitemap is discovered and submitted to Google before you ever touch Search Console.
Wrong — no Sitemap directive:
User-agent: *
Disallow: /checkout/
Disallow: /account/
Right:
User-agent: *
Disallow: /checkout/
Disallow: /account/
Sitemap: https://www.example.com/sitemap.xml
If you're using a sitemap index file, point to the index:
Sitemap: https://www.example.com/sitemap-index.xml
You can list multiple Sitemap: lines — one per sitemap or index file. The Sitemap: directive is not tied to any User-agent block; it's global.
How to Test Before You Ship
Google's robots.txt Tester (GSC)
In Google Search Console: Settings → robots.txt. Paste your robots.txt and test specific URLs to see whether Googlebot can or can't access them. Shows the exact rule that applied.
Bing Webmaster Tools
Bing has its own robots.txt tester under Diagnostics & Tools → robots.txt Tester. Test critical paths here too if Bing traffic matters.
curl
curl -A "Googlebot" https://yoursite.com/robots.txt
Fetches your live robots.txt exactly as Googlebot would see it. Useful for confirming the live file matches your local version and has no BOM or encoding issues.
robots.txt validator
The robots.txt generator and tester validates your file structure and lets you test URL patterns against your rules before deploying.
One Final Rule: Don't Block Your Own CSS and JS
A legacy mistake from the Googlebot-Smartphone era: blocking /wp-content/ or /assets/ to "save crawl budget." This prevents Google from rendering your pages correctly, which degrades how it understands and ranks your content.
# Don't do this:
User-agent: *
Disallow: /wp-content/
Disallow: /assets/
Google needs to fetch your CSS and JS to render the page. If those paths are blocked, Google sees a broken, unstyled page and may rank it accordingly.
Crawl budget concerns are real for very large sites, but blocking render-critical resources is not the fix. See the noindex vs. disallow guide for a more targeted approach to crawl control.
Summary
- Paths are case-sensitive — match the exact case of your URLs.
- Blank lines separate groups — missing them merges groups unintentionally.
*and$are pattern wildcards, not regex — use them carefully.- Longest match wins in Google; Bing behavior differs.
- Strip BOM from your file — invisible characters break parsers.
- Keep comments on separate lines to avoid parser ambiguity.
- Named bot groups override
*— named bots ignore the wildcard group entirely. - Test on both Google and Bing if both matter to you.
- Add a
Sitemap:line — don't rely on GSC submission alone.