Webflow Sitemap, Robots.txt and Staging Indexing Controls
Check Webflow staging and production separately, use current page and CMS indexing controls, and verify the published output.
Webflow's indexing settings and its published output should agree with your intention for each hostname and page. Do not assume a webflow.io staging site is permanently noindexed, or that editing an HTML tag automatically removes its URL from the sitemap.
The instructions below follow Webflow's current indexing documentation, checked September 28, 2026. UI labels and plan access can change. We did not operate a Webflow account for this guide.
Separate the controls
| Intention | Documented control | Published evidence to inspect |
|---|---|---|
| Limit staging discovery | Site settings → SEO → Indexing → Staging indexing off; save and publish | Staging robots.txt |
| Exclude one static page | Page settings → SEO → Sitemap indexing off; publish | Noindex tag and sitemap exclusion |
| Exclude one CMS item | Collection → item → Item details → Sitemap Indexing off; save/publish | That item's tag and sitemap entry |
| Change crawl access | robots.txt rules / traffic controls | Rules on the content hostname |
The documented staging switch requires a Site plan or paid Workspace. Per-item indexing controls require a paid Site plan. Check the current account's available controls before following older custom-code workarounds.
A robots disallow controls crawling; it is not the same as noindex. A crawler cannot read a page directive when access is blocked. Google explains this distinction.
If webflow.io appears instead of your custom domain
Make a paired inspection rather than changing every setting at once. This is an illustrative worksheet:
| Evidence | Staging hostname | Custom hostname |
|---|---|---|
| Intended role | Preview | Public site |
| Final URL/status | Record actual response | Record actual response |
| robots.txt | Record staging rule | Confirm intended production access |
| Page directives | Record actual tags/headers | Check for accidental noindex |
| Canonical | Record target | Confirm preferred production URL |
| Search Console evidence | What was indexed, if known | Selected canonical and crawl date |
Turn off staging indexing when that matches your policy and verify the published file. If the staging URL was previously indexed, a crawl block alone is not a reliable removal procedure. Follow the platform's supported controls and Google's removal guidance rather than assuming the switch immediately erases old results.
Reconcile sitemap entries with page settings
Open the actual sitemap and inspect the URLs you care about. A manually injected noindex or a robots block can leave a page in the auto-generated sitemap. Prefer the documented Sitemap indexing control when you need the platform to keep those signals aligned.
Use the sitemap diff on before/after snapshots to spot unexpected removals. The sitemap generator builds a supplied URL list; it does not configure Webflow or crawl your CMS.
For a failed Google fetch, follow the sitemap-fetch diagnosis. For pagination, keep separate pages with useful content distinct: see Webflow pagination examples.
Check after publication
- Compare the expected and actual hostname, redirects and canonical.
- Inspect the live robots response on each hostname separately.
- Check one allowed page and one intentionally excluded page.
- Check the sitemap after the same publish operation.
- Save dates and response evidence; do not promise a fixed Google update interval.
The noindex directive tool interprets copied directives. For a page that remains excluded after removal, use the post-noindex workflow. Neither a sitemap nor an allowed live test guarantees indexing.