You have just moved a staging site to production, and the first thing worth checking is whether the “Disallow: /” line from testing is still sitting in robots.txt. These five prompts cover writing the file, auditing it, and the mistakes that only show up after launch.
What’s on this page
- A baseline file for a normal content site.
- A block list for admin paths and internal search.
- An audit prompt for the robots.txt you already have.
- A decision prompt for AI crawlers, based on what you want to happen.
- A pre-launch checklist for staging-to-live moves.
The prompts
1. Baseline File
Write a robots.txt file for [site type] at [domain]. Allow normal crawling of content pages, reference the sitemap location, and add a brief comment above each group explaining what it does. Do not block anything that would stop content pages from being indexed.What it does: Produces a readable starting file instead of a copy-pasted block nobody understands.
How to use: Replace [site type] and [domain]. Keep the comments in — future you will need them.
2. Block Admin and Search Internals
Extend this robots.txt to block the paths a content site should not have indexed: admin and login paths, cart or checkout paths if relevant, internal search result URLs, and any filter or parameter URLs that generate duplicate pages. Group them by purpose and explain each group in one line. Current file: [paste file].What it does: Adds the practical exclusions that prevent thousands of near-duplicate URLs from being crawled.
How to use: Paste your current file, then verify each path exists on your site before you block it.
3. Existing File Audit
Audit this robots.txt: [paste file] against this site structure: [describe your main URL patterns]. Flag: rules that block content I probably want indexed, rules that conflict with each other, rules that do nothing, and anything that would stop a crawler from reaching the sitemap.What it does: Finds the accidental blocks that quietly remove pages from search results.
How to use: Describe your URL patterns honestly — the audit is only as good as that description.
4. AI Crawler Decision
I need to decide whether to allow AI training and AI answer crawlers on [domain]. List the questions I should answer first, then write two example robots.txt groups — one that allows them and one that blocks them — using placeholder user-agent names that I will replace with current strings from each provider's documentation.What it does: Gives you both versions of the rule so the decision stays yours, with the technical part done.
How to use: Replace the placeholder user-agent names with the exact strings published by each provider — they change, so check the source rather than a blog post.
5. Staging to Live Checklist
Write a robots.txt and indexing checklist for moving a site from staging to production at [domain]. Cover: what to remove, what to add, what to verify after launch, and which settings outside robots.txt also control indexing.What it does: Catches the launch-day mistakes that keep a finished site out of search results.
How to use: Run it the day before launch and again the day after. Some of the checks involve settings in your CMS, in your host panel and in search console.
How to adapt these prompts
- For an e-commerce site, extend prompt 2 with faceted navigation parameters and sorting URLs.
- If you run a multilingual site, add “list the language or region path patterns to allow” to prompt 1.
- Robots.txt only controls crawling, not indexing. If a page must stay out of results, ask prompt 3 for the page-level controls as well.
FAQ
Does a blocked page disappear from search results?
Not reliably. Blocking crawling can still leave the URL indexed without a description, because the crawler cannot see a noindex directive it is not allowed to fetch. Use the page-level control for pages that must not appear.
Where does robots.txt have to live?
At the root of the domain, at /robots.txt. A copy inside a subfolder is ignored by crawlers looking for the root file.
Should I reference the sitemap in robots.txt?
Yes. It is a cheap way to make the sitemap discoverable, and it does not replace submitting it in search console.
Keep going
These prompts are free to use. Copy them, tweak the [variables], ship something.