Build and test your robots.txt before it goes live
Generate valid robots.txt rules, then test any URL path to confirm exactly what a crawler would do with it — before you publish.
* to target all crawlers, or name a specific one like Googlebot.Why Your robots.txt File Deserves Attention
Robots.txt is the first file most crawlers check before reading anything else on your site — and a single mistyped rule can accidentally hide your entire site from Google. A missing sitemap reference, an overly broad Disallow: /, or a blocked asset folder are quiet mistakes that can go unnoticed for months.
Getting robots.txt right isn’t complicated, but it does need to be checked against real URLs before it goes live — which is exactly what the tester below does.
How the Generator and Tester Work
Building and verifying your rules takes three steps.
Start From a Preset
Pick Allow All, WordPress Recommended, or Block Entire Site, then customize the rules underneath.
Add User-Agent Rules
Target all crawlers with * or a specific one like Googlebot, with its own Allow/Disallow rules.
Test Before You Publish
Enter any path and see exactly which rule would allow or block it — before it’s live on your site.
What Each Preset Actually Sets Up
Allow All Crawlers
Sets a single Allow: / rule for all user-agents — appropriate for most public-facing sites that want full indexing.
WordPress Recommended
Disallows /wp-admin/ while explicitly allowing /wp-admin/admin-ajax.php — the same pattern WordPress itself ships with by default.
Block Entire Site
Sets Disallow: / for everything — intended for staging sites and pre-launch environments, not live production sites.
Custom User-Agent Targeting
Add a second group naming a specific crawler like Googlebot to apply different rules than your general * group.
Robots.txt Mistakes Worth Avoiding
- Never block your theme or plugin asset folders. Disallowing CSS or JS files can prevent Google from rendering your pages correctly, which can hurt rankings.
- Double-check for a stray
Disallow: /. This single line, left over from a staging site, accidentally deindexes an entire live site more often than any other mistake. - Always include your sitemap URL. It’s a small addition that gives crawlers a direct path to your full site structure.
- Test the exact paths that matter most. Your homepage, key landing pages, and admin-ajax.php are worth checking individually before publishing.
- Remember robots.txt only controls crawling, not indexing. A page can still appear in search results if it’s linked elsewhere, even if it’s disallowed here — use a noindex tag for that instead.
Frequently Asked Questions
Where does robots.txt need to be placed?
At your site’s root — yoursite.com/robots.txt. A copy placed anywhere else, including a subfolder, is ignored by crawlers entirely.
Does robots.txt stop a page from appearing in Google?
Not reliably. Robots.txt controls crawling, not indexing — a disallowed page can still appear in search results if it’s linked from elsewhere. Use a noindex meta tag if you want a page excluded from search results specifically.
Why does the WordPress preset allow admin-ajax.php specifically?
Many WordPress themes and plugins rely on admin-ajax.php to load content dynamically. Blocking all of /wp-admin/ without this exception can prevent Google from properly rendering pages that depend on it.
How accurate is the URL tester compared to real Google crawling?
The tester implements Google’s documented matching algorithm — longest matching rule wins, with Allow breaking ties over Disallow. It’s a faithful implementation of the published rules, though it isn’t a live call to Google’s own systems.
Can I have different rules for different crawlers?
Yes — add a second user-agent group and name it specifically (like Googlebot or Bingbot) to give that crawler its own set of Allow and Disallow rules, separate from your general * group.
Is this tool free to use?
Yes — there’s no sign-up or account required. Build your rules, test any path, and copy or download the file directly.
The WPOptimizers Robots.txt Generator produces standard, spec-compliant syntax — always verify your live file after publishing.