Free Tool
WordPress Robots.txt Generator
Free robots.txt generator for WordPress. Create a properly formatted robots.txt file with rules for Googlebot, Bingbot, and other crawlers. Block wp-admin, plugins, and sensitive directories.
Block /wp-admin/Recommended — block admin area
Block /wp-includes/WordPress core files
Block /wp-content/plugins/Plugin files
Block /xmlrpc.phpPrevent brute force attacks
Allow all bots (User-agent: *)Uncheck to only allow Google & Bing
What is robots.txt?
The robots.txt file tells search engine crawlers which pages and directories on your website they can and cannot access. It sits in your site's root directory (e.g., yoursite.com/robots.txt) and is the first file bots check before crawling.
For WordPress sites, a properly configured robots.txt prevents search engines from indexing admin pages, login screens, and other sensitive areas — improving both security and crawl efficiency.
WordPress Robots.txt Best Practices
- Block /wp-admin/ but allow /wp-admin/admin-ajax.php (needed by many plugins)
- Block /wp-includes/ — core files shouldn't be indexed
- Block search result pages (?s= queries) to prevent thin content indexing
- Include your sitemap URL — helps crawlers find all your important pages
- Block AI scrapers like GPTBot and CCBot if you don't want your content used for AI training