Generate a robots.txt file online with Allow, Disallow, sitemap, crawl-delay, AI bot blocking presets, and copy-ready examples.
Build a robots.txt file for Google, Bing, AI bots, and common CMS setups. Start from a preset, adjust the rules, then copy the final file.
This free robots.txt generator helps you create a valid robots.txt file without memorizing the Robots Exclusion Protocol. Add multiple user-agent groups, allow and disallow rules, crawl-delay values, and sitemap URLs, then copy the finished output instantly.
It is especially useful if you need a quick robots.txt example for WordPress, want to block AI crawlers such as GPTBot or ClaudeBot, or need a clean starting point for a new site before launch.
A robots.txt file tells well-behaved crawlers which paths they should or should not request. Search engines fetch the file before crawling your site and use the matching User-agent, Allow, and Disallow rules as guidance.
Not reliably. Robots.txt controls crawling, but a URL can still be indexed if Google finds it elsewhere. Use noindex or an X-Robots-Tag header if you need a page excluded from search results.
Add separate user-agent groups for bots such as GPTBot, ClaudeBot, Google-Extended, and CCBot, then set Disallow: / for each one. The preset in this tool builds that structure for you.
Upload it to the root of the site you want to control. For example, if your domain is example.com, the file must live at https://example.com/robots.txt.
Yes. The syntax is the same. What changes is which paths make sense to disallow. The supporting guides linked below cover common starting points for WordPress and similar platforms.