What is the safest robots.txt example for most sites?

A safe starting point is a robots.txt file that allows all crawlers, blocks only sensitive folders such as admin or private areas, and includes a sitemap line.

Should I block CSS or JavaScript in robots.txt?

Usually no. Search engines often need CSS and JavaScript to understand how a page renders, so blocking those files can hurt crawling and indexing.

Can I use robots.txt to hide staging sites?

Robots.txt should not be your only protection for staging. Use authentication or IP restrictions first, then add a blocking robots.txt file as an extra signal.

Robots.txt Examples You Can Actually Use

These robots.txt examples are practical starting points for common site types. The goal is not to memorize syntax. It is to understand which patterns are safe, which are risky, and when you should use a stronger control than robots.txt.

Basic website example

For a normal brochure site, portfolio, or SaaS marketing site, keep the file simple. Allow crawling by default, block only areas that should not be explored, and include your sitemap.

User-agent: *
Disallow: /admin/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

This pattern is safer than trying to micro-manage every folder. Search engines usually do better when you block less and keep the rules obvious.

Staging or private environment example

For staging, you want a stronger barrier than robots.txt alone. Authentication is the real protection. Still, a fully blocked robots.txt file is a useful extra layer.

User-agent: *
Disallow: /

Do not rely on this by itself for anything sensitive. If the staging URL is public and linked somewhere, it can still leak or be indexed from outside references.

Ecommerce example

For ecommerce, the usual goal is to let product and category pages be crawled while reducing noise from account, cart, and checkout flows.

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /search

Sitemap: https://example.com/sitemap.xml

Be careful not to block product assets, CSS, or JavaScript that search engines need to render the page correctly.

AI bot blocking example

If you want to block AI crawlers, use separate user-agent groups so the rule is explicit. This is clearer than mixing them into one large generic block.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

This affects only bots that respect robots.txt. It is a preference signal, not a guaranteed legal or technical barrier.

Common mistakes to avoid

Blocking CSS or JavaScript that Google needs for rendering.
Assuming robots.txt prevents indexing in every case.
Using it as the only protection for staging or internal content.
Overcomplicating the file with dozens of narrow rules that are hard to maintain.

Build your own version

If you want a cleaner starting point than copying from blog posts, use the generator and choose a preset that matches your site. You can then add your own folders, sitemap lines, and AI bot rules before copying the final file.

Open the Robots.txt Generator

Request a Tool