Should wp-admin be blocked in robots.txt?

Usually yes, but the admin-ajax.php endpoint is often left allowed because plugins and front-end features may rely on it.

Should wp-content be blocked in robots.txt?

Usually no. Blanket blocking wp-content can hide images, CSS, JavaScript, and other assets that search engines need to understand the page.

Does WordPress need a sitemap line in robots.txt?

It helps. Modern SEO plugins and WordPress setups commonly expose an XML sitemap, and adding it to robots.txt makes discovery easier.

WordPress Robots.txt Guide

A good WordPress robots.txt file is mostly about restraint. The safest setup is usually short, readable, and focused on preventing crawl waste without blocking important front-end assets.

Safe starting example

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap_index.xml

For many WordPress sites, that is enough. It keeps bots away from the admin area while preserving the AJAX endpoint that themes and plugins sometimes need on public pages.

What not to block too aggressively

One of the biggest WordPress SEO mistakes is over-blocking directories just because they look technical. Search engines may need assets from those locations to render and evaluate your pages.

Avoid blanket blocking all of /wp-content/.
Avoid blocking theme CSS or JavaScript files.
Avoid blocking plugin assets unless you know they are not used on public pages.

Plugin and SEO sitemap note

If you use Yoast SEO, Rank Math, or another SEO plugin, the sitemap is often /sitemap_index.xml. Add that exact URL to the file so Google can discover it quickly.

If you are not sure which sitemap URL your site uses, open the sitemap manually in the browser before publishing the robots.txt file.

When to use noindex instead

If a WordPress page should stay out of search results entirely, robots.txt is usually not the best tool. Use a noindex rule through your SEO plugin or send an X-Robots-Tag header instead.

Robots.txt is for crawl guidance. Noindex is for search result visibility.

Generate a WordPress version

The generator has a WordPress preset that starts with the common admin rule, keeps admin-ajax.php allowed, and lets you add your sitemap or extra disallow paths without writing the syntax by hand.

Use the WordPress Preset

Request a Tool