To edit robots.txt in WordPress, use the robots.txt editor in your SEO plugin or download, edit, and re-upload the file from your site’s root directory. Save a backup first. For a sensible WordPress baseline, block the private /wp-admin/ area, allow /wp-admin/admin-ajax.php, and declare your XML sitemap. Use a noindex meta tag or X-Robots-Tag header—not robots.txt—when the goal is to keep a URL out of search results.
Antihost offers managed WordPress hosting with free migration, one-click staging and a 14-day free trial.
This guide covers where the file lives, plugin and manual editing paths, safe optimization choices, Google’s rules, and a testing workflow for changes made on 14 October 2026. Plugin menu names can differ by version.
As an Amazon Associate I earn from qualifying purchases.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →What robots.txt does in WordPress
robots.txt is a plain-text file at the root of your host, normally available at https://www.example.com/robots.txt. Crawlers request that file before fetching other URLs and use its directives to decide which paths they may crawl.
It is a crawl-control file, not an access-control system. A disallowed URL can still appear in search if Google discovers it elsewhere, because the crawler cannot fetch the page to see its contents. If you need a page removed from or kept out of search, place a noindex robots meta tag on the page or send an X-Robots-Tag HTTP header. Do not block that URL in robots.txt at the same time: Google cannot retrieve the page to read the noindex instruction.
#1 Best Overall
Google reads these fields:
User-agent, which identifies the crawler to which a rule applies.Allow, which permits a path.Disallow, which asks a crawler not to fetch a path.Sitemap, which gives the crawler the address of an XML sitemap.
Google does not support crawl-delay. Directive names are not case-sensitive, but path matching is case-sensitive, so capitalization in a rule must match the URL path you intend to control. When rules conflict, Google uses the least restrictive applicable rule.
Before you change the file
A bad rule can stop search engines from fetching important pages or assets. Complete these checks before editing:
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errors- Confirm the site’s canonical host and the exact root URL where the file is served.
- Open the current
/robots.txtin a browser and save a copy of its contents. - Identify the live XML sitemap URL generated by your WordPress setup.
- List paths that are genuinely low-value for crawling rather than trying to hide every URL you do not want indexed.
- Keep a plain-text editor ready. Notepad or TextEdit in plain-text mode is suitable; a word processor can insert formatting characters that invalidate the file.
- Plan to test the resulting live file in Google Search Console.
Choose how to edit robots.txt
| Method | Current path or process | When it fits |
|---|---|---|
| Rank Math editor | Rank Math → General Settings → Edit robots.txt | Sites already using Rank Math and comfortable changing the file in the WordPress dashboard. |
| All in One SEO editor | All in One SEO → Tools → Robots.txt; enable Enable Custom Robots.txt | Sites already using All in One SEO and wanting a dashboard workflow. |
| Yoast editor | Open Yoast’s robots.txt box, which pre-fills default settings, then add or adjust lines. | Sites already using Yoast and wanting to work from its generated defaults. |
| Manual upload | Download the root file, edit it as UTF-8 plain text, and upload it back to the root folder through FTP or a host file manager. | Sites without an SEO-plugin editor or administrators who need direct file control. |
The exact dashboard labels are version-dependent. If a menu has moved, look for a robots.txt tool under the plugin’s general settings or tools area. Dedicated plugins such as Virtual Robots.txt, WordPress Robots.txt Optimization, and Robots.txt Editor also exist, but adding another plugin is unnecessary when an installed SEO plugin already provides an editor.
Method 1: edit robots.txt with a WordPress SEO plugin
Rank Math
- Sign in to WordPress with an account allowed to change Rank Math settings.
- Open Rank Math → General Settings → Edit robots.txt.
- Copy the existing contents into a separate text file before changing anything.
- Make the smallest rule changes needed: protect the admin area, retain the AJAX exception, and add the sitemap declaration.
- Save the editor, then check the publicly served file at your site’s
/robots.txtaddress.
All in One SEO
- Open All in One SEO → Tools → Robots.txt.
- Turn on Enable Custom Robots.txt.
- Review the generated or existing directives and preserve any rules that are still required by the site.
- Add, remove, or edit lines in the custom editor, save, and verify the live root file.
Yoast
- Open Yoast’s robots.txt editor or robots.txt box in the WordPress dashboard.
- Use the pre-filled defaults as a starting point rather than deleting them blindly.
- Adjust only the paths you have evaluated, keeping the sitemap line accurate.
- Save and fetch the root file in a browser to confirm that the saved text is the text being served.
Plugin editors reduce the need to access hosting, but they do not remove the need for a backup and a live-file check. A dashboard preview is not proof that the web server is serving the same contents.
Method 2: edit robots.txt manually with FTP or a file manager
- Open your hosting control panel’s file manager or connect with FTP.
- Locate the site’s document root—the folder that serves the public home page.
- Download the existing
robots.txtas a backup. If there is no file, record that fact before creating one. - Open the downloaded file in a plain-text editor such as Notepad or TextEdit. Keep the encoding as UTF-8.
- Make the required edits. Keep each directive on its own line and avoid rich-text formatting, smart quotes, or comments copied from a word processor.
- Upload the file to the same document-root folder, replacing the old copy only after the backup is safely stored.
- Request
https://your-site.example/robots.txtin a browser and inspect the returned text.
Editing a similarly named file inside a theme, plugin, or subdirectory does not change the file crawlers request. The effective file is the one at the host root.
A safe WordPress starting template
The following is a starting point, not a universal policy. Replace the example sitemap address with the XML sitemap URL actually generated for your site.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.example.com/sitemap.xmlThe important relationship is the exception: the broad /wp-admin/ disallow is paired with an allow rule for /wp-admin/admin-ajax.php. Removing that exception can interfere with crawlers that need the endpoint for permitted requests.
Optional low-value paths
Some WordPress guidance also proposes disallowing these paths:
Disallow: /wp-content/plugins/
Disallow: /trackback/
Disallow: /readme.htmlTreat each line as a site-specific decision. Confirm that no public resource or URL you expect crawlers to fetch depends on the path before adding it. The available guidance does not establish a universal rule for blocking CSS or JavaScript directories, so do not add asset blocks merely because they appear in a generic template.
How to decide what belongs in the file
Keep private administration out of routine crawling
/wp-admin/ is a common low-value crawl area. If you use that rule, retain /wp-admin/admin-ajax.php as an allow exception. This is a crawl preference, not a security barrier; protect administration with WordPress authentication and hosting security controls as well.
Declare the sitemap once, accurately
Add a Sitemap: line containing the full, publicly reachable XML sitemap URL. The directive can point crawlers to a sitemap without granting permission to any path. If your site has more than one sitemap, use the sitemap index URL or list each sitemap address on its own line according to the structure your WordPress installation publishes.
Use disallow sparingly
A disallow rule prevents crawling of matching paths for the user agent to which it applies. It does not erase already known URLs, remove links from search, or stop users from visiting a page. Broad rules can also prevent Google from seeing a noindex instruction or from fetching resources needed to understand a page.
Do not use crawl-delay for Google
Google does not support the crawl-delay field. Adding it will not control Googlebot’s crawl rate. Keep the file to fields Google supports: user-agent, allow, disallow, and sitemap.
Check capitalization and conflicts
Field names such as User-agent and Disallow are case-insensitive, while path matching is case-sensitive. A rule for /Store/ is not the same as a rule for /store/. If an allow and disallow rule overlap, Google selects the least restrictive applicable result, so inspect the combined effect rather than reading one line in isolation.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallrobots.txt versus noindex: choose the right control
| Goal | Use | Why |
|---|---|---|
| Reduce crawling of an administrative or low-value path | Disallow in robots.txt |
Instructs Google not to fetch matching URLs. |
| Keep a page out of search results | noindex robots meta tag or X-Robots-Tag header |
Lets Google retrieve the response and read the indexing instruction. |
| Tell crawlers where XML sitemaps are | Sitemap directive |
Supplies the sitemap location in the root file. |
Throttle Google with crawl-delay |
Do not rely on robots.txt | Google does not support this field. |
For example, if an obsolete article must disappear from search, adding Disallow: /old-article/ can prevent Google from seeing a noindex tag on that article. Place the noindex instruction on the response instead, and reserve robots.txt for crawl control.
Rank #4
Test before and after publishing
- Save the proposed text locally so you can compare it with the current file.
- Use the robots.txt report in Google Search Console to inspect the live file.
- Use its testing function to edit a test copy and check whether representative URLs are allowed or blocked. The test edit does not change the live file.
- Test both sides of every exception: check a URL under
/wp-admin/and check/wp-admin/admin-ajax.phpseparately. - Test a representative public page, the XML sitemap URL, and any path affected by an optional rule.
- When the result is correct, copy the tested text into the real plugin editor or root file and save it.
- Request the live
/robots.txtagain and compare it with the version you tested.
The Search Console test is a planning aid. It does not publish your changes; the actual plugin save or file upload is still required.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Troubleshooting common failures
The browser shows an old file
Confirm that you edited the document root for the correct host and that the upload completed. Fetch the exact root URL, including the host variant used by your site. If a plugin editor was used, verify that the plugin saved a custom file rather than displaying an unsaved preview.
A page remains in search after being disallowed
This is expected behavior when Google knows the URL but cannot crawl it. Remove the disallow for that URL long enough for Google to fetch the page, then serve a noindex meta tag or X-Robots-Tag. Robots.txt is not a deindexing mechanism.
Google cannot see a noindex tag
Look for a disallow rule covering the page or its response path. A crawler blocked before retrieval cannot read the noindex instruction. Adjust crawl rules so Google can fetch the response, then verify the indexing directive separately.
AJAX-dependent behavior stops working
Check whether a broad /wp-admin/ rule lacks the Allow: /wp-admin/admin-ajax.php exception. Restore the exception and retest both URLs in Search Console.
The file contains a crawl-delay line
Remove that line if it was intended to control Googlebot; Google does not support it. Keep only supported directives and test the resulting file.
A rule affects the wrong capitalization
Compare the case of every path segment in the directive with the actual URL. Because path matching is case-sensitive, change the rule or add a correctly cased rule for the path you intend to control.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Maintenance after the first edit
Revisit robots.txt whenever your WordPress information architecture, sitemap URL, or administrative paths change. Keep the backup from before each edit, record why a non-default path was blocked, and retest after plugin or hosting changes that may regenerate the file. A short, understandable file is easier to audit than a collection of copied rules whose purpose is unknown.
Use the narrowest rule that solves the crawl problem. Leave public pages and resources crawlable unless you have a documented reason to restrict them, and handle indexing decisions with noindex controls rather than increasingly broad disallow directives.
Frequently Asked Questions
Can I edit robots.txt from the WordPress media library?
No. The effective file is served from the host’s document root. Use an SEO plugin’s robots.txt editor or upload the root file through FTP or your hosting file manager.
Will adding Disallow remove a URL from Google?
No. Disallow controls crawling. A URL that Google already knows can remain indexed. Use a noindex meta tag or X-Robots-Tag, and do not block the URL before Google can read that instruction.
Does Google honor crawl-delay in robots.txt?
No. Google does not support the crawl-delay field.
Do changes made in Search Console automatically publish?
No. The robots.txt testing tool works on a test version. You must copy the approved text into the live plugin editor or upload it to the root file and save it.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
