Robots.txt Generator
Generate a clean robots.txt with a user-agent, allow or disallow paths and an optional sitemap line. Validate paths and download the file instantly.
- 100% free
- No sign-up
- Runs in your browser
- Updated
Your result
Your converted output and its statistics appear here.
How this was calculated
About Robots.txt Generator
A robots.txt file sits at the root of your website and tells search engine crawlers which sections they should not crawl. A single typo in it can hide important pages or leave admin and search pages open to bots, so it pays to generate the file carefully. This Robots.txt Generator writes a correctly formatted rule group from a few simple fields and lets you download it as a ready-made robots.txt file.
How to use Robots.txt Generator
- Enter * in User-agent token to address every crawler, or a single bot name such as Googlebot.
- Choose Disallow or Allow under Rule for listed paths; the choice applies to every path you list.
- Type one path per line in Paths, one per line, each starting with a slash, for example /wp-admin/ or /search/. Leave it empty to allow everything.
- Optionally paste your full sitemap address into Sitemap URL.
- Click Convert to preview the file, then use Copy result or Download to save robots.txt.
- Upload the file to the root of your domain so it opens at yourdomain.com/robots.txt.
Key features
- Writes a User-agent line followed by one Allow or Disallow line per path, in standard robots.txt syntax.
- Produces an allow-all file (User-agent: * with an empty Disallow:) when the path list is left blank.
- Adds a Sitemap: line when you supply an absolute HTTP(S) sitemap URL.
- Validates input inline: paths must start with / and contain no spaces, # or control characters, and the user-agent must be * or one crawler token.
- Offers a Download button that saves the output as a file named robots.txt, plus Copy result.
How it works
Write User-agent plus one directive per path; an empty path list emits Disallow: to impose no restriction; optionally append Sitemap.
Robots.txt Generator examples
Block private and search pages for all crawlers
Input: User-agent token: * | Rule: Disallow | Paths: /private/ and /search/ | Sitemap: https://example.com/sitemap.xml
Result: User-agent: *
Disallow: /private/
Disallow: /search/
Sitemap: https://example.com/sitemap.xml
One crawler group. Install at the site root yourself. Robots rules are public crawl guidance, not access control or guaranteed deindexing; no live inspection is performed.
WordPress rules for Googlebot only
Input: User-agent token: Googlebot | Rule: Disallow | Paths: /wp-admin/ and /cart/ | Sitemap: empty
Result: User-agent: Googlebot
Disallow: /wp-admin/
Disallow: /cart/
One crawler group. Install at the site root yourself. Robots rules are public crawl guidance, not access control or guaranteed deindexing; no live inspection is performed.
Important notes before you rely on the result
- These generators create correct code, but rankings depend on content quality, links and many other factors no tool can guarantee.
- A robots.txt file must be uploaded to the root of your domain to take effect, and it does not hide pages from people who know the URL.
- After changing meta tags or robots rules, use Google Search Console’s URL Inspection tool to request a recrawl.
Privacy
Runs in your browser. Nothing you enter is uploaded or saved.
Robots.txt Generator: frequently asked questions
Can I create rules for several bots or mix Allow and Disallow?
Not in one pass. The generator creates a single crawler group: one user-agent and one rule type applied to every listed path. For a more complex file, generate each group separately and join them in a text editor, leaving a blank line between groups. Most small and medium sites only need one group for all crawlers, which is exactly what the default produces.
Why is my path rejected?
Each line must begin with a slash and must not contain spaces, the # character or invisible control characters, because these either break robots.txt syntax or start a comment. Write /my-page/ rather than my-page, and percent-encode special characters, for example %20 for a space. Empty lines are simply ignored, so spacing between entries is fine.
What does an empty Disallow line mean?
Disallow: with nothing after it means no restrictions at all, so compliant crawlers may visit every page. The tool writes it automatically when you leave the path list empty. It is the standard way to publish a valid robots.txt that allows everything while still giving you a place to add a sitemap line.
Will robots.txt remove pages from Google or protect private content?
No. The file is public, voluntary guidance that well-behaved crawlers follow; it is not a password or security measure, and blocked URLs can still appear in search results if other sites link to them. To keep a page out of search, use a noindex tag or remove it, and protect private areas with a login.
Where do I put the downloaded robots.txt file?
Upload it to the root folder of your domain so it opens at https://yourdomain.com/robots.txt. A file inside a subfolder is ignored by crawlers. Each subdomain needs its own copy. On WordPress, check whether an SEO plugin already serves a virtual robots.txt, since a physical file in the root normally takes priority.