LectionFree Tool

Robots.txt Generator

Create a valid robots.txt from user-agent groups, allow and disallow rules, crawl-delay, and sitemap entries. Copy or download the finished file for free.

Free robots.txt generatorCreate robots.txt onlineCopy or download output
Free to runNo credit cardStructured results
Try:
Paste existing rules or start from a preset. Use one Directive: value per line. Comments starting with # are kept in the output. User-agent lines start a new rule group.

Directives follow the robots.txt standard used by Google, Bing, and most crawlers.

What is a robots.txt generator?

A robots.txt generator creates the plain-text robots.txt file that tells automated crawlers which parts of a website they may request. The file lives at the root of a domain, for example https://example.com/robots.txt, and is read by search engines, scraper bots, monitoring tools, and any other client that respects the standard.

Writing the file by hand is easy for two rules and error-prone once a site grows. Common mistakes include missing slashes, a stray space in a path, duplicate user-agent groups that override each other, or a Sitemap line that points to a relative URL. A generator normalizes directive casing, validates each path, flags conflicting groups, and produces a clean file you can drop into production.

robots.txt controls crawling, not indexing. A Disallow rule stops most bots from requesting a page, but it does not keep an already-indexed URL out of search results. If a page should not appear in Google, the page itself needs a noindex directive as well. For crawlers, robots.txt is a cooperation signal: well-behaved bots honor it, so keep the rules precise and test them after every change.

How to read and use a robots.txt generator

Step 1

Choose your user-agents

Use User-agent: * for all crawlers, or create separate groups for Googlebot, Bingbot, and other specific clients.

Step 2

Add allow and disallow rules

List the paths crawlers should skip with Disallow and the exceptions with Allow. Paths must start with / and match from the root.

Step 3

Generate and test

Copy or download robots.txt, upload it to the domain root, then verify specific URLs with the Lection Robots.txt Tester.

Read the generated file from top to bottom. User-agent starts a rule group that applies until the next user-agent line. Within a group, crawlers apply the most specific matching rule: Disallow: /wp-admin/ blocks everything under that folder, while Allow: /wp-admin/admin-ajax.php re-opens one exception path. An empty Disallow: means the group blocks nothing, and Disallow: / blocks the entire site.

For example, a typical WordPress setup keeps crawlers out of private and duplicate areas while pointing search engines at the sitemap:

# robots.txt generated with Lection (free tool)
User-agent: *
Disallow: /wp-admin/
Disallow: /feed/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap.xml

The Sitemap: line is optional but useful: it advertises the location of your sitemap.xml so crawlers discover it without a reference in Google Search Console. Only absolute URLs are valid there. Crawl-delay: is a legacy directive that Google ignores and only some other crawlers honor, so treat it as a hint, not a guarantee.

Common Use Cases

  • Generating a first robots.txt before a new site or subdomain goes live.
  • Blocking search engines from admin panels, carts, checkouts, and internal search pages.
  • Keeping aggressive scraper bots with a known user agent off heavy endpoints.
  • Advertising sitemap locations so crawlers find new content faster.
  • Pausing crawling during a migration or maintenance window with a temporary Disallow: /.
  • Standardizing crawl rules across multiple domains before testing them with a robots.txt tester.

FAQ

What is a free robots.txt generator used for?
It turns user-agent groups and allow or disallow rules into a clean, valid robots.txt file. You paste or build the rules, the tool validates each line, and you copy or download the finished file.
What is the difference between Disallow and Allow in robots.txt?
Disallow lists paths a crawler should not request, while Allow re-opens a specific path inside a blocked directory. Most crawlers apply the longest matching rule, so Allow can create a precise exception to a broader Disallow.
Does robots.txt stop a page from being indexed?
No. robots.txt blocks crawling, not indexing. A page that is blocked but linked to can still appear in search results without a description. To keep a page out of the index entirely, combine robots.txt rules with a noindex meta tag on the page itself.
How do I block all crawlers with robots.txt?
Use a group with User-agent: * and Disallow: /. That tells every crawler that respects robots.txt to avoid the whole domain. Note that it does not remove already-indexed pages and some bots still crawl regardless.
Can I set different rules for Googlebot and other crawlers?
Yes. Create one group per agent, each starting with a User-agent line such as User-agent: Googlebot, then add the rules for that crawler. Avoid repeating the same user-agent in multiple groups, because crawlers use the first matching group and ignore the rest.
Should I add a Sitemap line to robots.txt?
It is good practice when you have one or a few sitemaps. Use one Sitemap line per file, always with an absolute URL such as https://example.com/sitemap.xml. Large sites usually keep sitemap discovery in Search Console instead and list only a sitemap index.
What happens if two rules match the same URL?
Crawlers apply the longest matching path first. If two rules match at the same length, the first rule in the file wins. That is why order matters when Allow and Disallow rules overlap.
Is Crawl-delay still supported by search engines?
Mostly no. Crawl-delay is a non-standard directive that Google has never supported and now ignores, and Bing no longer honors it either. Some smaller crawlers and scraper bots still read it, so include it only if you know the target client honors it.

Related Resources

Robots.txt Analyzer

Inspect an existing robots.txt and confirm your directives are readable by crawlers.

Open tool →

Robots.txt Tester

Test specific URL paths against your generated rules and the winning directive.

Open tool →

XML Sitemap Generator

Build the sitemap.xml that pairs with the Sitemap line in your robots.txt.

Open tool →

Complete Guide to Robots.txt

Understand how robots.txt is read by search engines and scraper bots.

Open tool →

Web Scraping Legality by Country

Review the legal side of crawling sites that do or do not set robots.txt rules.

Open tool →

If you want to move from one-off robots.txt cleanup to repeatable site analysis, start on the Lection home page or compare workflow depth on the pricing page.

Need fresh data on a schedule? Automate this workflow with a free account.Start Free