Robots.txt Generator

Create a valid robots.txt from a template (WordPress, Shopify, block AI crawlers) or your own rules, test any URL against it, then copy, download, or export it as a Next.js robots.ts.

Built-in robots.txt testerBlock AI crawlers in one click

Quick answer: a robots.txt file is a plain-text file at https://yourdomain.com/robots.txt that tells crawlers which paths they may fetch. Start with User-agent: *, add a Disallow: line for each private path (for example Disallow: /admin/), add your Sitemap: URL, and upload the file to the root of your domain. Pick a template below, edit it, and check any URL in the tester before you publish.

Templates:
Paste existing rules or start from a preset. Use one Directive: value per line. Comments starting with # are kept in the output. User-agent lines start a new rule group.

Directives follow RFC 9309, the robots.txt standard used by Google, Bing, and most crawlers.

Robots.txt tester: check a URL against these rules

Uses the rules in the editor above. Longest matching path wins, Allow wins a tie, and * and $ wildcards work the way Googlebot reads them.

Allowed

Line 3: Allow: /wp-admin/admin-ajax.php (group User-agent: *).

What is a robots.txt generator?

A robots.txt generator creates the plain-text robots.txt file that tells automated crawlers which parts of a website they may request. The file lives at the root of a domain, for example https://example.com/robots.txt, and is read by search engines, scraper bots, monitoring tools, and any other client that respects the standard.

Writing the file by hand is easy for two rules and error-prone once a site grows. Common mistakes include missing slashes, a stray space in a path, duplicate user-agent groups that override each other, or a Sitemap line that points to a relative URL. A generator normalizes directive casing, validates each path, flags conflicting groups, and produces a clean file you can drop into production.

robots.txt controls crawling, not indexing. A Disallow rule stops most bots from requesting a page, but it does not keep an already-indexed URL out of search results. If a page should not appear in Google, the page itself needs a noindex directive as well. For crawlers, robots.txt is a cooperation signal: well-behaved bots honor it, so keep the rules precise and test them after every change.

How to create a robots.txt file

Step 1

Choose your user-agents

Use User-agent: * for all crawlers, or create separate groups for Googlebot, Bingbot, and other specific clients.

Step 2

Add allow and disallow rules

List the paths crawlers should skip with Disallow and the exceptions with Allow. Paths must start with / and match from the root.

Step 3

Generate and test

Copy or download robots.txt, upload it to the domain root, then check specific URLs in the robots.txt tester on this page before you publish.

Read the generated file from top to bottom. User-agent starts a rule group that applies until the next user-agent line. Within a group, crawlers apply the most specific matching rule: Disallow: /wp-admin/ blocks everything under that folder, while Allow: /wp-admin/admin-ajax.php re-opens one exception path. An empty Disallow: means the group blocks nothing, and Disallow: / blocks the entire site.

For example, a typical WordPress setup keeps crawlers out of private and duplicate areas while pointing search engines at the sitemap:

# robots.txt generated with Lection (free tool)
User-agent: *
Disallow: /wp-admin/
Disallow: /feed/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap.xml

The Sitemap: line is optional but useful: it advertises the location of your sitemap.xml so crawlers discover it without a reference in Google Search Console. Only absolute URLs are valid there. Crawl-delay: is a legacy directive that Google ignores and only some other crawlers honor, so treat it as a hint, not a guarantee.

Common use cases

  • Generating a first robots.txt before a new site or subdomain goes live.
  • Blocking search engines from admin panels, carts, checkouts, and internal search pages.
  • Keeping aggressive scraper bots with a known user agent off heavy endpoints.
  • Advertising sitemap locations so crawlers find new content faster.
  • Pausing crawling during a migration or maintenance window with a temporary Disallow: /.
  • Standardizing crawl rules across multiple domains before testing them with a robots.txt tester.
  • Opting a site out of AI training crawlers such as GPTBot, ClaudeBot, Google-Extended, and CCBot while staying in Google Search.

How to block AI crawlers with robots.txt

AI companies publish the user-agent tokens their crawlers obey. To keep your content out of AI training and AI answer crawls while staying in normal search results, list those tokens in one group with Disallow: / and leave your User-agent: * group as it is. The Block AI crawlers button above adds this group for you:

# Block AI training and AI search crawlers
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: PerplexityBot
User-agent: CCBot
User-agent: Bytespider
User-agent: meta-externalagent
User-agent: Amazonbot
Disallow: /

Two of these tokens are not separate bots. Google-Extended controls whether Google may use your pages for Gemini training and grounding; blocking it does not affect Googlebot or your rankings. Applebot-Extended does the same for Apple's AI models. Blocking OAI-SearchBot or Claude-SearchBot can remove you from ChatGPT or Claude search answers, so keep those allowed if AI search traffic matters to you. robots.txt is a request, not access control: crawlers that ignore it need to be blocked at the server or CDN.

robots.txt templates and where to upload them

WordPress: WordPress serves a virtual robots.txt until you add a real file. Upload robots.txt to the folder that contains wp-config.php, or edit it with an SEO plugin such as Yoast or Rank Math. Keep Allow: /wp-admin/admin-ajax.php so themes that load content with AJAX still render for Googlebot. Since WordPress 5.5 the core sitemap lives at /wp-sitemap.xml; SEO plugins often use /sitemap_index.xml instead.

Shopify: Shopify generates robots.txt for every store and already blocks cart, checkout, account, and sorted collection URLs. To change it, add a robots.txt.liquid template in the theme editor and append your extra rules there; the Shopify-style template above shows the rules most stores need.

Next.js, Vercel, and static sites: put robots.txt in the public/ folder, or use Copy as Next.js robots.ts after generating to get an app/robots.ts file with the same rules. For any other server, the file must be reachable at exactly /robots.txt on each host: www.example.com and shop.example.com each need their own.

After uploading, open https://yourdomain.com/robots.txt in a browser to confirm it returns the file with a 200 status, then check the robots.txt report in Google Search Console. Google caches robots.txt for up to 24 hours, so changes are not instant.

Frequently asked questions

What is a free robots.txt generator used for?
It turns user-agent groups and allow or disallow rules into a clean, valid robots.txt file. You paste or build the rules, the tool validates each line, and you copy or download the finished file.
What is the difference between Disallow and Allow in robots.txt?
Disallow lists paths a crawler should not request, while Allow re-opens a specific path inside a blocked directory. Most crawlers apply the longest matching rule, so Allow can create a precise exception to a broader Disallow.
Does robots.txt stop a page from being indexed?
No. robots.txt blocks crawling, not indexing. A page that is blocked but linked to can still appear in search results without a description. To keep a page out of the index entirely, combine robots.txt rules with a noindex meta tag on the page itself.
How do I block all crawlers with robots.txt?
Use a group with User-agent: * and Disallow: /. That tells every crawler that respects robots.txt to avoid the whole domain. Note that it does not remove already-indexed pages and some bots still crawl regardless.
Can I set different rules for Googlebot and other crawlers?
Yes. Create one group per agent, each starting with a User-agent line such as User-agent: Googlebot, then add the rules for that crawler. Avoid repeating the same user-agent in multiple groups, because crawlers use the first matching group and ignore the rest.
Should I add a Sitemap line to robots.txt?
It is good practice when you have one or a few sitemaps. Use one Sitemap line per file, always with an absolute URL such as https://example.com/sitemap.xml. Large sites usually keep sitemap discovery in Search Console instead and list only a sitemap index.
What happens if two rules match the same URL?
The longest (most specific) matching path wins, no matter where it appears in the file. If an Allow and a Disallow rule match with the same length, Google and RFC 9309 use the less restrictive one, Allow. The tester on this page applies the same logic.
How do I block ChatGPT, Claude, and other AI bots with robots.txt?
Add a group that lists the AI user agents (GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, CCBot, Bytespider, meta-externalagent, Amazonbot) followed by Disallow: /. The Block AI crawlers button adds it. Your User-agent: * rules for search engines stay unchanged.
Where do I upload robots.txt?
To the root of the host, so it loads at https://yourdomain.com/robots.txt. A file in a subfolder is ignored. Each subdomain needs its own file. On WordPress it goes next to wp-config.php, on Shopify you edit robots.txt.liquid, and on Next.js it goes in public/ or app/robots.ts.
How do I test my robots.txt file?
Paste the file into the editor and enter a URL and user agent in the tester: it shows Allowed or Blocked and the exact line that decided it. After publishing, open /robots.txt in a browser and check the robots.txt report in Google Search Console.
Do I need a robots.txt file?
No. Without one, crawlers assume the whole site may be crawled. Add one when you want to keep crawlers out of admin, cart, or internal search pages, point them at your sitemap, or opt out of AI crawlers.
Is Crawl-delay still supported by search engines?
Mostly no. Partly. Crawl-delay is not in the robots.txt standard and Google ignores it (set crawl rate in Search Console instead). Bing, Yandex, and some scraper bots still read it, so include it only for crawlers that honor it.

Related resources

XML Sitemap Viewer

Check the sitemap your Sitemap line points to and export its URLs.

Meta Tag Checker

Confirm noindex, canonical, and title tags on pages you keep crawlable.

Complete Guide to Robots.txt

Understand how robots.txt is read by search engines and scraper bots.

Web Scraping Legality by Country

Review the legal side of crawling sites that do or do not set robots.txt rules.

If you want to move from one-off robots.txt cleanup to repeatable site analysis, start on the Lection home page or compare workflow depth on the pricing page.

Scrape any site, every page, on a schedule.

Add to Chrome — free

This free tool is one ready-made template of Lection, an AI web scraper Chrome extension that extracts lists, tables and detail pages from any website. It can also scrape any website to Google Sheets on a schedule. Need a recurring data feed for your team or clients? We can set it up for you.