LectionFree Tool

Robots.txt Tester

Test whether a specific URL is allowed or blocked for a chosen crawler by reading the site's live robots.txt file and applying the matching rules.

Free robots.txt testerURL path rule checkCrawler-specific crawl review
Free to runNo credit cardStructured results
GuidesRobots.txt guideLegality by country
Paste the exact page URL you want to evaluate. The tester derives the robots.txt file from the URL's host, then checks the path and query string against the matched rules.
Use a product token such as Googlebot, GPTBot, MJ12bot, or * for the generic wildcard group.
Try:

This tester applies live robots.txt fetching, crawler-group matching, and longest-rule evaluation so you can review a URL before you crawl it.

What is a robots.txt tester?

A robots.txt tester checks whether a specific URL path is allowed or blocked for a particular crawler. Instead of only listing raw directives, it applies the crawler-group matching and path-matching logic to the exact URL you care about, which is what teams usually need before they launch a scraper, audit a blocked page, or debug a crawl issue.

That matters because most robots.txt files are not simple. A site may publish different rules for Googlebot, GPTBot, MJ12bot, or a generic wildcard group. On top of that, the file can contain overlapping allow and disallow rules, wildcard patterns, end-of-string anchors, and multiple matching groups. A tester turns that raw text into an answer you can act on.

Lection's free robots.txt tester fetches the live file from the host you enter, finds the most relevant crawler group, compares the exact path and query string against the matching rules, and shows you which rule won. That makes it useful for SEO audits, web scraping checks, site migrations, and page-level crawl debugging.

How to use a robots.txt tester

Step 1

Paste the exact URL

Use the full page URL you want to test, not just the homepage. Query strings can change the outcome on sites that block search or filtered URLs.

Step 2

Set the crawler token

Choose the crawler identity you care about, such as Googlebot, GPTBot, MJ12bot, or the wildcard * group when you want the generic rule set.

Step 3

Read the winning rule

Focus on the matched group, the winning allow or disallow pattern, and whether the path was allowed by a rule or by default because nothing matched.

Example 1: if you test https://www.google.com/search?q=seo with *, the wildcard group blocks that URL because disallow: /search is more specific than the default allow behavior.

Example 2: if you test https://www.google.com/search/about with *, the matching rule allows the path because the more specific allow rule overrides the broader block.

Example 3: if you test a site that has no robots.txt file at all, crawlers commonly treat that as no published crawl restriction for the host. That does not override site terms, rate limits, or other legal and technical constraints, but it answers the narrow robots.txt question.

Common Use Cases

  • Check whether a product, search, category, or filtered URL is blocked before scraping it.
  • Debug why an SEO team cannot get a page crawled even though the page itself looks fine.
  • Compare wildcard rules against crawler-specific rules for Googlebot, GPTBot, or a custom bot token.
  • Review query-parameter paths that often behave differently from clean content URLs.
  • Validate crawl rules during migrations, redesigns, or staging-to-production launches.
  • Confirm whether a block comes from robots.txt or from something else such as authentication, noindex, or response headers.

FAQ

How does this free robots.txt tester decide whether a URL is allowed or blocked?
The tester fetches the live robots.txt file, finds the most relevant crawler group for the token you enter, then applies the longest matching allow or disallow rule to the exact path and query string.
What should I enter in the crawler field when I want the generic robots.txt rules?
Use * to test the wildcard group. That is the default fallback group sites usually publish for crawlers that do not have a dedicated user-agent section.
Can two robots.txt rules both match the same URL path?
Yes. That is common on large sites. In that case the most specific rule wins, and when two equally specific rules conflict, an allow rule is usually treated as the less restrictive result.
What if the site does not have a robots.txt file?
A missing robots.txt file usually means there is no published crawl restriction for that host. It does not mean every kind of automated collection is acceptable, so you should still review terms, rate limits, and the broader context.
Does robots.txt block indexing or only crawling?
Robots.txt is a crawl-control file. It tells crawlers which URLs they should or should not request. Indexing decisions can also depend on page content, canonical tags, noindex directives, and response headers.
Why would a URL still fail even if the robots.txt tester says it is allowed?
A path can be allowed in robots.txt and still fail for other reasons, such as redirects, login walls, bot protection, rate limiting, noindex tags, or canonical signals that point elsewhere.

Related Resources

Pair this page with the Robots.txt Analyzer when you want the full file and sitemap list, or with the Indexability Checker when crawl access is only one part of the problem.

For broader context, read The Complete Guide to robots.txt for Web Scrapers and Web Scraping Legality by Country. If you want to move from one-off checks to repeatable extraction, start from the Lection home page or compare plans on pricing.

Robots.txt Analyzer

Read the full crawl rules, directives, and sitemap locations for a host.

Open tool →

Indexability Checker

Check status codes, redirects, canonicals, and noindex signals on a live page.

Open tool →

Sitemap Viewer

Inspect sitemap indexes, nested sitemaps, and URL counts from a live XML sitemap.

Open tool →

Link Extractor

Extract internal and external links from a page once you know a path is crawlable.

Open tool →
Need fresh crawl-rule data on a schedule? Automate this workflow with a free account.Start Free