Lection

Robots.txt Checker & Scraper Rules Analyzer

Check any website's robots.txt file instantly. Understand what you can scrape before you start.

Free to runNo credit cardStructured results

Try: amazon.com, google.com, linkedin.com, or any website

What is a robots.txt File?

A robots.txt file is a plain text file placed at the root of a website (e.g., example.com/robots.txt) that provides instructions to web crawlers and bots about which pages or sections of the site they should or should not access. This file follows the Robots Exclusion Protocol, a standard used by search engines like Google, Bing, and web scraping tools to understand a website's crawling preferences.

While robots.txt is not legally binding and cannot technically prevent access to pages, it serves as an important signal of the website owner's intent. Respecting robots.txt directives is considered best practice for ethical web scraping and SEO crawling.

How to Read robots.txt Directives

User-agent: *

Applies to all crawlers and bots. The asterisk (*) is a wildcard that matches any user agent.

Disallow: /private/

Tells crawlers not to access any URL starting with /private/. An empty Disallow means everything is allowed.

Allow: /public/

Explicitly permits access to URLs starting with /public/, even if a broader Disallow rule exists.

Crawl-delay: 10

Requests that crawlers wait 10 seconds between requests to avoid overloading the server.

Sitemap: https://example.com/sitemap.xml

Points to the site's XML sitemap, helping crawlers discover all pages efficiently.

Need fresh data on a schedule? Automate this workflow with a free account.Start Free

Frequently Asked Questions

Can I scrape a website if robots.txt blocks it?
Technically yes, but it's not recommended. While robots.txt cannot physically prevent access, ignoring it may violate the website's terms of service and could have legal implications depending on your jurisdiction. Always check the site's terms and consider the ethical implications.
What does Disallow: / mean?
A single forward slash after Disallow means the entire site is blocked from crawling. This is the most restrictive setting and indicates the site owner does not want any automated access.
Why would a site not have a robots.txt file?
Many smaller websites don't have a robots.txt file because they haven't configured one, or they have no restrictions on crawling. The absence of a robots.txt typically means there are no explicit crawling guidelines, but you should still respect the site's terms of service.
Do search engines always follow robots.txt?
Major search engines like Google, Bing, and Yahoo generally respect robots.txt directives. However, malicious bots may ignore these rules. That's why robots.txt is considered a polite request rather than a security measure.
How often should I check a site's robots.txt?
If you're running ongoing scraping operations, check robots.txt periodically as site owners may update their rules. For one-time data collection, checking once before you start is usually sufficient.

Common Use Cases for Checking robots.txt

  • •SEO Audits: Verify that important pages aren't accidentally blocked from search engine indexing.
  • •Web Scraping: Determine which sections of a site allow automated data collection before building scrapers.
  • •Competitive Analysis: Understand how competitors configure their crawl rules and what they prioritize hiding.
  • •Finding Sitemaps: Robots.txt often includes links to XML sitemaps, helping discover all pages on a site.
  • •Developer Debugging: Troubleshoot why certain pages aren't appearing in search results.

Related Resources

Complete Guide to robots.txt for Web Scrapers

Understand crawl rules and best practices.

Open tool →

Web Scraping Legality by Country (2025)

Stay up to date on legal guidelines.

Open tool →

Browse All Free Tools

Explore the full tool directory.

Open tool →

Web Scraping Tutorials

Read the latest guides.

Open tool →