Free ToolHTTP Header Inspector
View any website's HTTP response headers instantly. Detect Cloudflare, WAFs, rate limits, and CORS policies — useful before you start scraping.
What are HTTP headers?
Every time your browser or a script requests a web page, the server sends back two things: the content (HTML, JSON, an image) and a set of headers. Headers are key-value metadata that travel alongside the response, telling the client how to interpret and handle what it received.
Headers carry information about the content type, caching rules, security policies, rate limits, and which CDN or security layer handled the request. They are invisible to end users but fundamental to how the web works — and they contain critical signals for anyone building a web scraper.
Unlike page content, which requires rendering JavaScript and parsing HTML, headers are available immediately in the initial response. This makes them the fastest way to understand a site's infrastructure and access controls before writing a single line of scraping code.
Which headers matter most for web scrapers?
| Header | What it means for scraping |
|---|---|
| cf-ray | Cloudflare is proxying this site. Scraping may work, but Cloudflare can add JS challenges, CAPTCHAs, or IP bans. |
| x-ratelimit-remaining | The number of requests you have left before being rate-limited. Back off as this approaches zero. |
| retry-after | Seconds to wait after a 429 response. Ignoring this will get you blocked faster. |
| access-control-allow-origin | If set to "*", the API is publicly accessible from any origin — great for scraping JSON APIs. |
| content-type | Tells you whether you're getting HTML, JSON, XML, or binary data — so you know how to parse it. |
| content-encoding | If set to gzip or br, the body is compressed. Most HTTP libraries handle decompression automatically. |
| cache-control: no-store | Content changes frequently and shouldn't be cached — useful for knowing when to re-scrape. |
| set-cookie | Session cookies may be required for subsequent requests. Some anti-bot systems fingerprint cookie handling. |
| x-robots-tag | The server-side equivalent of the HTML robots meta tag. Noindex or nofollow instructions for crawlers. |
Frequently asked questions
What is an HTTP header inspector used for?
An HTTP header inspector fetches the response headers from a web server and displays them in a readable format. Web developers and data engineers use it to understand caching behavior, detect CDN or WAF protection, troubleshoot CORS errors, and assess how accessible a site is for automated scraping or API requests.
How can I tell if a site uses Cloudflare from its headers?
The clearest signal is the cf-ray header, which is added to every response Cloudflare proxies. You may also see server: cloudflare or cf-cache-status headers. Cloudflare's presence doesn't automatically mean scraping is blocked — it depends on the site's security rules — but it does mean bot-detection and potential rate limiting are in play.
What does a 403 response header mean for web scraping?
A 403 Forbidden response means the server recognized your request but refused to fulfill it. For scrapers, this usually means your IP, user-agent, or request pattern has been identified and blocked. Common causes include missing authentication cookies, a blocked IP range, or a WAF rule triggered by your headers.
What are rate limit headers and how should I respect them?
Rate limit headers (x-ratelimit-remaining, x-ratelimit-reset, retry-after) tell you how many requests you have left and when your quota resets. Responsible scrapers read these headers and back off accordingly. Ignoring rate limits leads to 429 errors, IP bans, and potential legal issues. Aim to stay well below the limit — around 50-60% of the stated maximum.
What is the difference between HEAD and GET for header inspection?
A HEAD request asks the server for headers only, without sending the response body. This is faster and more efficient than GET. However, some servers do not support HEAD and return 405 Method Not Allowed. This tool automatically falls back to GET if HEAD is not supported, so you always get accurate header data.
Why do response headers matter for CORS and API access?
The access-control-allow-origin header controls which origins browsers allow to make cross-origin requests. If it is set to "*", the API endpoint accepts requests from any domain — making it straightforward to query from scripts. If it is restricted to a specific origin, direct browser-based access from your domain will be blocked, though server-side scraping (bypassing the browser) is unaffected by CORS.
Can I scrape a site protected by Cloudflare?
It depends on the site's Cloudflare configuration. Many sites use Cloudflare purely as a CDN for performance, with no aggressive bot rules, and are perfectly scrapable with standard HTTP requests. Others use Cloudflare's bot management product, which can require JavaScript challenge completion, fingerprint browsers, or block datacenter IP ranges. Tools like Lection run inside a real Chrome browser, which means Cloudflare sees a legitimate browser fingerprint rather than a raw HTTP client.
Common use cases
- Pre-scraping reconnaissance — Inspect headers before building a scraper to understand what protection is in place and what content type the server returns.
- Rate limit planning — Read x-ratelimit-* headers on API endpoints to calculate safe request pacing for large-scale data collection.
- CORS debugging — Verify whether an API allows cross-origin requests before building a frontend that fetches data directly.
- CDN and caching analysis — Understand cache-control directives to know how fresh the data is and when re-scraping will return updated content.
- Security auditing — Check whether a site sets important security headers like Strict-Transport-Security, X-Frame-Options, and Content-Security-Policy.
- Redirect chain investigation — Identify when a URL redirects to a different final destination, which affects canonical URL handling in scrapers.
Related resources
Ready to scrape? Install Lection and extract your first dataset in minutes — no code required.