You usually notice broken scraped data too late.
The CSV downloads successfully. The sync finishes. The dashboard refreshes. Then someone spots that 47 prices are blank, two columns swapped places, and yesterday's "new listings" are mostly duplicates.
That is the part many scraping guides skip. Extraction is only half the job. The real job is trusting the data enough to use it in a workflow, a spreadsheet, or a decision that affects revenue.
Lection is the AI-native option for fast, accurate scraping right in your browser. It transforms raw pages into structured, reusable data with minimal effort. That helps at the extraction layer, but strong teams still add a validation layer so bad runs do not quietly flow downstream.
If you already use recurring scrapes, this post pairs well with How to Schedule Recurring Web Scrapes. If you are still moving results by hand, the same checklist will keep your exports cleaner in Excel, Google Sheets, Airtable, or a warehouse.
Why validation matters after scraping
The standard approach to validation is usually a quick glance at the first ten rows.
That works when the scrape is tiny and the stakes are low. It fails when you are monitoring prices, leads, listings, job posts, or review data every day. The failure mode is not dramatic. It is quiet.
One run comes back with 620 rows instead of the usual 1,900 because pagination stopped on page two. Another run keeps the row count but fills the "price" column with shipping text because the site changed its layout. A third run appends fresh records without a stable ID, so your analysis looks bigger every week while accuracy gets worse.
Validation exists to make those failures loud.
If you have worked with structured JSON exports, the same logic shows up in tools like JSON Schema, which let teams describe the fields, types, and required values a dataset should contain. In data pipelines, Great Expectations' schema validation guidance follows the same principle: define what "good data" means before it reaches the next step.
Why does the standard approach fail?
Most scraped datasets break in familiar ways.
Spot checks miss partial failures
A human opens the sheet, sees that rows exist, and assumes the scrape worked. That check misses the harder issues:
- duplicate IDs buried halfway down the file
- empty fields in only one region or category
- numbers stored as text, which break formulas later
- columns that still exist but now hold the wrong content
The more routine the workflow becomes, the less likely anyone is to notice a subtle break.
Spreadsheets hide bad assumptions
Spreadsheets are forgiving in ways that feel helpful at first.
They accept mixed date formats. They tolerate trailing spaces in category names. They let one cell hold "$199", another hold "199 USD", and a third hold "Contact seller". The sheet still opens, which creates false confidence.
That is why scraped data often looks "fine" until someone builds a pivot table, joins it to another source, or sends it into a report.
Site changes rarely announce themselves
Websites do not tell you when a CSS class moved, a table became cards, or a results page added a sticky banner that shifts the extraction target. The browser still loads. The scraper still runs. What changes is the shape of the output.
That is where a checklist beats intuition. You are not asking, "Does this look okay?" You are asking, "Did this run satisfy the rules we agreed on?"
The pre-run checklist
Validation starts before the scrape runs, not after.
Define the record shape
Write down what one row represents.
Is it one product, one seller offer, one property listing, one company profile, or one review? If that sounds obvious, good. It should still be explicit.
Without a clear record definition, teams mix incompatible rows in the same export. That is how you end up comparing list prices to seller offers, or current listings to historical snapshots, without realizing the dataset changed shape.
This is also where a shared glossary helps. If your team uses terms like canonical URL, deduplication, or pagination inconsistently, keep the web scraping glossary close by.
Choose the fields that cannot fail
Not every column matters equally. Decide which fields are critical enough to block the run if they are missing.
For many business datasets, the critical set looks like this:
- a stable unique identifier
- a title or name
- a URL
- one primary numeric field such as price, salary, or rating
- a scrape timestamp
If any of those columns disappear or turn mostly blank, the run should be treated as failed, even if every optional column still appears.
Lock down unique IDs early
Duplicate handling gets harder after the data has already spread into sheets, dashboards, and automations.
Pick the best available stable identifier up front:
- product URL
- listing ID
- SKU
- company profile URL
- review permalink
If no natural ID exists, create one from a combination of stable fields. What matters is that every future run can answer a simple question: have we seen this exact record before?
Set acceptable null rates and value rules
Real-world scraped data is messy. Zero nulls is not always realistic.
Instead, define tolerances. For example:
- title can be blank in at most 1% of rows
- price can be blank in at most 3% of rows
- country must come from an allowed list
- currency must be one of USD, EUR, or GBP
This is where spreadsheet-level rules can help. Google documents built-in data validation rules in Sheets, which are useful for dropdowns, allowed values, and basic warnings when a column drifts outside expectations.
The post-run checklist
Once the extraction finishes, run the same inspection every time.
Check row counts against a baseline
Do not ask whether you got rows. Ask whether you got the right number of rows.
If a category usually returns 2,400 records and today returns 190, that is almost always a problem worth investigating. Row count thresholds catch pagination breaks, login expiry, filters resetting, and rate-limit interruptions faster than visual spot checks.
For recurring work, keep a small baseline table:
- typical row count
- acceptable variance
- expected category mix
- last known-good run
That turns "something feels off" into a clear pass or fail test.
Check duplicates before append operations
Appending without deduplication is one of the fastest ways to poison a useful dataset.
Before exporting to your destination, check whether the unique ID column suddenly repeats more than expected. If you are sending results to Google Sheets automations or a CRM, duplicates multiply into extra alerts, broken counts, and noisy outreach.
For scheduled monitoring jobs, this one rule alone can save hours of cleanup every week.
Check blanks in business-critical columns
A run can have the correct number of rows and still be unusable.
Look for columns where blanks create downstream damage:
- missing price values in competitor monitoring
- missing email or company URL in lead generation
- missing address in local business directories
- missing job URL in recruiting workflows
Treat these checks differently from optional enrichment fields. Missing seller bio text might be acceptable. Missing the primary field your team pays attention to is not.
Check data types and formatting
Validation is not only about presence. It is also about format.
Common examples:
- dates should parse consistently
- currency should be numeric, not mixed text
- URLs should look like URLs
- ratings should fall within the expected range
- percentages should not arrive as whole numbers in one run and decimals in the next
This matters most when scraped data lands in formulas or reporting tools. A column that mixes "1,299", "$1,299", and "Call for pricing" will quietly break calculations while still looking readable to a person.
Check category and enum drift
Websites often rename filters, labels, or stock statuses.
A field that used to contain "In stock" and "Out of stock" may now return "Available", "Limited", and "Sold out". That is a small content change with large workflow consequences if your formulas, CRM mappings, or alerts still expect the old values.
Keep a short allowlist for fields that feed automation. If new values appear, review them before they flow further.
Keep an exceptions log
Do not rely on memory when a run fails.
Track:
- date of the run
- page or source that failed
- failed rule
- row count deviation
- sample affected records
- what fixed it
Over time, this becomes a practical operating manual. It tells you which sites are brittle, which fields drift most often, and which fixes are worth automating.
How to implement this without building a giant pipeline
You do not need a full data engineering stack to get value from validation.
Start with the extraction contract
In Lection, make the output structure deliberate from the beginning. Choose fields that map cleanly to columns you actually use. Keep names stable. Avoid grabbing decorative text "just in case" if it adds noise.
That clarity makes every later validation step easier. A compact export with well-named fields is much simpler to audit than a wide table with twenty loosely defined columns.
Add lightweight spreadsheet rules
For small teams, the first validation layer can live in the destination:
- dropdowns for allowed values
- formulas that flag blanks in required columns
- duplicate checks on unique IDs
- conditional formatting for suspicious outliers
This is often enough to protect sales, research, or operations workflows without adding engineering overhead.
Escalate only when the workflow justifies it
If your scraped data feeds a production application, a client deliverable, or daily pricing decisions, upgrade from checklist-only validation to formal schema checks and automated alerts.
That is where standards like JSON Schema and pipeline tools like Great Expectations become useful. They let you define constraints once and enforce them on every run, instead of rediscovering the same failure patterns manually.
If your team is still comparing spreadsheet formulas by hand, you are probably carrying validation work in the most expensive place possible: human attention.
Troubleshooting and edge cases
The site changed visually, but your columns still populate
This is the sneakiest failure mode. Values appear in every column, but one column now contains nearby text instead of the intended field.
The fix is to validate content, not only presence. For example, "price" should contain mostly numeric values, not shipping notices or promo labels.
International formats break comparisons
A site may return "1.299,00 EUR" in one market and "$1,299.00" in another. If you normalize those too late, trend charts and alerts will misfire.
Add a normalization step before analysis, especially for currencies, dates, and decimal separators.
Partial runs look normal on summary dashboards
If your dashboard only reports total rows loaded, a partial run may look successful. This is why baseline comparisons by category, region, or page set matter. A total count can hide major gaps.
Teams over-validate low-value fields
Not every column deserves strict enforcement. Be selective. If you make the checklist too noisy, people will stop trusting it. Focus on the fields that change decisions, exports, or automations.
The checklist you can reuse
Before every new scraped dataset goes live, confirm these points:
- each row represents one clearly defined record
- a stable unique ID exists
- required fields are named and prioritized
- allowed values or formats are documented for key columns
- row count thresholds are set
- duplicate detection runs before append or sync
- blanks in critical fields trigger review
- numeric, date, and URL formats are normalized
- enum drift is reviewed before automation continues
- exceptions are logged somewhere your team will actually revisit
Validation is rarely the glamorous part of scraping. It is the part that makes the rest trustworthy.
Ready to start scraping? Install Lection and build a reusable data workflow that stays accurate after the first run. You can also review pricing if you are planning recurring cloud jobs for larger datasets.