Most data worth collecting is not on one page. A product category runs to 30 pages, a job board loads 20 results at a time, a directory hides the email address on each listing's own page. Scraping page one is easy. Getting all of it into one sheet is where people give up and start copying by hand.
This guide shows how to scrape multiple pages into a single table with Lection, an AI web scraper for Chrome, without writing code or configuring selectors. It covers the four ways sites split their data: numbered pages, Load More buttons, infinite scroll and detail pages. Then it shows how to rerun the whole thing on a schedule.
The short answer
Open the first page of results, describe the fields you want in the Lection side panel, and ask it to collect every page. Lection looks at the page's controls (a Next link, a Load More button, or a feed that grows as you scroll), picks the matching method and keeps going until there are no more results or your plan's page limit is reached. You get one combined table you can download as CSV or Excel or send to Google Sheets.
Which kind of pagination are you dealing with?
Before you start, scroll to the bottom of the page and look. The method depends on what you see.
| What you see at the bottom | Type | How Lection collects it |
|---|---|---|
| Page numbers, "Next", "»" or "Older" | Numbered pages | Follows the next-page link, page by page |
| A "Load more" or "Show more" button | Load More | Clicks the button and waits for new items |
| Nothing; more items appear as you scroll | Infinite scroll | Scrolls and keeps reading until the list stops growing |
| A list of links to individual items | Detail pages | Opens each link and adds the fields from that page |
Many sites combine two of these. A real estate search might be numbered pages of listings where the square footage only appears on each listing page. That is fine: collect the pages first, then add detail pages.
Before you start
- Desktop Chrome with the free Lection extension.
- The first page of the results you want, with any filters already applied (location, price range, category). Filtering on the site is cheaper than collecting everything and filtering later.
- If the data only shows when you are logged in, log in first in the same browser. Lection reads what your browser can already show. It does not get around logins, paywalls or CAPTCHAs.
Step 1: Get page one right
Do not start with all pages. Start with one, because every mistake on page one repeats on page 30.
- Open the first results page in Chrome.
- Open the Lection side panel.
- Describe the fields, for example: "For each product: name, price, rating, number of reviews and the product link." You can also click the elements you want on the page.
- Compare the preview with the page. Check three or four rows, especially items with a sale price, a missing rating or a sponsored badge.
If a column is wrong, correct it in plain English ("use the sale price, not the crossed-out price", "skip sponsored results") and preview again. Always include the item's link as a column. It lets you remove duplicates later and is needed for detail pages.
Step 2: Collect every page
Once page one looks right, ask for the rest: "Collect all pages" or "Get the next 10 pages" works. When Lection extracts the first page, it also notes whether the page has a Next link, a Load More button or a scrolling feed, and will offer to collect more.
Numbered pages
This is the most reliable type. Lection follows the next-page link, extracts the same fields from each page and appends them to one table. It works best when the page has a clear Next, "»" or "Older" control. If the site only shows numbered links, tell Lection which element moves to the next page.
Load More buttons
Lection clicks the button, waits for the new items to render and reads the list again. If the button disappears or stops adding items, the run ends there.
Infinite scroll
Feeds such as social profiles, some job boards and marketplace search pages load more as you scroll. Lection scrolls, waits and keeps reading until no new items appear. Infinite scroll is the slowest type because the site decides how fast new items arrive, so set a sensible stopping point ("the first 300 results") rather than "everything" on very long feeds.
How many pages per run
How many pages a single run covers depends on your plan: 10 on the free plan, 50 on Standard and 200 on Professional (see pricing). If the site shows 25 items per page, that is roughly 250, 1,250 and 5,000 items per run.
Two ways to go further when a list is longer than your limit:
- Split it with filters. Run one search per category, region or price band. Each run starts again at page one.
- Use several start URLs. Scheduled runs can take a list of URLs (up to 20 per run on Standard and 50 on Professional), so ten category pages become one run that writes one sheet.
Step 3: Add detail pages for the fields that are not in the list
List pages show a summary. The fields you often need, such as a full address, phone number, specification table, salary or seller name, only appear when you open each item.
Ask for them: "Open each product link and add the SKU, weight and full description." Lection opens each link from your table, extracts those fields and adds them as new columns on the same row.
Detail pages multiply the work. Fifty list pages of 20 items each is 1,000 detail pages. So:
- Get the list right first, and remove items you do not need before opening detail pages.
- Only ask for detail fields you will use.
- Test on five rows before running on all of them.
The tutorials page has a short video of this, and another for scraping across pages.
Step 4: Check the combined table
Multi-page runs fail in quiet ways. Before you use the data, check:
- Row count. Does it roughly match what the site says ("1,240 results")? If you got exactly 10, 50 or 200 pages' worth, you hit your plan's page limit.
- Duplicates. Sites repeat featured or sponsored items on every page. Sort by the link column and remove duplicates.
- The last page. Open the last page the run visited and confirm the items match the last rows in your table.
- Blank columns on later pages. Some sites change layout after page one (an ad row, a "you may also like" block). If a column goes blank partway down, add an instruction to skip that block.
Our data validation checklist covers this in more depth.
Step 5: Export, or put it on a schedule
For a one-off job, download the table as CSV or Excel. If the data is a single plain HTML table and does not span several pages, the free HTML Table Extractor does the job without the extension.
If you need the same list again next week (prices, new listings, new jobs), save the extraction and schedule it:
- Open Scheduled Scrapes and create a schedule for the saved scrape. Choose Daily or Weekly (Hourly on Standard and above) and your timezone.
- Set the maximum pages for the run, within your plan's limit.
- Under Delivery, choose Google Sheets, email, a webhook or another destination. The Google Sheets integration page walks through the connection.
- Click Run now to check the first delivery.
Scheduled runs happen in the cloud, so your computer can be off. They work best on pages that do not need a login. Sheets delivery appends rows on each run; the scheduling guide explains how to turn that into a history or a one-row-per-item sheet.
Honest limitations
- Page limits are per run. Very large sites (tens of thousands of pages) need splitting by filter, or a developer tool.
- Credits. Every page and detail page uses credits. The free plan includes 5 credits a month (about 500 rows), Standard 300 (about 30,000 rows) and Professional 1,500 (about 150,000 rows). Detail pages add up fastest, so estimate before you schedule a large job.
- Sites change. A redesign can break the next-page link or move a field. Reopen the extraction, adjust the description and save again.
- Some sites cap results. Many search pages stop at a fixed number of results no matter how many exist. More pages will not get past that; narrower filters will.
- Terms of use. Read the site's terms and robots rules, collect only what you need and keep the pace reasonable. Our robots.txt guide for scrapers explains what to look for.
Common jobs this covers
- Competitor prices across a whole category. See tracking competitor prices in Google Sheets.
- Supplier catalogs with hundreds of SKUs. See scraping a supplier catalog.
- Lead lists from directories where contact details sit on each listing. See building a lead list from a directory.
- Job listings across several boards. See tracking job listings in Google Sheets.
Try it on your own list
Install the free extension, open the first page of a list you need, preview the fields and ask for the next pages. The free plan covers 10 pages per run, which is enough to see whether it works on your site.
When you need more pages per run, several URLs in one run, Google Sheets delivery or hourly schedules, start the 30-day Standard trial. It is free for 30 days, then $29 per month, a card is required, and it is for first-time subscribers only.
FAQ
Can I scrape multiple pages for free? Yes. The free plan collects up to 10 pages per run, with CSV or Excel download.
Does it work with infinite scroll? Yes. Lection scrolls and keeps reading until no new items load or it reaches the limit you set.
How do I scrape data behind each link? Include the link as a column, then ask Lection to open each link and add the fields you need. Test on a few rows first.
Why did my run stop early? Usually the plan's page limit, a site that caps its results, or a Next button that changes on the last page. Check the last page the run visited.
Can I scrape multiple pages into Excel? Yes. Download the combined table as an Excel or CSV file. Scheduled delivery goes to Google Sheets, email, webhooks and other destinations.