Why does the crawler crawl more pages than the total number of records it collected?

Why does the crawler crawl more pages than the total number of records it collected?

Some crawlers can collect multiple records from a single page, while others might need to go to 3 pages to get a single record.
For example our Amazon Bestsellers crawler collects 50 records from one page, while our Indeed crawler needs to go through a list of all jobs and then move into each job to get more data about each job.
    • Related Articles

    • What happens to my unused quotas at the end of each billing period?

      All our Crawler page quotas and API quotas, reset at the end of billing period. Any unused credits do not carry over to the next billing period and also are nonrefundable. This is consistent with most software subscription services.
    • Why does the crawler display no data even after crawling a few pages?

      The crawler works by sending requests simultaneously to get the data. The data needs to be processed and is not displayed instantly, it takes some time to display the data, so please be patient and check back periodically. This is especially the case ...
    • Can I get data periodically?

      Sure, you can tell us to run a crawler periodically by adding a "Schedule". You can schedule crawlers to run in Monthly, Weekly, Hourly or Minute intervals.
    • How many page credits do I need to scrape reviews from Amazon?

      Our Amazon Reviews Crawler collects data from the product listing page (one review listing page usually contains 10 reviews).  e.g. If you want to scrape 100 reviews for 1 listing, then the scraper uses 11 page credits. 1 page to go to the product ...
    • Why does the Google crawler not return ALL results?

      Google will always display only a few hundred results per query. So if your query has thousands of results, all those results will not be displayed by Google and as a result, we cannot gather that data for your either. Our crawler is dependent on ...