Why does the Google crawler not return ALL results?

Why does the Google crawler not return ALL results?

Google will always display only a few hundred results per query. So if your query has thousands of results, all those results will not be displayed by Google and as a result, we cannot gather that data for your either. Our crawler is dependent on what Google displays along with all their restrictions.

In order to maximize the number results for a query, please create multiple similar queries with different locations and keyword combination. Please note that this may result in duplicate records if the same results ( business/place ) show up under different queries.

For example,
Instead of providing queries like "Zoo in Canada"
split this query like below
Zoo in Alberta
Zoo in British Columbia
Zoo in Manitoba
Zoo in New Brunswick
Zoo in Newfoundland and Labrador
Zoo in Yukon and so on
Please note that this may result in duplicate records if the same results (business/place) show up under different queries.


    • Related Articles

    • Why doesn't the Google crawler show me relevant results?

      Our Google crawler gathers data returned by Google for a particular search term. If the search term has a lot of ads and other sponsored content associated with it, then they show up in the search results along with the search query and end up in the ...
    • How many page credits do I need to scrape data about businesses from Google Maps?

      Our Google maps crawler needs to go through a list of all businesses and then go into each business' detail page to get additional data about each business such as their telephone, address, open hours, website address, etc. Here is the business ...
    • How many page credits do I need to scrape reviews from Google?

      Our Google Reviews Crawler collects data from the review listing page (one Google review page usually contains 10 reviews). e.g. If you want to scrape 100 reviews for 1 listing, then the scraper uses 11 page credits. 1 page to go to the Google ...
    • Do you cache results?

      No, we always provide the latest and most accurate data present on the website. However, you are free to cache the results in your systems to reduce API calls and costs.
    • How do I get geo-based results like delivery charges to a specific place?

      Most sites will display delivery charges or product availability automatically based on the user location or zip code. Our regular APIs use locations from all US states, so the pricing or availability may vary. To get accurate results based on a ...