Why does the Google crawler not return ALL results?
Google will always display only a few hundred results per query. So if your query has thousands of results, all those results will not be displayed by Google and as a result, we cannot gather that data for your either. Our crawler is dependent on what Google displays along with all their restrictions.
In order to maximize the number results for a query, please create multiple similar queries with different locations and keyword combination. Please note that this may result in duplicate records if the same results ( business/place ) show up under different queries.
For example,
Instead of providing queries like "Zoo in Canada"
split this query like below
Zoo in Alberta
Zoo in British Columbia
Zoo in Manitoba
Zoo in New Brunswick
Zoo in Newfoundland and Labrador
Zoo in Yukon and so on
Please note that this may result in duplicate records if the same results (business/place) show up under different queries.
Related Articles
Why doesn't the Google crawler show me relevant results?
Our Google crawler gathers data returned by Google for a particular search term. If the search term has a lot of ads and other sponsored content associated with it, then they show up in the search results along with the search query and end up in the ...
Do you cache results?
No, we always provide the latest and most accurate data present on the website. However, you are free to cache the results in your systems to reduce API calls and costs.
How do I get geo-based results like delivery charges to a specific place?
Most sites will display delivery charges or product availability automatically based on the user location or zip code. Our regular APIs use locations from all US states, so the pricing or availability may vary. To get accurate results based on a ...
What is the difference between a Crawler and an API on our website?
Our Marketplace has two types of self-service offerings Crawlers APIs Crawlers: A Crawler is something anyone can use - you can signup and use a crawler by simply copying and pasting some links you need data from and download the data. You do not ...
I want to add a new field to the cloud crawler
If you would like some modification to any existing cloud scrapers listed here https://cloud.scrapehero.com/library/crawler, we cannot customize the cloud scrapers BUT we can build a custom solution for you which we can run in our "full-service" ...