Why doesn't the Google crawler show me relevant results?

Our Google crawler gathers data returned by Google for a particular search term. If the search term has a lot of ads and other sponsored content associated with it, then they show up in the search results along with the search query and end up in the crawled data.

For example, if Google shows bakery data as ads while searching for bicycles, the crawler will gather bakery ads along with bicycle data results. We collect all that Google returns for a particular search query. 

To avoid unwanted ads/duplicate data gathered from google search, you could try using google power search (https://sites.google.com/view/togetherlearning/learn/digitalliteracy/powersearching) to generate input to use in the crawler that can help you get the exact results for your search.

 I.e., If you want to find results that use a specific phrase, surround your search term in "quotation marks" to tell Google you only want to see results that use that exact phrase and then copy that URL as input to the crawler.

• Related Articles

• Why does the Google crawler not return ALL results?

  Google will always display only a few hundred results per query. So if your query has thousands of results, all those results will not be displayed by Google and as a result, we cannot gather that data for your either. Our crawler is dependent on ...

• Why doesn’t your crawler scrape some latest reviews?

  Reviews posted on sites such as Amazon take time to show up on Amazon.com and will show up within 72 hours of posting in some cases. Some reviews take a little longer to be posted, sometimes up to 4 or 5 days. It is completely up to Amazon on how ...

• Do you cache results?

  No, we always provide the latest and most accurate data present on the website. However, you are free to cache the results in your systems to reduce API calls and costs.

• How do I get geo-based results like delivery charges to a specific place?

  Most sites will display delivery charges or product availability automatically based on the user location or zip code. Our regular APIs use locations from all US states, so the pricing or availability may vary. To get accurate results based on a ...

• Why does the crawler crawl more pages than the total number of records it collected?

  Some crawlers can collect multiple records from a single page, while others might need to go to 3 pages to get a single record. For example our Amazon Bestsellers crawler collects 50 records from one page, while our Indeed crawler needs to go through ...