Why doesn't the Google crawler show me relevant results?
Our Google crawler gathers data returned by Google for a particular search term. If the search term has a lot of ads and other sponsored content associated with it, then they show up in the search results along with the search query and end up in the crawled data.
For example, if Google shows bakery data as ads while searching for bicycles, the crawler will gather bakery ads along with bicycle data results. We collect all that Google returns for a particular search query.
To avoid unwanted ads/duplicate data gathered from google search, you could try using google power search (https://sites.google.com/view/togetherlearning/learn/digitalliteracy/powersearching) to generate input to use in the crawler that can help you get the exact results for your search.
I.e., If you want to find results that use a specific phrase, surround your search term in "quotation marks" to tell Google you only want to see results that use that exact phrase and then copy that URL as input to the crawler.
Related Articles
Why does the Google crawler not return ALL results?
Google will always display only a few hundred results per query. So if your query has thousands of results, all those results will not be displayed by Google and as a result, we cannot gather that data for your either. Our crawler is dependent on ...Why doesn’t your crawler scrape some latest reviews?
Reviews posted on sites such as Amazon take time to show up on Amazon.com and will show up within 72 hours of posting in some cases. Some reviews take a little longer to be posted, sometimes up to 4 or 5 days. It is completely up to Amazon on how ...How many page credits do I need to scrape data about businesses from Google Maps?
Our Google maps crawler needs to go through a list of all businesses and then go into each business' detail page to get additional data about each business such as their telephone, address, open hours, website address, etc. Here is the business ...How many page credits do I need to scrape reviews from Google?
Our Google Reviews Crawler collects data from the review listing page (one Google review page usually contains 10 reviews). e.g. If you want to scrape 100 reviews for 1 listing, then the scraper uses 11 page credits. 1 page to go to the Google ...Do you cache results?
No, we always provide the latest and most accurate data present on the website. However, you are free to cache the results in your systems to reduce API calls and costs.