Crawlers
Why doesn’t your crawler scrape some latest reviews?
Reviews posted on sites such as Amazon take time to show up on Amazon.com and will show up within 72 hours of posting in some cases. Some reviews take a little longer to be posted, sometimes up to 4 or 5 days. It is completely up to Amazon on how ...
What is a "Lost" status?
If a job is marked as "Lost," it means our platform is having some internal issues with the crawler. In such cases, you can re-run the job. If the problem persists, you can contact us. Please note that the crawler job may (occasionally) be marked as ...
Amazon - What is the difference between Reviews vs Ratings
Amazon products have two different metrics for Reviews and Ratings. Here is a popular product with 1419 Ratings BUT only 259 Reviews. What are Ratings Ratings are just "stars" rated by people - they may or may NOT add a review. A lot of people just ...
Why does the crawler display no data even after crawling a few pages?
The crawler works by sending requests simultaneously to get the data. The data needs to be processed and is not displayed instantly, it takes some time to display the data, so please be patient and check back periodically. This is especially the case ...
What kind of integrations do you provide?
We provide Dropbox integration to allow you to automatically write data to your Dropbox on a schedule instead of downloading it from our website. We are working on other integration partners to enable better automation.
Why does the crawler crawl more pages than the total number of records it collected?
Some crawlers can collect multiple records from a single page, while others might need to go to 3 pages to get a single record. For example our Amazon Bestsellers crawler collects 50 records from one page, while our Indeed crawler needs to go through ...
Can I get data periodically?
Sure, you can tell us to run a crawler periodically by adding a "Schedule". You can schedule crawlers to run in Monthly, Weekly, Hourly or Minute intervals.