Jon Carder, CEO of Mojopages, just asked Matthew Berk if they have ever run into any problems crawling other review sites.
Matt: we are respectful of who we crawl – we typically contact the publisher first and often have partnerships.
My POV is Google does it so why shouldn’t you?
Another question re what about negative reviews? Neither Openlist nor NWSource deletes negative reviews as they are often good viral content. They encourage unhappy businesses to get their customers to post good reviews.
6 Response Comments
I think it is a matter of respect. To crawl someone’s site without permission is like, going through a person office draws. And if you take something that is stealing, because it did not belong to you. We experience behavior constantly on our site, MyDealerReport.com.
Agreed in theory John, but did Google, Yahoo, Ask, MSN, etc ask for your permission? Aren’t they stealing your content and making money off of it? Last time I looked Google had scraped over 16,000 pages from your site.
We’ve asked some content providers about this issue and they’re usually pretty ambivalent.
Typically sites take a sentence or two and link off to the source of the content, so it’s an exchange that’s hopefully a mutual benefit: You get a little bit of content, and they get traffic driven to them. Taking any more and giving any less than that may be stepping too much on the source’s toes.
I agree with this Jesse. The only time I might make an exception is in the case of businesses and reviews. If I am a biz and your site has a review of me that is publicly available I think the biz should be able to lift the review text and add it to their own site with attribution to your site.
If you don’t want your content crawled, put it behind authenticated login. Sites seeking traffic from organic results, quit whining, liberate your 8track on craigslist, buy an ipod and enjoy the brave new world.
Oh … don’t forget to recycle your printed yellowpages with all the other junkmail.
I have been involved in a project that crawls sites for data. However, I have a problem crawling and mining review sites. Can someone get me a product that specifically can crawl review sites? Thanks