Re: Blogs & Porn & Roller

Phil wilson responded to my rant about blogs & porn

Porn sites add themselves to your referrer list in order to boost their
position in search engines! It emerged as the latest trick about two months
ago, I think Dave Winer mentioned it on his blog at some point.

I don’t think there’s any way around it, but you could always add your
referrers page to robots.txt to stop search engines listing it.

and here’s Dave’s mention

Referer spam is starting to get pretty annoying. A couple of weeks ago we finally figured out why porn sites add themselves to referer pages on high page-rank sites: to improve their placement in search engines. Last night at dinner Andrew Grumet came up with the solution. In robots.txt specifically tell Googlebot and its relatives to not index the Referers page. Then the spammers won’t get the page-rank they seek. Maybe they’ll bother someone else.

More info about robots.tx here. So the robots.txt would look like:

User-agent: * Disallow: /Referrers/

Now should a default Robots.txt be created by roller for every user? Dave/Lance/Matt/Anthony any ideas?


5 thoughts on “Re: Blogs & Porn & Roller”

  1. Where is the “referrers” page everyone keeps referring to? The only way that anyone sees your referrer logs is if you add a macro to your page to display them, otherwise they are private. Anyhow, I don’t think that robots.txt is the going to deter porn operators.

  2. Part of the benefit of referrer lists is referrer lists. Often-referred-to blogs filter to the top of search engines, resulting in more reads for that blog, which then leads to people following referrer links back to the referring site. The entire blogging community benefits from referals and their affect on search engines. I think changing your robots.txt is a horrible idea.

    One idea I’ve heard thrown around is shared filtering lists with trust relationships. If I have your blog on a trust list, and you list as a spammed referral, my blog will also believe it is spam and ignore it. That certainly will take much work to build, but is defenitely better.

  3. Sorry, that first sentence in my comment should say “Part of the benefit of referrer lists is referrer lists contributing to page rank, for both sites involved.” I guess I lost my train of thought. 😉

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s