The Wayback Machine - https://web.archive.org/web/20040922204751/http://mail.wikipedia.org:80/pipermail/wikien-l/2003-September/006715.html

[WikiEN-l] Broken link on your website

Vicki Rosenzweig vr at redbird.org
Mon Sep 29 21:48:09 UTC 2003 

At 09:22 AM 9/30/03 +1200, Richard Grevers wrote:
>On Mon, 29 Sep 2003 11:07:26 -0700,  Brion Vibber <brion at pobox.com> gave 
>utterance to the following:
>
>>If one of the list admins sent this through; please don't. It's *spam*.
>>
>>They crawl the web for e-mail addresses and then send these things out
>>as advertisements for their link-checking / availability service. Which,
>>of course, means they're crawling our web site and loading it down.
>>*Thanks* so much.
>I wonder if their crawler respects robots.txt?

I would assume not--robots.txt is a protocol for honest people, not
spammers and liars.
-- 
Vicki Rosenzweig
vr at redbird.org
http://www.redbird.org

More information about the WikiEN-l mailing list