Re: [xml-dev] Re: web crawling (was: Re: [xml-dev] HGRAB. Syndication. Google. Grey area.)

----- Original Message -----
From: "Paul T" <>
To: "K. Ari Krupnikov" <>
Cc: <>
Sent: Tuesday, January 08, 2002 9:13 PM
Subject: [xml-dev] Re: web crawling (was: Re: [xml-dev] HGRAB. Syndication.
Google. Grey area.)

> <aside>
> I've looked at many robots.txt files and nobody
> disallows the /. Maybe there are some especial websites
> that *do* that, but http://www.metasystema.org/terms.mhtml
> looks like a  *very* rare example to me. But that's
> irrelevant, because you make a stronger point.
> </aside>


http://www.kpmg.com/robots.txt

--
THINGS TO DO IF I BECOME AN EVIL OVERLORD #71
If I decide to test a lieutenant's loyalty and see if he/she should be made
a trusted lieutenant, I will have a crack squad of marksmen standing by
in case the answer is no.


_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com