[Search-l] User Agent For Search Wikia

[John McCormac]

Jimmy Wales wrote:
> We are also crawling using Nutch... not sure what the user agent is for 
> that.
It would be best for Search Wikia to have its own unique UA. That way it 
  will not get banned accidentally.

Both Grub and Nutch are common UAs in blacklists. Nutch tends to upset 
some directory owners as any muppet can generally set it up and hammer 
directories. This tends to result in a UA ban and is quickly followed by 
an IP range ban. The Apache mod_rewrite rules generally look for the 
term 'Nutch' in the UA.

Regards...jmcc
-- 
******************************************************
John McCormac  *  e-mail: jmcc at whoisireland.com
MC2            *  voice:  +353-51-873640
22 Viewmount   *  web:  http://www.whoisireland.com/
Waterford      *  blog: http://blog.whoisireland.com
Ireland        *  Irish Domain Stats & Market Research
******************************************************