[Grub-dev] C# Grubng 0.1 and few questionsHi everyone I create today first alpha version of graphic Grub client. It have all options of first release - minimal GUI, unstable program and ugly source code ; ) But i think it can be useful for test p

[Mark (Markie)]

wouldn't it be better to just set the user agent as something completely
different like WikiaSearchBot or something, so we can loose the Grub
reputation?? and then be able to crawl many sites??

mark


On Jan 28, 2008 8:45 AM, jer <jeremie at jabber.org> wrote:


> > And one thing - maybe funny, maybe weird - but wiki projects
> > (wikipedia,
> > wikinews, wiktionary, wikibooks) have banned Grub client in robots.txt
>
>
> The URLs in the workunits are obeying and have been validated against
> any robots.txt, but we're also using a different user agent string.
> I also fixed this in the new batch of workunits, it's now properly:
>        User-Agent: GrubNG 20080128
>
> Any sites that are blocking because of the old Grub, we'll just have
> to explain that this is an entirely new and improved and fully obeys
> robots.
>
> Jer
>
> _______________________________________________
> Grub-dev mailing list
> Grub-dev at wikia.com
> http://lists.wikia.com/mailman/listinfo/grub-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.wikia.com/pipermail/grub-dev/attachments/20080128/13782ba8/attachment-0001.html