Bartek Jasicki wrote: > > My English is little rusty, thus I'm not sure if i good understanding ;) > Client fetch only pages, don't look at robots.txt? Server check > robots.txt before add url's to workunit? > > Bartek Yes. It's a bit strange that precisely you are asking that, given that the C# client (or any of them) don't check robots.txt ;-)