[Grub-dev] LZMA support in upload server

[Bartek Jasicki]

On 2009-02-25, at 18:57:10
Balinny <balinny at gmail.com> wrote:

> > > It's a two-minute change. Specially given that the last entry
> > > must be generated on-the-fly anyway.
> >   
> >
> > That same like run again new workunits with Accept header ;) If i
> > good count, this 2 minutes take now around 1 month ;) 
> Workunits are pre-generated, but I think the last entry (containing
> the user name) is made on request?
> 

AFAIK, yes - http://svn.swlabs.org/grubng/trunk/perl/dispatch.cgi


> > Unfortunately Jeremie
> > don't have free time for made any changes in current dispatch
> > server. So probably this 2 minutes task must wait few months before
> > i start work on new dispatch server (at this moment i don't have
> > access to dispatch server).
> >   
> What really does the dispatch server? Which interface does it use to
> grab the urls?
> 

Dispatch server: for me, this is server which send workunits to users +
workunit generator + robots.txt checker (i put all this things in one
bag).

About urls: AFAIK (or again something was changed without public
announcement ;) Jeremie, please fix me if i wrong) Grub have own
database with URLs from which workunit generator get URLs to fetch.
And only one way to get new URLs in system are sitemaps generated by
clients or send by users (during creating workunits, Grub not connect
to Nutch).

> >> As a bonus, also add  Accept-Encoding: gzip, lzma
> >>     
> >
> > 1. Where add it?
> >   
> To the last workunit entry.
> 

But Accept-Encoding is only for response from server, thus if you add
this, server send you compressed information about .arc file ;)

> > 2. IIRC LZMA isn't registered compression for any HTTP version and
> > cannot be placed in HTTP headers.
> >   
> Neither is bzip2 or sdch.
> Treat that as an extension to the protocol :)
> 

Okies ;)

Bartek

-- 
Grub Next Generation: http://grub.org
Mailing List: grub-dev at wikia.com
IRC: #wikia-search at irc.freenode.net
Jabber: thindil at jabberpl.org