[Grub-dev] do we even really need a native client

[Balinny]

jer wrote:
> I just checked this into subversion and will attach it here, if you 
> know perl hopefully it isn't too obtuse to read and get an idea of how 
> it works, which is very very simple (compared to the old SOAP interface).
>
> I'm trying to get a new dispatch.grub.org officially set up and a 
> whole bunch of work units available under it.  It'll just use HTTP 
> basic auth and probably 250-url sized work units.  It should be really 
> really easy to code grub clients in any language or environment :)
>
Nice! :-)
However, as i'm not a fluent speaker of Perl, i went to create a C 
version :-)

I found a number of doubts:
*At the perl version you do print $arc "http://$host$path $ip 
19691231175959 $ctype ",length($body),"\n$body"; so the end of previous 
page body touches the start of next chunk, but from 
http://www.archive.org/web/researcher/ArcFileFormat.php a \n would be 
required in front of http://

*$body contains both headers and body. Separated by CRLF CRLF or only CRLF?
*What should the client do if the dns doesn not respond / can't connect to host / host doesn't answer?  Add an entry with size 0? Not include it on the index?

Include a fake answer as the perl client does?
500 Can't connect to www.nantah.com:80 (Bad hostname 'www.nantah.com')
Content-Type: text/plain
Client-Date: Wed, 09 Jan 2008 16:53:42 GMT
Client-Warning: Internal response
Should the fake Client-* be present?

*What to do with Transfer-Encoding: chunked?
**Remove the header + decode body
**Keep header + decode body
**Keep Header + keep body
(perl renames header to Client-Transfer-Encoding and decodes body)
I vote for Keep header + decode body, so that allows the client to do 
things like fetching the content compressed but preserve original data 
for the server.

*Is the client allowed to change the url positions in the file? Say a 
web is down on first fetch and is retrieved later and appended...

Some comments about the perl version (using Yousef's)
There's an unterminated string. at line 78. It should be print 
$res->status_line,"\n"; or you get an error on line 96: print STDOUT...
The final call to upload_archive(); is wrong. It is already called at 
line 52.
At lines 11 and 12 $reg and $ua are wrongly defined as local, so they 
aren't available for upload_archive
Grub user-agent isn't being used, but added as an "Agent" header. Is 
that intentional? i changed to:
          foreach my $header (@headers) {
            $host = $1 if($header =~ /Host\: (.+)$/);
            $useragent = $1 if($header =~ /(User-)?Agent\: (.+)$/);
            }
          $ua->agent($useragent);