[Grub-dev] do we even really need a native client

[jer]

> However, as i'm not a fluent speaker of Perl, i went to create a C
> version :-)

I was tempted to do that too, but it'd be even more shaky than the  
perl version given the time :)

> I found a number of doubts:
> *At the perl version you do print $arc "http://$host$path $ip
> 19691231175959 $ctype ",length($body),"\n$body"; so the end of  
> previous
> page body touches the start of next chunk, but from
> http://www.archive.org/web/researcher/ArcFileFormat.php a \n would be
> required in front of http://

Perhaps I mis-read it!

doc == <nl><URL-record><nl><network_doc>

URL-record-v1 == <url><sp>
<ip-address><sp>
<archive-date><sp>
<content-type><sp>
<length><nl>


So, there should be two newlines? "\n\n$body" ?

> *$body contains both headers and body. Separated by CRLF CRLF or  
> only CRLF?

It should be the exact response, CRLF CRLF separating them.

> *What should the client do if the dns doesn not respond / can't  
> connect to host / host doesn't answer?  Add an entry with size 0?  
> Not include it on the index?
> Include a fake answer as the perl client does?
> 500 Can't connect to www.nantah.com:80 (Bad hostname 'www.nantah.com')

Yep, for now just generate a 500 error with a human-readable  
explanation of the error.  Very similar to what a proxy does for  
these kinds of errors.

> Content-Type: text/plain
> Client-Date: Wed, 09 Jan 2008 16:53:42 GMT
> Client-Warning: Internal response
> Should the fake Client-* be present?

Not required, though including a Client header to show which IP  
resolved for the name might be nice for post-processing or tracking  
down issues later on.

> *What to do with Transfer-Encoding: chunked?
> **Remove the header + decode body
> **Keep header + decode body
> **Keep Header + keep body
> (perl renames header to Client-Transfer-Encoding and decodes body)
> I vote for Keep header + decode body, so that allows the client to do
> things like fetching the content compressed but preserve original data
> for the server.

Easier, rfc 2616: "A server MUST NOT send transfer-codings to an HTTP/ 
1.0 client" and we're doing 1.0 requests.

> *Is the client allowed to change the url positions in the file? Say a
> web is down on first fetch and is retrieved later and appended...

Nope, time out and insert an error, needs to be sequential.

Jer