[Grub-dev] soap upload server very unstable

[ab]

hello,

Bartek Jasicki wrote:
> This is not problem with server. Crawling looks like:

i wouldnt be so sure about that... ;)

> 1. Client download workunits with urls
> 2. Client start crawling, write to .arc file informations (not only url
> or host ip but page length too and few other things)
> 3. Client send work to server

now here seems to be a problem, and in my humble knowledge there seems 
to be a possiblity that the server interferes with the upload from the 
client.

please see the last comments of my bug at:
<http://dev.grub.org/cgi-bin/trac.cgi/ticket/9#comment:11>

today i had a nasty crawl here, with workunit and the .arc and .arc.gz 
resultfiles, which the soap server dont seem to accept. the resultfiles 
are 14 megs uncompressed, the .gz file is still 13megs.

the client is only able to send something between 7 and 8 megabytes 
worth of upload traffic (bytes measured) to the soap server, then 
something seems to go very wrong and today the client (the same build 
from yesterday i have been using "successfully" crashes in the middle of 
the upload process...

i dont even get the errorreply from the server any more that the results 
are invalid or anything, the client simply crashes as it never manages 
to send the correct amount of upload data (.arc.gz filesize)

> 4. Server check integrity of .arc.gz file (about amount of url, length)
> and filename

i never get to this step (with this very workunit and resulting .arc and 
.arc.gz files) today.... no way. i have tried many times by now.

as i wrote in my bugreport, i am keeping the workunit, arc/arcgz and 
some logfiles as a backup, but this is all to no avail at the moment.

if anybody gonna doublecheck on that soap server and the tech/platform 
it is being ran on, maybe some networking issues, resultsizelimiation, 
whatever else there might be.

oh one more question, what about the performance of the soap server 
(networking bandwidth), the soap-upload maxes out at like slightly over 
1megabit/s, at maximum like 140kbytes/sec of upload traffic, and my line 
can do much better than that in upload direction. the uploadcurve is 
pretty much flat, and it seems as if that was some kind of server/thread 
  limitation in your soap/server config. or something.

cheers.