[Search-l] Numbers (Using Grid Computing for Wikia Project)

[Seth Finkelstein]

On Fri, May 11, 2007 at 10:27:49AM +0200, Michael Christen wrote:
> there is enough bandwith for every home-user. no problem. you need
> only a fraction of that what you use for file-sharing with other
> software.

	Just curious, does anyone have good, recent, numbers for what's
required these days for a hobbyist search index? That is, say someone
is a "power user" who could dedicate a reasonable desktop-class machine.
Not a supercomputer, but a setup around 4GHz CPU 4GB RAM 500 MB disk,
with a home broadband connection. How well does this work for a text-only
crawl of the web, to get something useful, and how long would it take?
That is, after crawling X days, you can expect a usable index of Y
documents, which would use Z amount of disk.

	Anybody on this list know?

-- 
Seth Finkelstein  Consulting Programmer  http://sethf.com/
Infothought blog - http://sethf.com/infothought/blog/
Interview: http://sethf.com/essays/major/greplaw-interview.php