[Search-l] Architecture for Self Scaling Search

[Philip Haynes]

Hi,

 

I have only recently been lurking in this community. I hope my question
isn't out of school or already well understood. 

 

I have been developing / researching architectures to support very large
scale reliable computing at low cost. It would appear search is a good
candidate application. The purpose of this note is to solicit interest in
collaborating with me to flesh out and prototype a specific Search
architecture for Search. 

 

The architectural model I have being used is self-scaling (as per figure
below). The goal is being able to treat a network of computers as a logical
computing space. The implementation integrates HTTP servers & clients into a
single execution space using an asynchronous I/O engine. With this approach
the capability to serve 10,000+ concurrent HTTP connections and maintain a
multi-gigabyte throughput has been demonstrated on a single device. Combined
with technologies such as ZFS, means terra bytes of content can be cheaply
stored, search and served. This enables 100-1000x lowering in the cost to
serve a web request, but with a highly reliable configuration. The
capability to serve 1/7th of Australia internet traffic with a single $2,500
PC was recently demonstrated.

 

My thinking for search was that if a proxy server & client was deployed on
each device, content could be pre-downloaded and served locally. Anyone
using the search service would have a much faster internet experience. This
content could also be processed and monitored in real time for actual usage.
The communities accessing sites would do most data aggregation, only finally
transferring source data to a much smaller number of root computers. Network
& computing costs for search are thus mostly offloaded to the edge of the
network. A better service that improves the more people use it, but at a
decreasing or static cost. 

 

If this is an approach worth pursuing, I would appreciate the feed back.

 

Regards,

Phil Haynes



Figure 1 Self Scaling computing

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.wikia.com/pipermail/search-l/attachments/20070809/23461f66/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 15101 bytes
Desc: not available
Url : http://lists.wikia.com/pipermail/search-l/attachments/20070809/23461f66/attachment.jpe