[Search-l] Processing from mulitple indexes...

[Aerik Sylvan]

Sweet!  That's beautiful.  I can picture hundreds of of little brokers
serving up interesting bits of information, and being pulled into
meta-search results in the client, providing a richer and more up-to-date
experience for the searcher... very cool.  Tell you what, if there's any
interest, I've got small datasets at both wikidweb.com (domain level,
mostly, about 40k listings, GFDL) and tagthis.info (page level, don't know
how many off the top of my head, no license selected yet... gotta do
that...) that I'd be happy to publish via js api call, to help feed such a
thing.  Getting data from larger vendors (del.ico.us, stumbleupon, whoever)
would be better of course, but... :-)
Cool...

I've got this whole vision (I continue to think it's pretty close to your
vision) of a much more interconected, cross-pollinating internet, with data
shared, aggregated, ranked, and shared again much more freely...  Somewhat
off topic, I'm working on something simple and waaay compelling: RSS type
feeds for events, and then an aggregator/search engine to find events.
 (Think of it, garage sales, concerts, sports events, all published in an
atom feed, and searchable by date and location...) - I'm working on a
prototype/promotional website at http://eventfeed.org - I plan to open
source the (relatively simple) code in an effort for greater adoption.
 (It's really simple and rough right now, but basically works.)

Best Regards,
Aerik


On Thu, Sep 4, 2008 at 9:44 AM, Jeremie Miller <jeremie at jabber.org> wrote:

> Heh, that's almost exactly how the JS behind the current re.search
> stuff works, it's pulling in and merging between two indexes currently
> (nutch and KT contribs, and another recent idea is to build a third
> index of just page summaries so we can fit more into our limited
> resources).  One of our important principles is to try and use only
> open-source-built and freely-licensed indexes too :)
>
> The "widgets" (or whatever you want to call them) discussion is very
> close to this also, triggering the merge of other custom results into
> any search, and there's some working going on now to clean up the JS
> to support this even better.
>
> Jer
>
> On Sep 3, 2008, at 3:45 PM, Aerik Sylvan wrote:
>
> > Crossposting this - not sure of the best home for it...
> >
> > So, we have the the beginnings of code and a framework that really
> > facilitates multiple indexes, with factories and brokers and
> > collectors... How about moving the processing of the final result to
> > the client?  In other words, meta search via javascript.  You get a
> > lot of nifty things with that architecture - one of the big bits is
> > that it reduces the server resources needed to aggregate results.
> > It increases bandwidth, becuase you'd need to server perhaps the top
> > 100 results from each index you're querying, and let the client
> > whittle them down to the top 10 combined result.
> >
> > Anybody working/thinking in that direction?
> >
> > (Had this thought for awhile, but the news of Google Chrome's js
> > engine being 10+ times faster makes it even more intriguing).
> >
> > Best Regards,
> > Aerik
> >
> > --
> > http://www.wikidweb.com - the Wiki Directory of the Web
> > http://tagthis.info - Hosted Tagging for your website!
> > _______________________________________________
> > Wikia Search mailing list
> > http://re.search.wikia.com/
> > Change options or unsubscribe:
> http://lists.wikia.com/mailman/options/search-l
>
> _______________________________________________
> Wikia Search mailing list
> http://re.search.wikia.com/
> Change options or unsubscribe:
> http://lists.wikia.com/mailman/options/search-l
>



-- 
http://www.wikidweb.com - the Wiki Directory of the Web
http://tagthis.info - Hosted Tagging for your website!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.wikia.com/pipermail/search-l/attachments/20080904/36f0d61c/attachment.html