[Search-l] Sorry to do this but its coming, yes a rant :-(

[Aerik Sylvan]

On Mon, Mar 31, 2008 at 8:08 PM, Dennis Kubes <kubes at apache.org> wrote:
>
<snip lots of context>.  One of the
>  problems I see is being able to determine if a change actually improves
>  search results (besides just looking at the results).  Any ideas on how
>  to determine this?
>

At the risk of being dense/overly stubborn:  direct feedback coupled with
indirect feedback?  The direct feedback would be a score given by willing
participants.  Indirect feedback is clickthroughs, stuff like that.  I
really think the area where wikia search has a chance to shine is in how we
engage participants in the process.  I'm *sure* google and the other big
guys collect indirect feedback.   They've got an army of scientists.  We
probably won't beat them at that game, but we shouldn't ignore indirect
feedback and pagerank style algorithms either.. BUT, where we can be
diiferent - maybe even better - is in engaging the direct involvement of the
community.  I don't mean picking the entrenched results for a given search,
as Mahalo is doing (BORING!  I *still* say it's "just" a spin on the dmoz
approach - not that it's bad, it just isn't new or impressive any more) but
instead engage the community to continually provide feedback on results and
keep the results *fresh*.  Incorporate ideas like Alexas movers and
shakers<http://www.alexa.com/site/ds/movers_shakers>.
I have always like how netfilx asks you how you liked a movie, after you've
returned it.  Maybe we can think of better methods.

The trick is going to be volume.  If you have 10 million searches, and 1
percent of the searches have active feedback, that's 100,000 data points.
Given an assumed very wide spread (that 100,000 points will have a very
small head and a very long tail), it is hard to draw statistically
significant conclusions.  So, you need either a) more searches or b)a higher
percentage of feedback.  Not sure how to tackle that, but if any project
ever was going to do it, it's this one.

Best Regards,
Aerik


--
http://www.wikidweb.com - the Wiki Directory of the Web
http://tagthis.info - Hosted Tagging for your website!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.wikia.com/pipermail/search-l/attachments/20080331/bf632754/attachment.html