[Search-l] Sorry to do this but its coming, yes a rant :-(

[jer]

Ask my kids would say, "burrrrrn!" ;-)

I feel and share your frustration, I know at least for my part I've  
done a poor job keeping going a steady stream of updates on the  
things I've been playing with.  Of course, I've never personally been  
good at regularity, tend to step back and push things out in  
spurts... all I can do though is try to catch up for the last month  
or so of various chaotic activities.

First, Grub is partially stalled right now as we began in Feb the  
migration to the NG version, who's server side is back-ended by  
hbase, a very alpha open source bigtable clone written atop hadoop.   
That process has been painfully sputtery, a cluster of ~7 servers is  
maxing out at a few mil docs, not nearly good enough to push GrubNG  
to grow significantly.  Just last week though I think I came up with  
a way to partially work around the challenges of using hbase before  
it's ready, so with luck in the next week or so we can get that  
distributed crawler growing again :)

I mentioned Grub/hbase first as while working on it I spent the time  
to really think about the bigtable model with hbase and how we can  
use it, I even went so far as to design and build a system I called  
KT (for Keyword Tuple, it's in the re.search svn repo), that should  
be able to scale up while supporting a huge variety of really  
exciting features.  I even prototyped an implementation of both the  
server and a search result interface with a bunch of features which  
those that I IM with regularly have seen.  I've hesitated to show it  
off to a wider audience as the prototype was both horribly insecure  
and unusable unless someone showed you first.

There's a light at the end of the tunnel though, I've attached a pdf  
of a ROUGH DRAFT that I hope we can actually let everyone start  
playing with in as little as a week.  Here's a quick list of the  
various features that KT will be enabling in our search results:

	- add new result
	- edit any result's title/summary (and see revisions, revert)
	- highlight any single result as the best one
	- star ratings (already exist)
	- delete/trash any result (leaves the title but grey'd out, can be  
undeleted easily)
	- full change history for that set of results, and rss feed to watch  
for changes
	- add comments to any result
	- select text, images, or input forms from the target site to be  
shown below it's search result
	- alternative related searches and did-you-mean (user driven)
	- custom backgrounds in the header for given searches

I've found myself doing a bunch of these things because it's simply  
*fun*, so if anyone just can't wait and is willing to suffer through  
some UI or random breakage pain, just ping me on IM and I can  
accommodate some you've-been-warned early access to play.

It's possible to do lots more than what's above as well, KT is a very  
simple and flexible framework.

Besides helping with the things above, I know Dennis and Seth have  
been working hard on improving the index, the result quality has  
improved significantly and our infrastructure/systems are a lot  
healthier than how they ended up in the mad rush to get it ready for  
that crazy first week (we did 5mil queries the first day and didn't  
break, not bad).

As a reminder in general though, this is overall a long term project,  
there's going to be lulls and storms.  It's also a pretty insane  
project, and insane people seem to be rather poor organizers and  
communicators, lots will be broken and go un-addressed for way too  
long, guess we're well on track for something :)

Jer
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Search Wikia - Web Search Results.pdf
Type: application/pdf
Size: 340992 bytes
Desc: not available
Url : http://lists.wikia.com/pipermail/search-l/attachments/20080326/3c411108/attachment-0001.pdf 
-------------- next part --------------

On Mar 26, 2008, at 5:13 PM, Mark (Markie) wrote:

> Right, im afraid the time has come once again where i have been  
> wondering to my self again, and i feel that things need to be said,  
> so here they are.
>
> *Whats happening with the project.  AFAIK overall (and i know  
> somethings have happened) but *very* little seems to have happened  
> since the launch.  Now i know that things are probably happening  
> with the team, but any chance of actually telling the users about  
> this, cos its not looking good from here atm.
>
> Ive copied in the so called pillars of search
>
> Transparency - riiiiiiight :-(
> Community - hmmm contribute to stale projects?
> Quality - well....
> Privacy - hmm yes that seems to have been done to an extent ( by  
> the community mind)
>
> Ive been on the project since dec 2006, and so have been waiting  
> along time for this to happen, so its not purely a case of i want  
> everything to happen NOW, i just want it to look like SOMETHING  
> will happen SOON.
>
> *This brings me onto the next topic of where is the project  
> going??? There has been practically no progress, and frankly i cant  
> see much being done from my point.  The launch has happened, many  
> people were interested, contributed but have now left, because  
> NOTHING has happened. so overall the net gain of launching the  
> project?? bad press and a few (relative to the web) minis.
>
> *Many things have been promised by various people, which havent  
> happened. Most specifically this has come from a certain member of  
> staff, one specifically, that has said that they will do many  
> things, but even the most basic of tasks seem to have not happened.  
> so Broken/missed promises. Well iirc (name here) said he would make  
> sure that the about pages etc were created, hmm... (http:// 
> alpha.search.wikia.com/about.html in case you forgot where those  
> were).  This is a wikia project, any chance of getting ANY  
> involvement/input/co-ordination from the team who, ultimately, want  
> us to make them more successfull and a profit (if were being frank).
>
> Now i know i havent been that active recently on the wiki, but i  
> have been reading the mailing lists and talking in irc, but the  
> main reason for me not being active on the wiki, is mainly the fact  
> that i just dont have the motivation to do anything because of the  
> above.  Frankly atm its a stale project, but hopefully this rant  
> (which i hate doing) will mean that the project will hopefully  
> become better.
>
> If i have offended anyone above then i am sorry, but i feel that  
> certain things need to be said right now, in order to make the  
> project better, which is my aim.
>
> Many thanks and look forward to the responses to this, especially  
> from wikia staff
>
> Regards
>
> mark
>
> (user:Markie)