[Search-l] wrong questions

[Jeremie Miller]

Excellent meaty thread Paul, cool :)

As asked I won't touch on the meta-topics, but on the tech side I at  
least want to draw an analogy: I'm seeing this process of building an  
open search platform evolve as more of a frankenstein, combining  
protocols, code, piles of data, lots of experiments, and gobs of  
zelektrisity!  I also think we're a ways away yet from seeing the  
vision really start to come together, I didn't expect this to be easy,  
*g*.

/me goes to tend to his pile of half working code and formats...

Jer

On May 11, 2008, at 5:53 PM, Paul Vixie wrote:

> various folks here, as well as sethf in his blogs, have asked plenty  
> of hard
> questions about wikia.com's search project -- who owns it, is it  
> truly open,
> what is wikia's agenda, and so on.  these are the wrong questions,  
> since they
> focus on the company who sponsors the work, and not on the work  
> itself.  i've
> been trying to think of the right questions.  here's an  
> approximation of what
> critics of this project ought to be investigating.
>
> 1. why is ISC's the only backend?  jer's vision is backend  
> syndication, so,
> if his XML schema is stable and if there's at least one f/l/oss  
> implementation
> of crawling and of indexing, then, why aren't there more crawlers  
> and more
> indexers, conforming to jer's XML, possibly flooding data between  
> each other
> and possibly dividing up the workload so that all crawlers don't  
> have to
> crawl all sites?  ISC ought to have peers, and we ought to be able  
> to have
> gentlemen's agreements, like, "we'll do [a-l].com, you do [m- 
> z].com", etc.
>
> 2. why is Wikia's the only frontend?  again referring to the  
> syndication
> model, and knowing that there are other "social search engines",  
> when will
> we see someone other than wikia use ISC's backend, or any other  
> backend whose
> data can be reached using jer's XML?
>
> 3. who is driving the syndication model?  it's clear that ISC knows  
> how to
> provide network and power, and that jer knows how to design the  
> system and
> build various parts of it, but who is the champion for jer's vision  
> -- who
> will drive us to better answers for #1 and #2 above?  who ought to  
> be in here
> answering critics and beating the drum, which is a distraction to  
> jer (and
> candidly he's too busy to do this part well unless he drops other  
> stuff
> that's already late)?  remembering that jimbo keeps this issue alive  
> in the
> press, the overall project still lacks a day to day "programme  
> manager".
>
> 4. what else is jer working on?  has wikia dedicated him to this  
> project or
> does he also handle day to day fire fighting on wikia's existing  
> services to
> justify his paycheck?  and while we're on that topic, what other  
> personnel
> has wikia dedicated to this -- how seriously are they really taking  
> it, in
> terms of cash on the barrel head?
>
> 5. who else is working on this, outside of wikia?  what outside  
> volunteers
> or wikia competitor's employees have commit access to the source  
> pool for
> the crawler, or indexer, or front end, or have root access to the  
> donated
> back end machines hosted by ISC?  if the answer is nobody, then is  
> that due
> to lack of outreach (see #3 above) or is it wikia's preference that  
> outsiders
> contribute content rather than code and sysops?  (is that written  
> anywhere?)
>
> 6. where are the mini-articles stored?  if outside volunteers are  
> mostly
> contributing data, is that data stored on wikia's front end?  if so,  
> what are
> the redistribution terms -- would wikia flood this data to competing  
> front
> end operators, and accept incoming floods of similar data from  
> competitors?
> or, is this the "secret sauce", there's no way to get access to  
> contributed
> data of this kind except one article at a time, inside wikia's  
> advertising
> system?
>
> 7. given that the idea of "taking on google" is silly, given their  
> size and
> focus and ambition and brand strength and so on, and that what we can
> actually hope to achieve with this project is to change the game and  
> make
> search part of the internet infrastructure, where are the white  
> papers,
> journal articles, and outreach glossies explaining what the new  
> world of
> internet search could look like, and what effect this change will  
> have on
> google, microsoft, yahoo, and the current market hierarchy, and the  
> rest of
> the "social search" scene?
>
> 8. has anybody reached out to yahoo and microsoft to see if they'd  
> like to
> join this effort or at least sponsor it, since as #2 and #3 in  
> internet
> search today, they're the ones with the most to gain if we change  
> the game.
> and if nobody's doing this now, and i did it, what would wikia say  
> about
> sharing the sponsorship burden with other players, perhaps larger  
> players?
>
> this list of questions isn't meant to be exhaustive.  but as in my own
> controversial efforts over the years, i find the quality of  
> criticism here
> somewhat low.  forget about jimbo and his tv news girlfriend and the  
> instant
> messenger chat logs.  forget about wikia's corporate interests, or  
> whether
> wikipedia was a once in a lifetime event, or whether wikipedia's  
> admins are
> running amok, or what the wikimedia foundation board is up to.   
> there are
> plenty of excellent questions about the wikia-sponsored search  
> project whose
> backend is hosted at ISC, which are not salacious or even delicate.   
> the
> above list is meant to show the kind of questions i mean.
>
> for the record, i'd like it if jer did not jump in and answer any of  
> these,
> because he's got more important stuff to do, and because i'd like to  
> see
> wikia provide another face, another name, another voice, to this  
> forum.
>
> also for the record, ISC's hosting of this project has been a cash  
> neutral
> event for us, which is important since we don't have cash for this  
> kind of
> thing.  the 15-ton air handler wikia bought feeds a room that has  
> other
> projects in it too, and our network is a fixed cost, and wikia has  
> agreed
> to pay for the power we use for search, and the servers were all  
> donated,
> and that donation was targetted for this project, and we got a lot  
> more
> servers than we needed, and we've been passing the excess along to  
> other
> f/l/oss and internet security projects.  so no matter whether this  
> project
> changes the world, ISC is already winning.
> -- 
> Paul Vixie
> _______________________________________________
> Wikia Search mailing list
> http://alpha.search.wikia.com/
> Change options or unsubscribe: http://lists.wikia.com/mailman/options/search-l
>