[Search-l] I'm confused...

Wed May 9 00:31:56 UTC 2007

	Well, the following is the sort of post that I might get in
trouble for writing, but I think it's a legitimate topic of discussion,
and my opinions are honest.

On Wed, May 09, 2007 at 08:10:33AM +0900, Ed Whittaker wrote:
> I suspect the following has been discussed before but I couldn't
> find it so can someone 'in the know' explain the following:

	I'm not "in the know", but I believe I have some insight, and
have been following the business-related issues.

> 1) If all algorithms, user data etc are available so essentially anyone can
> build a new search engine (presumably as good as Search Wikia if everything
> is indeed open) what is to stop anyone doing just that? If Google can
> incorporate anything useful into their own algorithms, data collection, UI
> they certainly will. Won't they simply be able to produce a faster,
> better(?) version of search wikia built on top of what they have already?

http://news.com.com/2100-1032_3-6180379.html
"Contributors will likely include graduate students as well large
companies that want to include search functionality in their products
but don't want to pay royalties to a search company, according to
Wikia CEO Gil Penchina. Another constituency will likely be smaller
search companies that don't have the time or money to do everything
required for a complete search service themselves.

"Everyone has to crawl the Web, but it costs a lot of money," Penchina
said. "There has been a lot of interest in academia for better tools."

http://www.newscientist.com/article/mg19325896.300-interview-knowledge-to-the-people.html

"Since news of this venture broke (see search.wikia.com) we have been
contacted by more than one second-tier company that develops search
engines. They recognize that acting individually they are going to
have a hard time catching up with Google, because Google has so much
money and so many great people.

What's your plan for search?

It's too early for specifics, but one thing that has worked is an
alliance in which people contribute to a free software project. We saw
this succeed with Apache, the open-source webserver. Apache was a tiny
group of volunteers, yet the vast majority of its code has come from
companies who paid people to work on it. It's essentially an
industrial consortium that has been able to fend off Microsoft's
closed-source webserver. So it makes sense for second-tier search
companies who are falling behind Google to contribute to a free search
software project that will make us equal to Google in terms of search
quality."

	So, the above is the *pitch*. I think it's a reasonable idea,
I hope they can pull it off. HOWEVER ... the difficulty of negotiating
such an agreement goes up dramatically with the number of investors
involved. And potential investors might be asking exactly the question
you raise, roughly, why should we put up the money to fund this, when
Google (or somebody else) might just grab the fruits of the research
without even paying? Heck, some Venture Capital types might be
wondering if Wikia is trying to get bought by Google! (not my view,
but you know some of those VC guys have to be thinking it, it's the
way they play a game with Other People's Money).

	One way to try to argue out of this dilemma would be that
those second-tier search companies need to try *something*, otherwise
they're very likely to continue losing market share. So at least by
signing up with Wikia's search project, they have a chance of getting
some innovation, some new wrinkle, they might be able to exploit
faster than Google. And it'll yield a *huge* amount of free
programmer labor for comparatively little money.

> 2) Why will users help in the first place? I'm definitely missing something
> here. The proposed model is very different to Wikipedia since it's Wikia
> that's making the profit - there's no way for a simple user to benefit
> monetarily by participating, is there?

	Ah, that one is very clear:

http://www.fastcompany.com/magazine/114/features-why-is-this-man-smiling.html

"The community has responded quite enthusiastically so far, catching
even Wales by surprise. "I just thought we'll put a couple of
developers on it and kind of play with it on the side and see what
comes up," he says.  "But now there's a huge developer community
that's really interested."  As Gil Penchina, Wales's handpicked CEO to
lead Wikia, says, "Since the news leaked out, people have been lining
up, saying, 'I'll clean the toilet bowl, let me in here.'"

	There is an *enormous* amount of emotion around Google that
can be tapped, people who want to be part of Killing The Champion.
If this were still the big bubble, there could have been an IPO,
getting people not only to work for free, but even to *pay* to do it.

> I'm not being pessimistic, I'm just confused because someone in Wikia has
> definitely considered these issues and decided they're not important.

	Oh no. All indications are that they are *very* important. But
the first one (funding) is very, very, hard to solve well.

> If someone in answering claims the "wikipedia works in practice but
> not in theory" line I guess we'll just have to see what happens when
> there's a profit motive involved.

	Actually it does work (for a small value of "work") in theory,
but people usually model it wrong. That's a different topic. I've
already taken the flames for this article, so I might as well cite it:

http://technology.guardian.co.uk/weekly/story/0,,2028328,00.html

-- 
Seth Finkelstein  Consulting Programmer  http://sethf.com/
Infothought blog - http://sethf.com/infothought/blog/
Interview: http://sethf.com/essays/major/greplaw-interview.php