<span class="gmail_quote"></span><br><div><span class="q"><span class="gmail_quote">On 6/2/07, <b class="gmail_sendername">Jason Calacanis</b> <<a href="mailto:jason@calacanis.com" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
jason@calacanis.com</a>> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>The even bigger problem is that the folks who have the best<br>information to crawl--folks like delicious and google--do not allow<br>metasearch and would take action if you sucked their data into another<br>dataset that competed with them.
<br><br>They would, of course, have a very good point: it's not very fair to<br>build a business overnight by indexing their information. Folks have<br>been trying to do this to Craigslist and Craig Newmark has blocked
<br>them.</blockquote></span><div><br>Exactly the point I'm making. There's a tremendous amount of pretty good human generated data out there ( I would argue that, at a minimum, the categorization of urls in dmoz is of some value, anyway), and integrating it would be a great way to get a decent signal to noise ratio. I think the wikia search has the capability to create some novel and interesting filtering and weighting mechanisms, but all by our lonesome, it will be difficult and timeconsuming to develop a really interesting dataset.
<br><br>So, since Wikia is a for-profit venture, perhaps it makes sense to look into licensing some data from closed providers (stumbleupon, for instance - since <a href="http://del.icio.us/yahoo" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
del.icio.us/yahoo</a> is unlikely to want to feed a competing search engine).
<br><br>Aerik<br></div><br></div><br>