Instead of publicly crawling the human indexes (<a href="http://del.icio.us">del.icio.us</a> / stumbleupon, etc.) ourselves, why don't we have our users to do it? It's not a complete work around, but might be an approach that works for a bit. Here's how I envision it:
<br><br>* A client side crawler (similar to yacy, but not targeting the entire web, just metadata rich places) implemented through a Firefox extension or Greasemonkey script.<br>* When a client visits a social network with valuable data (as determined by a list managed by us) their local client makes a copy of all the data delivered to their client side browser.
<br>* The server can't tell the difference between a user surfing with the extension or without the extension. <br>* That data is then meta-tagged and packaged properly locally and sent to the Wikia Search servers from the client's machine.
<br>* The Wikia Search servers then index and make sense of all of this data culled from the various clients running the Wikia Search client / extension.<br><br>This way we're able to work around the bandwidth concerns that Yahoo and company would have with us crawling their databases. And the data that we're getting is merely stuff that is being browsed naturally, by live humans, so it's likely of more value.
<br><br>But bandwidth is obviously not just what they're concerned with. As has been mentioned, these sites view these databases of useful human tagged information as enormously valuable assets that give them a competitive edge. So then it's a question of the "intellectual property" contained in those databases. Now, I'm not sure if other networks do this, but
<a href="http://Del.icio.us">Del.icio.us</a> and Flickr have Creative Commons license implementation. That means that a particular user's stream of content that they've created (links, photos, etc.) can be set for people to share it. I think this would be a perfect opportunity for our distributed crawlers to take advantage of.
<br><br>Thoughts?<br><br><br>Fred<br><br><br><br><br><div><span class="gmail_quote">On 6/3/07, <b class="gmail_sendername">Nitin Borwankar</b> <<a href="mailto:nitin@borwankar.com">nitin@borwankar.com</a>> wrote:</span>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">jer wrote:<br><br>>> So, here it is: Getting data from existent social bookmarking
<br>>> services may be an option we should consider. Think of it -<br>>> aggregating data from <a href="http://del.icio.us">del.icio.us</a> <<a href="http://del.icio.us">http://del.icio.us</a>>,
<br>>> stumbleupon, etc. Now, I can't imagine how we'd get Yahoo to<br>>> give us the data from <a href="http://del.icio.us">del.icio.us</a> <<a href="http://del.icio.us">http://del.icio.us
</a>>, but maybe<br>>> there are other providers who would be willing to do this. Or<br>>> perhaps we look at paying them for it, at least enough to cover<br>>> their bandwidth and other overhead.
<br>>><br>>> Anybody got an ideas around this type of thing?<br>>><br>>><br>>> Yeah.. its a good way to find the actual interest of the people thru<br>>> social book marking, digg and many other social websites.. But it all
<br>>> matters whether they are ready to release data open to such open<br>>> source search projects..<br>><br>><br>> For the most part, all of those sites and all of that data *is* open,<br>> it just needs to be intelligently crawled and indexed. They're great
<br>> seed sites for keeping a crawler fresh.<br>><br>> Sure it would be nice to have it in a more digestible form, but it's<br>> all there already :)<br>><br>> Jer<br><br><br>Hi All,<br><br>I did some work for a university professor who is trying to tackle the
<br>problem that publishers of technical periodicals own the bibliography<br>citations in articles. However individual researchers own the<br>bibliographies of their own publications, so by aggregating the<br>bibliographies of individual researchers one can build an alternate open
<br>source of bibliography data. Apply the same principle to del.ici.ous,<br>stumbleupon etc data.<br><br>Individuals who have data on these services can individually and<br>voluntarily copy their data out of those systems and into any public
<br>aggregation<br>of such data. Now that Yahoo has BBAuth - a single-login authentication<br>service, one could build a single web page where such a volunteer<br>individual could go and authorize the download of their del.ici.ous
data<br>into their account(s) on any other web service(s).<br><br>There is no need to crawl the web pages and get into the arms race of IP<br>blocking etc. that will naturally come up.<br><br>The bigger picture here is that we as individuals own our own data and
<br>we should not let it be captive on web applications, rather we should be<br>able to aggregate it wherever we choose - and if we should choose to do<br>so we should be able to very simply push a few buttons and have *our own
<br>data* transferred between web applications.<br><br><br>Nitin Borwankar.<br><br><br><br>>------------------------------------------------------------------------<br>><br>>_______________________________________________
<br>>Search-l mailing list<br>><a href="mailto:Search-l@wikia.com">Search-l@wikia.com</a><br>><a href="http://lists.wikia.com/mailman/listinfo/search-l">http://lists.wikia.com/mailman/listinfo/search-l</a><br>>Change options or unsubscribe:
<a href="http://lists.wikia.com/mailman/options/search-l">http://lists.wikia.com/mailman/options/search-l</a><br>><br><br><br>--<br><br><br>Nitin Borwankar<br><br><a href="http://walruscarpenter.wordpress.com">http://walruscarpenter.wordpress.com
</a> Of shoes and ships and sealing wax of cabbages and kings<br><a href="http://greener.com">http://greener.com</a> Find, Learn, Act .... Greener, the search engine for the planet<br><a href="http://tagschema.com">
http://tagschema.com</a> Implementation of tag database applications<br><br><a href="mailto:nitin@borwankar.com">nitin@borwankar.com</a><br>510-872-7066<br><br><br>_______________________________________________<br>Search-l mailing list
<br><a href="mailto:Search-l@wikia.com">Search-l@wikia.com</a><br><a href="http://lists.wikia.com/mailman/listinfo/search-l">http://lists.wikia.com/mailman/listinfo/search-l</a><br>Change options or unsubscribe: <a href="http://lists.wikia.com/mailman/options/search-l">
http://lists.wikia.com/mailman/options/search-l</a><br></blockquote></div><br>