I couldn't agree more.... if you look at the amount of time Google, Yahoo and Microsoft spend working with websites to improve indexing and the groups they have working improving results to query's through hired communities, I am not sure there is room to play. It would seem more reasonable just to mash in an engine when you need one and let community take on the short tail. Almost like an open
<a href="http://About.com">About.com</a> approach just opening up to anyone... maybe use <a href="http://wikia.com">wikia.com</a> as the knowledge store to mine (although I don't like the break from UI constancy when your trying to organize the entire web).
<br><br>I bet I could project out how Google would do it... Move Google docs over to a global wiki with a constant UI allow people to create an article on anything (nuggests, pointers where stuff can really be found.... whatever) and then index it and create a new presense that allows you to look there first before turning to a web crawl. Pretty easy to envision the interface and see how powerful it would be if you could get people to organize information not into a encyclopedia but into match of every query to article that acts as a real pointer. From there it's just a matter of writing a great trust API....
<br><br>Why try to reinvent the wheel when it really doesn't need to be? With enough enough community involvement over time a web crawl would become less and less relevant. <br><br>Just my two cents....<br>Seth<br><br>
<div><span class="gmail_quote">On 7/31/07, <b class="gmail_sendername">John McCormac</b> <<a href="mailto:jmcc@hackwatch.com">jmcc@hackwatch.com</a>> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Jimmy Wales wrote:<br>> One of the first jobs for the OS version of the client is to make<br>> absolutely 100% sure that it behaves itself exquisitely well, both for<br>> the clients and for the sites being crawled.
<br><br>Unfortunately it is not a question of the behaviour of Grub or any other<br>crawler. The owners of large directories and sites tend to be far more<br>aggressive now in protecting their resources. That means that many of
<br>them are tired of scrapers and bots and will block anything outside the<br>Google/Yahoo/Microsoft crawlers. Others have blocked entire countries at<br>IP level or by extension.<br><br>> I think this misunderstands how Grub works. Grub distributes the
<br>> crawling and checking to see if sites have changed, it does no<br>> distribute the decisionmaking about which sites to crawl. In this sense<br>> it is much more like Seti@Home than like Gnutella networks or the like.
<br>> It is "distributed" not "peer to peer".<br><br>Again this runs into the "shoot on sight" attitude of some webmasters.<br>The crawler will be seen as coming from dynamic/dialup IP ranges, many
<br>of which are already iffy due to scrapers. With the main search engines,<br>the IPs have proper reverse DNS so that webmasters can be certain that<br>they are who they claim to be.<br><br>> And YES you are 100% right - crawling is only a piece of the search
<br>> solution. In theory a distributed crawler can spider the web more<br>> quickly and thoroughly than a centralized solution. And another part of<br>> the theory here is that be reducing the *cost* of a high quality crawl,
<br>> it becomes possible to make the *results* of the crawl available under a<br>> free license. (Which, of course, Wikia will do no matter what the cost,<br>> because that's the whole point of what we are doing here.)
<br><br>In June, I spidered the index pages from all active .eu websites from a<br>tracking dataset of .eu domains (approx 1.436M websites out of 1.78M<br>actively resolving domains from a list of 2.13M .eu domains). The aim
<br>was to create some estimate of how many active .eu websites there were.<br>The results were quite startling - only about 16.13% of the domains with<br>websites (roughly 19.90% of the websites) were actively developed. The
<br>data was then broken down over active websites, parked sites, holding<br>pages, frame src redirects etc. A similar first run on .mobi had only<br>10% of the websites actively developed and that was before any dupe and
<br>holding page algorithms were applied to the data.<br><br>The problem with building a good index is that this kind of work is<br>never really seen or heard about. The enthusiasts tend to think that<br>they know how search engines work and, to a certain extent, they do. But
<br>they do not appreciate what goes into creating and maintaining a high<br>quality search index. This process has to be highly automated to be<br>successful as handling millions of websites is not something that can be<br>
done efficiently by hand.<br><br>The reason that most of these mini search engines fail after eighteen<br>months or so is because they run into the brick wall of the acquisition<br>problem. (Similar to that of the web directories that rely on user
<br>submissions.) They have to compete with search engines like Google that<br>are far better equipped and URL detection is not the most efficient way<br>of detecting new sites. Many new sites are not linked. It often takes
<br>some time for the linkbacks to appear in directories. And since Google<br>has the greatest footprint, the site owners will often submit them to<br>Google. This gives Google a major head start on the dwindling number of
<br>active web directories.<br><br>The cost of a high quality crawl is probably a magnitude or so lower<br>than those estimates that have been published. Most of the ones I've<br>read fail to take into consideration the numbers of duplicate, PPC,
<br>holding pages and assorted junk in an extension. This is the stuff that<br>is removed in the pre-index process. They extrapolate the number of<br>domains to the number of websites and work from there. The reality is<br>
that the webspace of most extensions is like a large, bumpy plain with<br> a handful of skyscrapers and a lot of small tents. The interesting<br>thing is that the ccTLDs tend to be different to the TLDs like .com etc.<br>
The Irish .ie extension had an active development figure of<br>approximately 57%. I haven't worked out a figure for .uk yet but I would<br>expect it to be somewhat higher than that of .com or .eu.<br><br>Most of the work in a high quality crawl actually goes into building a
<br>high quality index as its starting point. It is then a process of<br>continual refinement. This is why I tend to wonder about distributed<br>search when there is no corresponding thought being put into the<br>critical question of "searching for what?".
<br><br>Regards...jmcc<br>--<br>******************************************************<br>John McCormac * e-mail: <a href="mailto:jmcc@whoisireland.com">jmcc@whoisireland.com</a><br>MC2 * voice: +353-51-873640
<br>22 Viewmount * web: <a href="http://www.whoisireland.com/">http://www.whoisireland.com/</a><br>Waterford * blog: <a href="http://blog.whoisireland.com">http://blog.whoisireland.com</a><br>Ireland * Irish Domain Stats & Market Research
<br>******************************************************<br><br>_______________________________________________<br>Search-l mailing list<br><a href="mailto:Search-l@wikia.com">Search-l@wikia.com</a><br><a href="http://lists.wikia.com/mailman/listinfo/search-l">
http://lists.wikia.com/mailman/listinfo/search-l</a><br>Change options or unsubscribe: <a href="http://lists.wikia.com/mailman/options/search-l">http://lists.wikia.com/mailman/options/search-l</a><br></blockquote></div><br>