[Search-l] Grub + Wiki = Open Crawler

[peter burden]

jer wrote:
> I love all your suggestions peter, thanks. What I was originally 
> outlining is the *human* contributed attributes (what value can a 
> human provide or override to the crawling process). What you started 
> outlining is a meta-index of the output of a crawler, and I absolutely 
> agree that all those attributes and more should be available.
>
> I want to use a wiki with grub.org as an input for the human side, and 
> the output of the crawler will be an open DB with the kinds of 
> discovered attributes you described.
Excellent. This will be the mechanism by which "the community" can 
identify useful site and page
attributes that can then be incorporated into the database. My only 
concerns are that the community
discussion that this envisages should address the issue of how the 
attributes might be quantified
and also, and perhaps more important, that the discussion should 
actually get started.

My list of page attributes was intended, in part, as a seed for such a 
discussion. I focussed on
automatically derivable attribute values because I'm more familiar with 
such things, however
there's a very real role for human contributed attributes. There are, of 
course, issues of
standardisation and scalability for human contributed attributes but 
this doesn't mean they
shouldn't be used. If we go for user parametrisable ranking, as I hope 
we will, if you're
unhappy with the user contributed stuff you don't have to use it.