[Grub-dev] where's Grub going?

[Jeremie Miller]

This is my personal roadmap for Grub, anyone else with ideas please  
chime in with your own :)

My larger vision is pretty simple, I want Grub to be a decent and open  
snapshot of the web that is kept fresh, and then stored/shared in ways  
useful to the whole community (individuals and companies alike,  
whoever is contributing).  One way of doing this is having the data in  
Hadoop and running community-directed MapReduce jobs on the whole  
dataset at the ISC, with the results being openly available to anyone  
to use.

I think we're getting close to at least the last part of that, Seth  
almost has the ARCs being uploaded into HDFS.  Things get a little  
more complicated after that as we both need to write MapReduce jobs  
that will slice or index the ARCs in some useful ways, as well as get  
something to better create the workunits from that dataset.

It's great to see the project making progress though, even if slowly.   
It's a really big task to take on, but if we keep it simple and keep  
trying, we'll get there :)

Jer