[atlas-l] Atlas - Internet Search Infrastructure

[jer]

This is a brief overview of a large vision: enabling search to become  
a part of the Internet's infrastructure.  Building on Atlas as an  
open protocol, search can become a fully distributed and  
interoperable world-wide community.  All of the participants can  
interact openly and in any role where they believe they can add value  
to the network.

A search engine can be constructed from many independent entities  
serving different roles instead of one monolithic system. These  
entities are exchanging aggregate information, or knowledge, and can  
decide with whom they want to work with. To design this working  
economy based on knowledge, there must be balance between these  
various entities.  Each actor must have incentive to act both for  
their own benefit and for the benefit of the whole, and enough  
information to make and validate those decisions.  Reputations and  
relationships are the essential fabric of Atlas, just as they are in  
a real-world free market.

There are three primary roles within Atlas:

     Factory - Responsible to the content.
     Collector - Responsible to the keyword.
     Broker - Responsible to the searcher.

Each of these actors must interact with the others to complete any  
search request. Any two roles could be performed by a single entity  
(whereas if all three are performed by one entity, the result would  
be a traditional, monolithic search engine).

A Factory is akin to a crawler in today's search engines.  An Atlas  
Factory must fetch and process the content as intelligently as  
possible, performing analysis (such as Natural Language Processing)  
and normalizing it into distinct units.  A Factory shares its highly  
refined and processed output with one or more Collectors based on who  
they believe is best utilizing it.

A Collector absorbs and indexes output from one or more Factories,  
with one primary goal: ranking.  An Atlas Collector must provide the  
most intelligent ranking and relationship analysis possible.  A  
Collector has to compete for the output of a Factory, as well as  
compete to provide the best ranking quality for Brokers.

A Broker must provide a searcher with the best possible results. It  
does so by combining diverse ranking results from Collectors and also  
by retrieving content from the original Factories.  This last step, a  
Broker interacting with a Factory, is critical to maintaining a  
balanced ecosystem.  All Factories must be aware of and approve how  
their results are being used and by whom.

Reputation and reward is bi-directional between all parties (Factory- 
Collector, Collector-Broker, and Broker-Factory).  Each entity may  
choose to interact on principle (free, Commons), attribution (results  
provided by), or commercially (as a paid service), the Atlas protocol  
is purely a facilitator and does not restrict how the relationships  
between any entities are formed.  In considering these motives for  
the various entities, it's likely that the free-based networks will  
tend to become more specialized, commercial ones will compete on  
quality, and attribution based networks will mature in both directions.

This simple yet powerful division of roles, responsibilities, and  
relationships will result in a distributed economic foundation for an  
Internet Search Infrastructure.  The wire protocol and further  
definition of the interactions between these entities is openly  
evolving, anyone interested is welcomed to join the discussions and  
see the initial proposals at http://lists.wikia.com/mailman/listinfo/ 
atlas-l over the coming weeks.

Thanks, looking forward to a radically different search ecosystem in  
the coming years :)

Jer