(Up to: Code Ideas )

Excerpt from this post at ThoughtStorms:

There's still a lot of interesting stuff that lies ahead for search engines, especially once we start to get distributed "spidering" that allows individual users to index whatever content they find, a process touched upon by StumbleUpon but that could be extended a whole lot further - by indexing the information you yourself can see, it starts to become possible to at least keep pointers of the "dark web" - all the content hidden behind databases, CGI and log-ins. It also makes cataloguing and linking of any documents possible, rather than just whatever Google writes an interface for. Secondly, everything'll change if we can get widespread intercompatible semantics for all information.

I like the idea of "distributed, contributory indexing", but not sure if it'd scale too well, as most people would rather their search experience be passive, and when-they-need-it.

It would have to work as an abstract informational concept, rather than simply based on a system judged by links and their weights, if it is to take advantage of the disparate nature of the information being indexed. In fact, the relation between the data would probably need to just as much a priority as the indexing of it in the first place. Besides simple mark-up style links, a semantic approach could be considered, the accuracy of which would improve as more people subjected their links into the system. Failing the presence of a universal semantic network, this could potentially be achieved through some form of human marking that could apply a form of semantics on an abitrary level of scalability (i.e. from site level to document level to sentence/word level).

Disadvantages currently include the fact that this seems to be just as "influencable" as any other algorithm, if not more so due to its open nature, and it's ranking ability is indeed only as good as the people that do the indexing. I also think there's a little bit too much effort involved, so perhaps it could be used on top of other, traditional search techniques, to provide a semantic organisation and extension of existing, automated results. It could also make it easier to collate groups of "functionality", e.g. information vs commercial, etc.

I shall continue to think about this, though as I'm not really up-to-date on search engine optimisation techniques, I'm wandering in the dark quite a bit really...