How do search engines work?

How can search engines work so fast?

• Im not sure who/where to ask this of, plz advize if is out of your expertise:: When one searches the I/Net for a target (eg a set of words) using  eg., Yahoo, Altavista, google, etc::  the replies seem always to come back -too fast- so fast as if it cannot be!! how can that be? My question is:   how MANY processors (CPU's) are/might be used/invoked at one time for any one search?  Certainly, MPP goes on here, but is the actual # of CPU's possibly in the TENS of thousands?  ALL searching at same time for my request -target- of maybe -only- 10-20 words!?  I have read where 100s or 1000s are used/talked about,  but is there perhaps an "average" # CPUs used, for an "average" task search,  that is actually >> just a thousand or so CPU's?  How can so much searching occur, looking at/thru millions of web pages, in so little time? Unless some huge number of these CPU's are in play at same time? Is this written up anywhere?   thanx, I could not find this as a FAQ type of thing!! bob mackey,  a long-time user of AltaVista adv search engine        [email protected]  

• Answer:

  Hi Bob. Thousands is more likely. You see, it is impossible to store all the data in one DB, so what such large search engines go for is partitioning the search space. So, your query goes in paralel to several hundred nodes simultaneously. I worked on a search engine project (smaller, though) myself, we had N paralel search engines, doing load balancing. The next step is partitioning. So, the numbers you give are not at all surprizing. Nix.