What is a database engine?

What is the concept behind making a search engine robot collect website meta-data, store in a database, and follow a link to visit a new website?

• to be more precise, i would like to draw interest of Programmers, designers and software engineers to help me make me understand the basic algorithm of designing the search engine bot.

• Answer:

  A Search Engine (for example - Google) deploys number of techniques for optimizing the search results. A Crawler on the first place crawls almost every website in the World Wide Web. To be short, they copy the content of the pages they crawl. For Google it takes about three months (approx) to crawl the entire World Wide Web. Once the Crawler copies the content of the web pages, the Search Engine's Indexing algorithm comes into action. Indexing is the process of parsing and storing the relevant information. Web indexing also involves Language Processing, Linguistics, Psychology.  Keywords are mapped with the appropriate information (web pages). These goes into the database then. When a keyword or search term is entered, Google searches its index and matches the keyword. It then displays the result (a set of web pages) that are associated with the keyword. This is how most of the search engines work. Hope this enlightens!