What makes one search engine better than another?

How can one build a search engine for some specific search?

• I want to buld a serch engine like http://www.indiabookstore.net/ for specific search on some collection of website. Fro where should i start, what tools or courses should i take like IR?

• Answer:

  Here's what you need to doStep 1 : Find the domain and set of website (or type of websites) you need to crawl and index. Step 2 : Write a crawler to get the data from these websites. Recursively check all the pages within each category and subcategory like traversing a tree. Step 3 : Parse their content from their webpages and index the data based on Book Titles, Description and Reviews. (totally depends on you) Step 4 : Create a retrieval mechanism. Define proper ranking measures so that you'll show the results that a customer expects. Step 5 : Create a front-end (of course, you know that pretty well) Optional : To stay up-to-date, repeat from Step 2 periodically. Additional : Make sure your systems are capable of handling big amount of data and retrieval requests. Here's how you're going to do theseStep 1 is one of the most crucial parts. Well emulating IndiaBookStore would be simple as they index a fixed set of websites. Remember the amount and quality of stuff you retrieve depends on the 'seed'. Step 2 http://stackoverflow.com/questions/102631/how-to-write-a-crawler http://www.makeuseof.com/tag/build-basic-web-crawler-pull-information-website/ If you don't know much about IR you can have an overview from a Udacity course that gives very nice intro. This course introduces you to crawler as a part of programming projects. https://www.udacity.com/course/cs101 Step 3 : Begin with a SAX parser or a third party library to retrieve the body of the pages. You will need lots and lots of regex. Next part is the most important component, definitely needed in a Search facility : INDEXING. You need to build an inverted index which maps a word to name/ID of a book or webpage. These links will introduce you to indexing and parsing : http://stackoverflow.com/questions/1786689/parsing-web-pages Step 4 : All you have to do is retrieve relevant data from the the index you already created in Part 3. To make retrieval faster, make sure to create a hierarchy of secondary indexes (like hashing). Step 5 : You know it pretty well ;) If you are not a champ with that, you'd like to hire a professional web designer who can improve the aesthetics and usability while you work on the tech part. Prof Chris Manning's book : http://nlp.stanford.edu/IR-book/ will teach you all you need to work with this project. And yes, use JAVA for this project. Java is cool! = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = p.s. If you have some experience with software projects and have time to learn something new, try out a tool like Lucene, Solr and Nutch. (All are Apache Projects)