How do I go about creating a custom search engine for my website?

Website Structure and Affect on Search Engine Placement

• My website (www.dogbedsetc.com) was programmed by someone who said they knew e-commerce. After research on this website and other sources I have found that no backend stuff was done for the Search Engines to find my site. I have pushed to finally get meta descriptions, keywords, etc. on the home page. Some items that may be causing problems: ** dynamic site with URLs that have no relevant words in title. (I have hired someone to change the 2nd tier URLS - Sections - to reflect relevant names and to add alt tags to pictures). ** I have hired someone to add meta keywords, descriptions, to the section pages. Will this help? ** Can you identify other problems with pages (text, descriptions, alt tags, etc.) ** I read in Elizabeth Castro's book "HTML for the WWW" (page 422) that a Crawler Page should be created to link each page to each other and have the Crawler Page submitted to the search engines. Please comment and tell me if that would help. ** Until last month Google had finally found my home page. I had a ranking of "4". Google also reflected links to my page. Suddenly my site was lowered to a "3" and no links can be found. Is there something on my site creating this problem? ** Please advise as to other structural problems with my website which is affecting Search Engine placement. Please advise as to any ideas on the structure of the website and its pages for improvement and other critical ideas for Search Engine placement. I am frustrated after paying so much for an "expert" to create the site and to find that the basics were not done.

• Answer:

  Hi there, Here is some practical advice to help your site get more visitors from Google and other search engines, plus some responses to your specific questions. Dynamic Content --------------- This is the biggie. Although Google is able to index dynamic websites, it doesn't usually. In your case, the only pages it has indexed are ones which do not have a ? in the URL: www.dogbedsetc.com/ www.dogbedsetc.com/orderinfo.aspx www.dogbedsetc.com/contactus.aspx www.dogbedsetc.com/aboutus.aspx www.dogbedsetc.com/viewcart.aspx ://www.google.com/search?q=inurl:dogbedsetc.com/+site:www.dogbedsetc.com+ "We are able to index dynamically generated pages. However, because our web crawler can easily overwhelm and crash sites serving dynamic content, we limit the amount of dynamic pages we index." ://www.google.com/webmasters/2.html#A1 Some webmasters believe that dynamic URLs are fine as long as they don't include session IDs or anything that looks like a session ID. GoogleGuy (a Google engineer) says: "The little mantra I repeat to myself is "dynamic urls ok, avoid sessionids if at all possible." Spiders don't like sessionsids. :) " http://www.webmasterworld.com/forum3/9256-2-15.htm The trick could be to remove any parameters from your URLs that are, or look like, session IDs. However, seeing as Google can and does index sites with URLs like yours, it appears that Google has simply decided that your site is one of the many with dynamic pages that it chooses not to index. To be certain of being indexed, you need to convince Google that your pages are static. I am not a programmer, but it is clear to me that you have two options: 1) Redesign the entire site so that every page is static 2) Use mod_rewrite or a product like the XQASP filter which will rewrite the URL so that it appears to be static A Users Guide to URL Rewriting with the Apache Webserver http://www.engelschall.com/pw/apache/rewriteguide/ XQASP filter http://xde.net/xq/tool.xqasp/qx/index.htm Page Titles ----------- Page titles are extremely important. Firstly, when someone bookmarks a page, it should say what the topic of the page is. Otherwise there is less chance of them returning. For search engine optimization, keep these points in mind: 1) To get the most chances of appearing in search results, make every page title in your site is different. 2) Try and make the title read the same as the keywords someone might search for, in the same order. 3) For Google, shorter is better. Two or three words is best. 4) Experiment! Every month change the title of any pages not receiving search engine traffic, and see if the new title makes a difference. There is one problem which can only be decided by individual webmasters - your company name. Including it in the title is good for the visitor and bookmarks. It just doesn't help get search engine traffic. I recommend having your company name in the title of your home page, about us, contact us, and shopping cart pages. For individual product pages, and any articles or FAQ pages, no company name. Spider Simulator ---------------- You need to enter an actual file name, such as: http://www.dogbedsetc.com/default.aspx Although http://www.dogbedsetc.com resolves to the above file, it is not a file in itself. The funny character appearing in the title is because the hyphen used in the title is non-standard. It looks like this – instead of this -. If I look too close I get cross-eyed, but I think it is slightly longer than normal. It would surprise me if, when I post this answer, it comes out with the same funny characters. I suggest changing it. Random Text ----------- If by random text you mean: <input type="hidden" name="__VIEWSTATE" value="dDwtMTQ2NzM3OTIyMjt0PDtsPGk8MT47PjtsPHQ8O2w8aTwzPjtpPDU+O2k8OT47PjtsPHQ8O2w8aTwxPjtpPDM+Oz47bDx0PHA8cDxsPFRleHQ7PjtsPDA7Pj47Pjs7Pjt0PH.... etc It's just a way for ASP to keep track of what the customer was doing: "ViewState is a feature provided by ASP.Net that stores information for server controls used on a web page. The primary advantage of this feature from a developer perspective is that it keeps the values of controls on a form. If the user forgot a single required field on a form, ASP.Net can put back the other values they had entered rather than forcing the developer to repopulate them. The information is encoded into a string and placed in a hidden form field called “__VIEWSTATE” that is sent to the client with the response (you can see this by using View Source in your browser when you view a .aspx page). ASP.Net by default also calculates a “hash” of the information and adds it to the ViewState, then on a postback it can validate that the data has not been corrupted." http://www.aspalliance.com/robertb/articles.aspx?articleId=2 META tags --------- Keyword tags are a waste of time. They have been abused to the point that no reputable search engine takes any notice of them. They are harmless though, so don't stress about removing any that exist already. Description tags are important. They should be written in everyday language, one or two sentences. Not only do some search engines use them, editors at directories like Open Directory use them - it saves them from thinking one up. The description for your home page is fine. For products, something simple like this will suffice: <meta name="description" content="Wheel Away Pet Carrier, an easy-to-pull soft dog carrier that converts to backpack, car seat and bed - from Dog Beds Etc."> ALT attribute ------------- More popularly known as the ALT tag, it is a description for an image. All images on your site should have a description - it makes your site more usable for vision-impaired users, and it greatly increases the odds of someone finding your site via an image search. Instead of this for your logo: <IMG SRC="images/siteImages/main.jpg" border="0"> Have: <IMG SRC="images/siteImages/main.jpg" border="0" alt="Dog Beds Etc"> Click Picture to order ---------------------- Most of your first-tier pages don't say much about the product, and advise: Click Picture to order This could be costing you a lot of sales - customers want to know as much as possible about a product before ordering, and they will not guess that clicking on the picture provides more info. Make sure you change the wording to: Click Picture for details [or] Click Picture for more info Crawler page ------------ It makes sense to make it as easy as possible for a search engine to find every page of your site. This is the only page you need to submit to search engines. If possible make it serve a dual purpose - it can also be a site map for your visitors to use. No matter how good your navigation system is, there are always a few folk (like me) who go straight to the site map when visiting a site. Make sure it is static. IMPORTANT: Google only follows the first 100 links on a page. If your site has more than 100 pages, you need to make several site maps / crawler pages. Backward Links -------------- Google only shows backward links for sites with a PageRank of 4 or more. Don't worry, Google still knows about the links, it just doesn't list them. To see all your backward links, enter your site's URL in the search box at http://alltheweb.com There could be many reasons why your site dropped from PR4 to PR3. One of the more likely one is Google deciding that the Yahoo category you are listed in is less important than before. It is normal for sites to have their PR go up or down a notch every now and then. Keep in mind that the PR only represents the true figure. Dropping from PR4 to PR3 might in reality might just be a drop from 564,789 to 564,682 - with sites above 564,700 being described as PR4, and below that as PR3. No-one outside of Google knows what the numbers really are. Directories ----------- Being listed in Yahoo is good for increasing PageRank: http://dir.yahoo.com/Business_and_Economy/Shopping_and_Services/Animals/Supplies__Equipment__and_Gifts/Beds/ What would help even more would be a listing in the Open Directory. This is the best category to submit to: http://dmoz.org/Shopping/Pets/Supplies/ If you need any part of my answer clarified or expanded upon, just ask for a clarification and I'll get back to you. Best wishes, robertskelton-ga