How do some website content get indexed by search engines even though they require users to login to view those particular pages?

This is usually done via site maps, specialized XML files, that are prepared just for the search engines and submitted automatically on a regular basis, which allows for Google to "find" them without having to crawl past a logic screen.

Sometimes it's because of cloaking: websites recognize major search engine crawlers and allow them to see content that normal web users have to log in to see. Search engines penalize or deindex sites for cloaking, but it still happens. Quora, Facebook, and Twitter don't do this.

Most of this websites create virtual internal user in, as Integrated user in the CMS system. And connect the search engines to this users. Its done with some technical solutions , like attaching the user by useragent, or known ip address ranges. Good example will be a known phpbb forums ,if  you visit one of the- used the first one from the serp.. You will see on the sidebar a login box http://forums.phplist.com but if you check the google cache you will find a logged googlebot user  on the sidebar http://webcache.googleusercontent.com/search?q=cache%3Aro5S5VCfDPkJ%3Aforums.phplist.com%2F+&cd=1&hl=iw&ct=clnk&gl=il&client=firefox-a and on the fotter you will se same user for bing bot

For any website, only those pages get indexed in the search engines which are accessible without login. It may be the case that the content of a page is different when viewed without login, due to which the entire content may not get indexed, but the URL of such a page and the content visible without login would surely get indexed. Note that only those pages are not indexed by search engine which have a redirection applied with login check. That is, if there is a page which when accessed without login redirects to say login page then such a page will not get indexed or crawled by search engine. Taking example from Quora - try accessing this page in a browser without login - You will find that you are not being redirected to login page and can view some content (one answer) without login. Now if you go to google and search for - quora How did you end up falling in love with mathematics - you will find that first result is the link to same question and when you click at that link then you will find that again only one answer is displayed. Thus, unless there is no redirection applied at the page with login check, it can be crawled by the search engine. And so for websites, like quora, facebook all pages which are accessible without login get indexed with content filtered. UPDATE: Kindly don't take it from the answer that the websites which are haivng a login restriction can't be crawled at all. Say if you want to have your own crawler service and make it crawl a website having login restrictions and re-directions, then this is very much possible. One way of doing this is by using libraries like CURL in PHP. The above answer was shared in general perspective keeping search engines in mind.

How do some website content get indexed by search engines even though they require users to login to view those particular pages? If you are using some login to show the webpage then this can be used only for user, if you want search engine will not crawl your web page then this is possible by blocking that particular file or web page. If you did not block then crawler will automatically index your web page with your content. Hope you understood. this will be helpful fro you. For more information you can contact at http://www.synapseindia.com