How to get an RSS feed from a website that doesn't have an RSS Feed.
-
Setting up an RSS feed for a job board that does not have an RSS feed subscription. Is there a way to do this? I would like to add an RSS feed to my iGoogle pages for USAJOBS.GOV searches that I have set up. USAJOBS.GOV allows you to subscribe to searches and receive them by email, but does not have a way to subscribe via RSS. I went to http://, which has a tool that they call HTML2RSS, which is supposed to convert any HTML page to an RSS feed, but it doesn't work very well on USAJOBS.GOV. ANy ideas
-
Answer:
http://www.crummy.com/software/ScrapeNFeed/? "Scrape 'N' Feed is a simple Python wrapper around the http://www.dalkescientific.com/Python/PyRSS2Gen.html. It implements almost all of the code you need to create RSS feeds out of web pages. All you have to rite is the code that actually does the screen-scraping (and http://www.crummy.com/software/BeautifulSoup/ makes that easy). It stores feed state in a pickle file between invocations, freeing you from having to worry about most of the minor problems that get in the way of scraping RSS feeds."
gm2007 at Ask.Metafilter.Com Visit the source
Other answers
I think yahoo pipes might work for this.
rdurbin
You may be able to use an email-to-RSS service http://www.mmmmail.com/ (I haven't used it personally).
null terminated
http://www.page2rss.com could possibly work for you
deezil
http://www.xfruits.com might works as well.
chasles
In this day in age, most database driven pages without RSS are intentional, so automated tools like HTML2RSS generally fail (as it should). However, with government procurement, you never know why things are the way they are. Either way, you'll have to go beyond the easy 1 click approaches. You'll likely need to know how to write a program; I imagine there's books on the subject of http://en.wikipedia.org/wiki/Web_scraping that will be of use to you, but I don't own any. First, take a look at the page source and try to figure out where your data is coming from. Embedded HTML table? Javascript request for XML or JSON? Once you've figured out where the data is, you need to write a filter to request only the data you need. Once you have the filter, you need to figure out how to divide data into individual records and subdivide that into fields. With HTML tables it's usually a table row and table data tags (<tr> <td>field1...</td> </tr>). If your source is XML then it's usually straightforward. The final step is to map the data you have into RSS schema. Decide which source data populates the field, the link, the date, content etc. Bonus points for a consistent GUID that can handle revising fields. You may end up merging multiple fields here, because RSS is fairly lightweight. The tools you'll need for these are wget, HTML Tidy, Beautiful Soup, and PyRSS2Gen.
pwnguin
Much like Pipes, http://www.feed43.com/ will help you create a fairly reliable RSS feed from a site. Be forewarned that to make it work well requires some programming knowledge (regexes, specifically) or a bit of persistence. Having said that, I've created a number of feeds from local bars' calendars to get local events and it's held up over years.
yerfatma
http://ask.metafilter.com/136661/How-to-get-an-RSS-feed-from-a-website-that-doesnt-have-an-RSS-Feed#1953190: ""Scrape 'N' Feed is a simple Python wrapper around the PyRSS2Gen module. It implements almost all of the code you need to create RSS feeds out of web pages." I don't know how I found BeautifulSoup but not Scrape'N'Feed. This sounds like it implements exactly the features that kept me from writing more scripts; flexible archive, caching, expiring items, date guessing and GUIDs. Hot Damn.
pwnguin
https://www.changedetection.com/ provides an RSS feed of all pages you're following, as well as the ability to limit notifications to only sizable changes (no difinition of "sizable" provided), only additions/deletions, and additions/deletions of specific text. You can also limit notification to daily/weekly/monthly.
timepiece
Related Q & A:
- How to save RSS-feed to a database?Best solution by feeds2mysql.com
- How do you find out who made a website?Best solution by ChaCha
- Can I Get An RSS Feed For Facebook?Best solution by Web Applications
- How to create an animated picture in a website?Best solution by Yahoo! Answers
- How do I pay my bills for a website?Best solution by Yahoo! Answers
Just Added Q & A:
- How many active mobile subscribers are there in China?Best solution by Quora
- How to find the right vacation?Best solution by bookit.com
- How To Make Your Own Primer?Best solution by thekrazycouponlady.com
- How do you get the domain & range?Best solution by ChaCha
- How do you open pop up blockers?Best solution by Yahoo! Answers
For every problem there is a solution! Proved by Solucija.
-
Got an issue and looking for advice?
-
Ask Solucija to search every corner of the Web for help.
-
Get workable solutions and helpful tips in a moment.
Just ask Solucija about an issue you face and immediately get a list of ready solutions, answers and tips from other Internet users. We always provide the most suitable and complete answer to your question at the top, along with a few good alternatives below.