How do we remove duplicated rows (tr) in a large HTML file? (212 MB)
-
I know how to remove duplicated rows with jQuery as per http://stackoverflow.com/a/8234194 But the problem is when you have a 200MB HTML file, the browser load your file lazily. Like you'll start with 10k rows, but it's even bigger than 70k rows and counting. Also, the file crashed Sublime Text too
-
Answer:
I would either use nokogiri(ruby) or BeautifulSoup (python). A sample code in ruby using nokogiri -: require 'nokogiri' doc = Nokogiri::HTML(open('/path/to/the/file')) check = {} doc.css('table tr').each do |t| temp_text = t.text if check[temp_text] doc.css('table tr').delete(t) else check[temp_text] = true end end By the way, use stackoverflow for such questions to get answers quickly! Cheers!
Prateek Papriwal at Quora Visit the source
Other answers
Your browser will take fire with a 200Mb HTML file, there is no way it will work correctly. If you're on Linux though, you could try something to this effect : cat large_file.html | uniq I cannot guarantee that it will work with such a big file though. Give it a try and let it work for the night! Source : http://ubuntuforums.org/showthread.php?t=1454601
Damien Buty
I would use a script - either a shell script, or perl or ruby if that's your flavor. Iterate through the document, and whenever it finds a duplicate row, have it remove the line in question. You could even add some further logic to make it only act on lines that have tr elements. I sadly don't know enough to make this off the top of my head, but if you have a person on hand who's got some scripting expertise they should be able to whip one up that could take care of this.
John O'Hara
Related Q & A:
- How to convert a HTML file to XML file?Best solution by Stack Overflow
- How do I remove the old stereo in a 2001 Chevy cavalier to install a new stereo?Best solution by Yahoo! Answers
- What are .cue files and how to separate a large flac file?Best solution by techsupportalert.com
- How can I remove white noise/static from a video on my digital camera?Best solution by Yahoo! Answers
- How do you remove the door panel on a 2001 grand am?Best solution by answers.com
Just Added Q & A:
- How many active mobile subscribers are there in China?Best solution by Quora
- How to find the right vacation?Best solution by bookit.com
- How To Make Your Own Primer?Best solution by thekrazycouponlady.com
- How do you get the domain & range?Best solution by ChaCha
- How do you open pop up blockers?Best solution by Yahoo! Answers
For every problem there is a solution! Proved by Solucija.
-
Got an issue and looking for advice?
-
Ask Solucija to search every corner of the Web for help.
-
Get workable solutions and helpful tips in a moment.
Just ask Solucija about an issue you face and immediately get a list of ready solutions, answers and tips from other Internet users. We always provide the most suitable and complete answer to your question at the top, along with a few good alternatives below.