MATLAB - Delete elements of binary files without loading entire file
-
This may be a stupid question, but Google and MATLAB documentation have failed me. I have a rather large binary file (>10 GB) that I need to open and delete the last forty million bytes or so. Is there a way to do this without reading the entire file to memory in chunks and printing it out to a new file? It took 6 hours to generate the file, so I'm cringing at the thought of re-reading the whole thing. EDIT: The file is 14,440,000,000 bytes in size. I need to chop it to 14,400,000,000.
-
Answer:
I found Perl is much quicker to do this than MATLAB. Here are two examples from http://oreilly.com/catalog/cookbook/chapter/ch08.html: truncate(HANDLE, $length) or die "Couldn't truncate: $!\n"; truncate("/tmp/$$.pid", $length) or die "Couldn't truncate: $!\n"; You can run Perl script from MATLAB with http://www.mathworks.com/access/helpdesk/help/techdoc/ref/perl.html function.
Doresoom at Stack Overflow Visit the source
Other answers
There is no ftruncate() in Matlab, but you've got access to the full Java standard library in the JVM embedded in Matlab, and can use java.io.RandomAccessFile or the Java NIO classes to truncate a file. Here's a Matlab function that calls to Java to lop the last n bytes off a file. Should have minimal I/O cost. function remove_last_n_bytes_from_file(file, n) jFile = java.io.RandomAccessFile(file, 'rw'); currentLength = jFile.length(); wantLength = currentLength - n; fprintf('Truncating file %s: Resizing to %d to remove %d bytes\n', file, wantLength, n); jFile.setLength(wantLength); jFile.close(); You could also do it as a one-liner. java.io.RandomAccessFile('/path/to/my/file.bin', 'rw').setLength(n);
Andrew Janke
Since you don't want to read the file into MATLAB (understandably), you are dealing with system level commands. MATLAB has a facility to call system commands using the "system" command http://www.mathworks.com/access/helpdesk/help/techdoc/ref/system.html So now your problem is reduced to finding the shell command in your OS that will do it for you. Or you can write a program using truncate() (unix -- KennyTM) or SetEndOfFile (windows)
Marc
I don't know if MATLAB supports this, but see http://www.opengroup.org/onlinepubs/007908799/xsh/ftruncate.html.
KennyTM
Related Q & A:
- How to Read Binary File to XML?Best solution by SharePoint
- How to write binary data to a file?Best solution by Stack Overflow
- How can I open binary files, that are email attachments?Best solution by Yahoo! Answers
- How can I transfer all of my Yahoo email files to a document file in my documents?Best solution by Yahoo! Answers
- How to combine files into a RAR file?Best solution by smallbusiness.chron.com
Just Added Q & A:
- How many active mobile subscribers are there in China?Best solution by Quora
- How to find the right vacation?Best solution by bookit.com
- How To Make Your Own Primer?Best solution by thekrazycouponlady.com
- How do you get the domain & range?Best solution by ChaCha
- How do you open pop up blockers?Best solution by Yahoo! Answers
For every problem there is a solution! Proved by Solucija.
-
Got an issue and looking for advice?
-
Ask Solucija to search every corner of the Web for help.
-
Get workable solutions and helpful tips in a moment.
Just ask Solucija about an issue you face and immediately get a list of ready solutions, answers and tips from other Internet users. We always provide the most suitable and complete answer to your question at the top, along with a few good alternatives below.