How to Write Form Data to XML?

What is the fastest way to write large amount XML data (1GB+) from Java?

• If I have a use-case where close to 10GB data is be transferred between various services using XML what is the fastest way in Java to write this?

• Answer:

  You apparently have 10 GB of data that you need to move from point A to point B.   This means that you have at least three problems: 1. How to get the data from A to B.  A naive client-server implementation (e.g. HTTP POST to a servlet) will, most likely, cause your client or server to pagefault, thrash, and die.  Consider using one of the well-understood and supported point-to-point file transfer mechanisms, such as ftp or rsync. If you truly have a requirement to distribute all 10G of data at once, you will need to spool that to a temporary file using ordinary FileWriter techniques; it is unlikely that anything you do in the Java layer will affect performance since you're going to spend all your time waiting for your disk to handle the write. (And, yes, as Animesh Kumar notes, if you really need to distribute this to more than one receiver, there are protocols that are designed for that: look up UFTP, UDT, and related work) 2. Once the data is at B, you need to decode it in a way that will allow you to do useful work.  If you are ''required'' to use XML for this, you must read up on how to use a streaming XML parser, such as the SAX or expat systems.  You will need to stream the file from disk into your program, decoding it as you go, constructing the data structures you need progressively instead of recursively.  You will need to make certain that you handle, and then dispose of, any large data regions during your decode.  In Java, you will need to be especially careful of dangling references that would confuse the garbage collector. 3. As you finish processing each data region (subelement, big text, whatever it is you're doing), you'll need to commit it to whatever persistent store you've got -- presumably you're talking to a database, or a file system, or something.  By the time you inflate 10G of UTF-8 encoded XML into 2-byte Unicode-encoded Java strings, you're paying 20G just to get the thing into RAM, and you're probably making copies of the Text nodes anyway.  You will not want to just slurp it into memory. At this point, you should be asking yourself -- why are you using XML for this, anyway?  If what you're really doing is distributing a bunch of text or binary objects with XML wrapped around them for metadata, why not just create a ZIP archive of the objects with tiny XML files containing the metadata and be done with it?