How to add bulk virtual data in database?

What's the best way to go about inserting data and images in bulk into MySQL database tables with PHP or MySQL?

  • I'm looking to add new rows to certain tables in a database.  Can it be done via a CSV file?  XML file?  Are there other, or better alternatives?  Is there a way to just add rows to tables or would it need to be done as a total re-write, i.e. to download existing rows and overwrite the entire table with new data included?

  • Answer:

    The best way I've found to do this, input the data into a CSV with a reference to a filename, then in PHP use http://php.net/manual/en/function.fgetcsv.php to loop through each CSV row and update/insert the data (including reading the image file and inserting it)

Bryon Keck at Quora Visit the source

Was this solution helpful to you?

Other answers

While CSV is faster than XML, it is more error prone as there is no real way to validate it without writing code to count number of delimiters on each row. With XML, the script simply won't open an invalid file. I wrote this guide for my last employer on how to integrate an XML feed into a MySQL database using PHP. Hope it helps you in some way. http://files.perfiliate.com/solutions/BuyAtParseProductFeedsGuide.pdf

Simon Quick

Depend on the amount of data you will insert. As Saurabh said, LOAD DATA INFILE is the fastest way. Turning off indexes and rebuilding them at the end could be faster than updating them for each record you insert. If is really heavy the amount of data, assuming you want to optimize inserts over queries, would be having the tables using the tokudb storage engine.

Gustavo Muslera

The safest way would be to build a parser (in PHP) to validate the data, then insert as per Bryon's answers. The quickest way is via MySQL's shell using the "load data" command to append to your table. http://dev.mysql.com/doc/refman/5.1/en/load-data.html But: I recommend not inserting images into your db. It's faster to serve static files from the file-system and easier to backup small text-only databases. If you choose to insert the images into the database, watch out for problems with large images due to MySQL/PHP message-buffer-size settings.

Alex Barkan

Look into using a migration tool like <a href="https://github.com/ruckus/ruckusing-migrations">Ruckusing</a> or <a href="http://www.phinx.org">Phinx</a>. If you are using a framework, try leveraging their Migration feature if available. It makes it rediculously easy to do these sort of things.

Ryan Flores

I highly advise against storing images in a database.  Store the file name reference or you will have huge bloat and decreased performance.  Use the file system, it's much better at handling that sort of thing.  For data, use the http://dev.mysql.com/doc/refman/5.5/en/optimizing-innodb-bulk-data-loading.html rather than PHP.  Check out http://www.mysql.com/products/workbench/.

Keith Ensign

A even better way than csv is to create bulk insert queries and multiple update queries and dump them in a sql file. Then source the file in mysql. This method should be even faster than creating csv.

Jayant Kumar

Best way is LOAD DATA INFILE I was used it for daily upload  about 2 TB of Data into MySql this is best & optimize way to upload heavy data . I was create a php program which extract rar files into directory then Upload these files which is in form of comma separated (log files) into MySql DB . I was testing using all techniques like bulk insert into db, upload data into csv format table then modify table format from csv to myisam .

Saurabh Goyal

Just Added Q & A:

Find solution

For every problem there is a solution! Proved by Solucija.

  • Got an issue and looking for advice?

  • Ask Solucija to search every corner of the Web for help.

  • Get workable solutions and helpful tips in a moment.

Just ask Solucija about an issue you face and immediately get a list of ready solutions, answers and tips from other Internet users. We always provide the most suitable and complete answer to your question at the top, along with a few good alternatives below.