How to separate mongodb data files from the journal?

How do I migrate data from a MongoDB  to MySQL database? Can it be done in a real-time scenario? What are the pros and cons for each migration? Which one do you advice? What is your experience? Any reference DB expert who can do it?

  • We are going to launch a video based e-learning program by buying an existing precoded social LMS (not http://en.wikipedia.org/wiki/Software_as_a_service, but a precoded script that's developed in Java with http://en.wikipedia.org/wiki/MongoDB), but our lil tech team has experience in LAMP technology with experience in WordPress themes and development plugins dev, CMS customizations. So we want to develop a similar social LMS for the long term for our business in LAMP tech as it will help us to add different functionalities because of our experience in PHP, but to launch faster (as it will take around six months for our social LMS development in LAMP technologies), we are starting with the precoded script (with Java and MongoDB as mentioned above) my question is: if we start with the Java version of the social LMS now and later we want to transfer the data to the LAMP version social LMS when it's ready, 1. Can the database be migrated from MongoDB to MySQL? 2. Can it be done in a realtime scenario without losing any data?   3. What are the pros and cons for migration? 4. Is there a reference database expert who can do it? 5. Which database would you prefer if you have data related to e-learning like videos, documentation, etc.

  • Answer:

    Coursera managed this in October with a then user base of about 1.6 million users and thousands to tens of thousands of concurrent users on our site. We did it on our production databases with no downtime, either scheduled or unscheduled. Can the database (data) be migrated from MongoDB to MySQL? Yes. Coursera recently migrated all of our MongoDB database to sharded MySQL servers running on Amazon RDS. Can it be done in Realtime Scenario without losing any data?  Yes, we did it real-time with no down-time or loss of data. We did parallel writes into both MongoDB and MySQL for a time and ran a migration script to migrate old data into our MySQL servers. After that is done, we then flipped the switch to read data from both MySQL and MongoDB for a while to make sure that are consistent, then finally made the switch to read from MySQL as our "source of truth" or primary database. We did it all within 3 days on all our production databases. Three developers (including me) worked on the project. What are the pros and cons for migration? MySQL suited our needs better as it provided stronger consistency guarantees compared to MongoDB. We also preferred that Amazon manage our database servers for the moment as we did not require a DBA just yet. We also found that MongoDB did not play very well with Amazon EBS, frequently locking up or otherwise misbehaving and bringing our site down. We did lose the flexibility in schema, but it is actually okay because we were primarily using MongoDB as a key-value store. Any reference DB expert who can do it? It depends on the complexity of your database, but Coursera managed the migration within 3 days using only in-house developers. Which DB would you prefer if you have data related to e-learning like videos, docs etc. I personally found MySQL's performance to be much better than expected, so I would recommend MySQL (or PostgreSQL).

Frank Chen at Quora Visit the source

Was this solution helpful to you?

Other answers

Let me address your bullet points first: 1. Can the database be migrated from MongoDB to MySQL? Not the database per se, but the data, yes. MongoDB is a schema-less (NoSQL) database, whereas MySQL is a relational database (RDBMS). So, it'll largely depend on your data model and how the data is stored in MongoDB. NoSQL databases generally eschew normalization (e.g., references from one 'collection' or entity to another collection) and make you store related information all together. For instance, an "articles" collection might have a sub-collection "comments". On top of that, NoSQL tends to overlook data duplication in favor of performance. Again, in the above example, you might have another collection "users" with its own sub-collection "comments" which contains the same data from all comments made by that user across all "articles". So, you will have to come up with carefully detailed migration strategy specific to your data model and database to take existing collections of objects, translate those into tables and rows, possibly with lots of NULL-able columns in the meantime (i.e., relaxed constraints). Then, take any sub-collections and figure out how to detect duplicates if necessary, then create the corresponding rows in other tables and the necessary foreign key references to and/or from. 2. Can it be done in Realtime Scenario without losing any data? Technically, it should be possible to have a background process migrate your data one object at a time, then simply mark or replace the object in the NoSQL store with the primary key in the corresponding table. This means, however, that your application will have to support two data stores, with vastly different APIs and paradigms—one NoSQL, one relational. If you have an absolute requirement of six-nines uptime, then I can see you having to go that route. On the other hand, it will be a lot simpler (from a development standpoint) and cost-efficient (from a service delivery standpoint) to simply develop the relational version of your application, then migrate your data as it comes along (essentially, replicate, including any updates) from MongoDB to MySQL, then, as soon as you have nearly 100% of your data replicated over decide on a cut-off time when you take your existing system offline (or prevent any updates) while you switch over to your new system running on LAMP. 3. What are the pros and cons for migration? Honestly, I don't see why you need a LAMP version running on MySQL. For one, PHP can access NoSQL/MongoDB just fine. Two, planning and executing the migration properly will take your little tech team significant time and effort and given your 6-month launch window I'm not sure how much time that'll leave you to add features to your app. 4. Any reference DB expert who can do it? Can't speak for where you are exactly, but I'm sure there are people with experience in NoSQL to RDBMS migration or integration and vice-versa. 5. Which DB would you prefer if you have data related to e-learning like videos, docs etc. Frankly, if it were up to me and building something from scratch I would definitely lean toward a traditional RDBMS first, but augment it if necessary with a NoSQL alternative for schema-less data or for performance reasons. For example, we use MongoDB with PostgreSQL in one Web site where almost all data is stored permanently in the RDBMS, but the user 'feed' (which is a heterogenous collection of events that can come from multiple sources, some 3rd-party) is in MongoDB. Now, having said all the above—if it were entirely up to me another way to do it is to partition the functionality of your site. That is, build on top of the existing Java + MongoDB based LMS and add additional features using PHP connected to MongoDB. At this point, I don't see the need for MySQL (even if your data model might be a good fit for a traditional RDBMS). Until you identify a concrete and compelling business value to bring in a totally new, foreign architectural piece such as MySQL and expend effort to migrate all or some data to it, then stick with what you have. tl;dr: Build on your strengths, rather than fix your weaknesses. Stick with Java + MongoDB and bring in PHP to the mix, forget about MySQL for now.

Alistair Israel

http://www.scalebase.com/ can do the job for you!

Yehuda Hofri

You can use this pattern to migrate between two different databases, for instance between MySql and MongoDB or between two schemas in the same database.The idea of this pattern is to do a lazy database migration using http://www.aviransplace.com/2013/03/27/continuous-delivery-part-3-feature-toggles/ to control the behaviour of your application and progressing through the phases of the migration.You can read about http://www.aviransplace.com/2015/12/15/safe-database-migration-pattern-without-downtime/#ixzz3ve56nqDX%20(http://www.aviransplace.com/2015/12/15/safe-database-migration-pattern-without-downtime/#ixzz3ve56nqDX) in this post: http://www.aviransplace.com/2015/12/15/safe-database-migration-pattern-without-downtime/#ixzz3ve56nqDX

Aviran Mordo

Related Q & A:

Just Added Q & A:

Find solution

For every problem there is a solution! Proved by Solucija.

  • Got an issue and looking for advice?

  • Ask Solucija to search every corner of the Web for help.

  • Get workable solutions and helpful tips in a moment.

Just ask Solucija about an issue you face and immediately get a list of ready solutions, answers and tips from other Internet users. We always provide the most suitable and complete answer to your question at the top, along with a few good alternatives below.