How to Scale a Mobile App?

What is the ideal database server architecture for a Facebook App that is anticipated to scale up very quickly?

• We intend to use MySQL, and users will be uploading media (mostly photos) through the app. We anticipate a few hundred thousand users in a relatively short period of time, and want to architect the app and the server side to be prepared to handle the load. Any recommendations on preferred services, alternative platforms, and best practices would be greatly appreciated. Thank you in advance. 

• Answer:

  Ensure Application Layer Scalability From The Start On the backend, I would go ahead and utilize VMs to create several "small" servers instead of one big one. Ensure your application code can run on two separate application VMs. This will essentially prove that the application layer is scalable and will allow you to scale the app and database layer independently. Tune The Platform I would focus on making things fast across the board by tuning your infrastructure first. Read tuning docs for your platform / language of choice. Even simple stress tests along with simple systems-level measurement tools like 'vmstat' can reveal performance bottlenecks. Figure out the best practices for high performance on whatever application platform you're working with, and get that out of the way. Use Proven Technology NoSQL and other hot, more "experimental," tech designed for exotic scaling is very tempting, but not proven. Use well-documented and well-understood technology. You'll be able to get out of a sticky situation much quicker and find help when you need it. In addition, if you can, leverage existing infrastructure that's already been proven to scale, like some of the services provided by Amazon Web Services or Google App Engine. Always Measure Don't assume things are faster because some guy on Quora says it is ;), have ways of measuring it. Even if they are simple and crude, any measurement is several orders of magnitude better than shooting in the dark. Caching is Tempting, but Hard. Use Sparingly I would stay away from caching unless the implementation is extremely simple. Cache invalidation can be hard and complex, and often you won't realize exactly what you need to cache until you've got hard performance data from the real-world. If you cache the wrong thing or do not invalidate things properly, things can really get messed up. You open yourself up for things like "cache stampedes" that can suddenly overload your database. If you've got a dynamic page that you know will get a ton of traffic and looks the same for everyone, like a home page, go ahead and cache it. Use something simple like a background job that runs, say, every five seconds, to update the cached copy of the page. Of couse, as soon as you personalize any page, caching becomes much harder. Uploading Sucks Most web platforms are terrible at uploading. Most have crude implementations that will tie up an entire worker (think 50-250MB RAM, depending on platform) while the upload is taking place. Considering this is going to be a core feature of your application, I'd make sure you can handle thousands of concurrent uploads without crashing and burning. Nginx can buffer uploads to disk while they are taking place, and then flush the entire upload to your application once it's complete. This means your application code is only actually running while Nginx is transferring the upload to it (happens at hundreds or thousands of megabytes per second), versus while the client is streaming it (happens at kilobytes per second). Files Don't Belong In The Database As far as the database, definitely do not store files in MySQL. As others have said, use a CDN like S3. It will cut out a huge amount of issues you may have by removing an entire class of scaling problems, allowing you to focus on other things. The Database Will Likely Be Your Bottleneck I prefer PostgreSQL, but if you must, use at least 5.4 and always use the InnoDB storage engine. If you expect to have lots of writes (hundreds of INSERTs per second peak) and can afford to lose a few transactions if the database crashes (by few I mean less than a seconds worth of transactions worst case), make sure you set innodb_flush_log_at_trx_commit to 0 and sync_binlog to 10. This allows the database to perform disk writes in a more efficient manner, by batching them and performing them once per second. It also allows a synchronous client (like PHP, Rails, etc) to continue operating without waiting for the physical disk write to complete. In most free/consumer applications, losing a few transactions isn't going to impact you much. Try to avoid master/slave replication for scaling as long as possible, as it introduces relaxed consistency guarantees that are often difficult to predict and can cause coding problems. Use as few indexes as you can get away with. Don't put sessions in your database. Instead, store sessions using an encrypted string that the client stores in a cookie or use an in-memory store like memcached.