Are There Any Databases Of Digitalised Natural Stimuli On Milliseconds Scale?

What does it take to scale Django (web framework)?

• Question asked on Stackoverflow (http://stackoverflow.com/questions/22412768/what-does-it-take-to-scale-django/22415907?noredirect=1#22415907) but that community's prudish response to my question sent me here. Description below. ------------------------------ Problem So I've been Django-ing for a number of months*. I find myself in a position where, I'm able to code up a Django web app for whatever, but am terrified by my inability** to come up with solutions as to how to go about building a Django web app for a large (LARGE) audience. Good to know that Django scales, at least. How I'm thinking about it It seems like there would need to be a relatively large leap of knowledge to understand how to (let alone actually execute) scale a Django web app. I say this because my research has given me the impression that scaling (or, enabling scalability) is a process of fitting aftermarket solutions to the different components of your web app to enhance the performance of each of these components. There'sjustsomanythings~~ So there's a ton of solutions, and a bunch ofcomponents. For instance, there's Elastic Beanstalk for hosting, Django's cache framework, Memcached and Varnish forcaching, Cassandra, Redis and PostgreSQLfor databases, and uWSGI, Nginx andApache for deployment. If what I think is right, anyway. I'm still not sure. What I Need I crave that amazing response that becomes the canonical answer to the question, but would also appreciate leads on where to begin, or suggestions of an approach to take to solve the problem, or your approach to scale Django. Thank you in advance for your been-there-done-that words of wisdom. <<Edit: SO disapproves :( BRAND NEW & EXCLUSIVE: A QUESTION FOR STACKOVERFLOW! What I need What are the 3 most important/effective things I should do/implement to improve the preparedness for scaling of the Django web apps that I'm building? List the approach, and explaining how they help would be nice. ------------------------------ *I've been cheating. I deploy on Pythonanywhere and have only used Sqlite3 up till now. I have also managed to keep my hands clean of WSGI/Apache deployment stuff to date. **With Django is when I first managed to create something of value through programming. Before, I had only used Pascal to cheat at Runescape and Java to make some shitty Android apps. Which could perhaps explain why I feel this is that large of a leap.

• Answer:

  Every app scales differently.  It really depends on how you grow the app, what the various inputs are (auth/unauth, locales, etc). For example, if you have a read-heavy site, you can get a long way with caching.  You can even cache at the Nginx layer. If you are doing a lot of DB transactions, you'll find sqlite3 will not serve you well.  You'll probably want to use Postgres or MySQL (whichever you are more comfortable with). If you can separate your read traffic from write traffic that will help you since you can aggressively cache read traffic. None of this is unique to Django.  Django is flexible and gives you options, but the options aren't that different from other frameworks. So the real answer to scaling is this - stress test your site.  Find out what makes it break, and what it would take to fix that break.  If alleviating that stress can be done simply by making multiple instances (e.g. launching more Django frontend boxes), then you have a relatively scalable site.  If you'll need to re-architect things (e.g. add cache, switch the underlying database, write special logic) your site is less scalable. So if I was in your position, I'd try to play a lot of traffic concurrently, and then gradually dial the concurrency up.  At some point (N queries/second) your site will break (it's inevitable, you are DDOSing yourself).  Then figure out what changes you need to make to go to 2N or even 10N.  You want to get to a point where you can watch traffic grow, and respond to the growth by adding more servers.  E.g. if a single box can serve 10qps, and you need 20qps - build an architecture that can do this, simply by doubling the boxes. This is certainly oversimplifying things, but once you've done this a few times, you'll get the hang of it.  It won't always be "easy," but you should be able to make some headway.