How to cache Web API response?

What is the ideal architecture for an API that will need to handle thousands of requests per a second?

• The goal is to: - Reduce development time (i.e use pre-existing code and open source solutions) - Be able to develop on, and change the code easily should you hit any scaling issues - Be able to distribute requests This API would also be different from standard web services API's, i.e Twitters API is doing a query on just their DB of info. However this API would need to accept requests and then be a client to other systems. Some of these systems do not necessarily even have a rest API, and you may have to communicate with them in different protocols, also communicating with as much as 10 or more downstream systems depending on the request. The network architecture to the systems could also get quite complicated with several mediators/middleware/control points in between. So you want to ensure reliability and a relatively quick response. - What kind of technology stack would you use? - What existing servers/frameworks/libraries exist that can be used? - i.e Would you go with something like tornado/twisted web-server to handle API requests? and then some kind of middleware to handle requests to down stream systems? Any or all info would be useful! Thanks!

• Answer:

  The server(s) that accepts the API requests cannot be synchronous in request processing since you can serve only a few hundred to a couple of thousand requests per second with such an architecture. (Even if your API servers are fronted by a load balancer, the per-server request throughput still suffers with a synchronous processing.)  So, you should look at some form of asynchronous architecture. This is especially important for cases where your server acts as a client since you don't want to wait on the request until you get the data from the downstream services. From your description, it looks like Node.js is an ideal platform to base your architecture on. However, it is still in early development, so there is some risk involved. I have used Jetty with continuation support previously (async servlets) with much higher concurrency than deploying the same (well, not the same set of servlets; you get the idea) on Tomcat. So, if you are on Java, this is certainly something you should consider. Jetty is pretty stable and powers Google App Engine (although like many other, Google would have modified it to suit their needs, but the basic architecture is pretty good) so it is production ready. I have not used Tornado, but if you are on Python, you  should evaluate that. It is one of the so called, C10K servers (http://en.wikipedia.org/wiki/C10k_problem), so it should be good. I would personally evaluate node.js and do a risk assessment before embarking on any other because of its architectural elegance. You should also consider your servers being stateless. So, any API request can be handled by any available server without any affinity. Typically, user sessions are pinned to a server so it can be cached and so on, but that would work negatively when you need to support a large number of concurrent sessions.