What mathematical background is needed to understand game theory, algorithmic game theory, and mechanism design?

It depends substantially on what your intended use of these areas is.  Some formal training in probability and game theory is likely to be helpful but it need not be very advanced. An undergraduate course in game theory is more than sufficient to understand the principles at work behind many of the mechanisms you'd frequently encounter in various major private sector applications (such as ads auctions and buyer/seller feedback mechanisms).  Having done both coursework and practical work, I've found that the hours spent thinking about the actual implementation of those mechanisms or perhaps studying a few very specific canonical research papers (which whoever you are working with in your field should be able to point you to) makes you much more able to have a practical impact (make improvements to those mechanisms, optimize your own play in such a game, teach others about the mechanism, etc.) than additional coursework.  There are a few principles that come up over and over again, but being able to relate to real-world examples of these principles is more useful than being able to do deep mathematical derivations. Similarly, having a high intuitive comfort with basic probability will make your life much easier in learning about mechanism design applications, but this is more about your comfort with thinking about relatively basic concepts (e.g. independence of events, expected value calculations) than exposure to more complex concepts.  If you are hoping to study the field at a deep mathematical level or contribute to the literature, however, advanced work in real analysis, statistics and probability would be useful.

Real analysis, advanced probability, and some linear algebra and differential equations is more than enough. If you're looking for grad school, those classes are recommended. Computer skills are nice for modeling or running simulations.

I would like to elaborate the answers provided by and . Real analysis Most of the theoretical results in this area, for example Nash's theorem, deal with extremely general class of games. An undergraduate course in real analysis is usually sufficient to understand most of the them. http://store.doverpublications.com/0486612260.html is a nice book to follow. 2. (Convex) Optimization Every once in a while you would encounter general models of utility functions of the players, and a good knowledge of convex analysis, optimization and duality comes in handy while analyzing the equilibrium concepts. A good course in linear algebra and linear programming is often necessary as well.  http://www.stanford.edu/~boyd/cvxbook/ is a fantastic text book in the subject, and it is also available online. 3. Probability As discussed already, probability theory is quite important. A measure theoretic introduction to probabilities also helps. Other than these three, the mathematical background necessary would depend on the nature problem you are trying to study. 4. (Approximation) Algorithms This is very useful for problems in mechanism design and while trying to quantify inefficiencies of equilibrium. Prof. Jason Hartline's notes in the subject (http://users.eecs.northwestern.edu/~hartline/amd.pdf) give a good introduction to the subject. 5. Differential Equations, Non-linear dynamics If you are interested in studying different learning algorithms and convergence of strategies to equilibrium, here is an excellent text http://www.ssc.wisc.edu/~whs/book/ in the subject.