Why are the orbits of planets in the Solar System nearly circular?

Just wanted to supplement the answers already posted with a few notes re: exoplanet eccentricity. In my understanding, the reason why exoplanets have a median eccentricity ~0.3 vs. almost circular orbits in the solar system is not quite satisfactorily explained just yet (http://adsabs.harvard.edu/abs/2008ApJ...686..603J is still my favorite simulation that attempts to address the origin of eccentricities). 1) The relative importance of disk-planet and planet-planet interactions is not understood completely. It is not clear whether Type I/Type II migration have an important role in shaping the final configuration at all. This is indicated by the number of planetary systems with large inclination wrt the star's equatorial plane discovered recently. 2) There is probably some degree of bias in e-Np histograms, given that there is a degeneracy between a single-planet eccentric solutions and multiple-planets circular solutions to radial velocity observations. See, e.g., http://adsabs.harvard.edu/abs/2010ApJ...709..168A and others: finding that (1) around 35% of the published eccentric one-planet solutions are > statistically indistinguishable from planetary systems in 2:1 orbital resonance, (2) another 40% cannot be statistically distinguished from a circular orbital solution, and (3) planets with masses comparable to Earth could be hidden in known orbital solutions of eccentric super-Earths and Neptune mass planets. 3) Planetesimals and resonance crossing have played an important role at some point in the history of the solar system re: the eccentricity evolution of Jupiter & Saturn (a http://adsabs.harvard.edu/abs/2010ApJ...716.1323B). Why this led to low eccentricities in the solar system might reside in the relative configuration and mass ratio of J & S, the specific mass in planetesimals in the disk compared to other protoplanetary disks, etc. 4) Large eccentricities, especially with small planets, tend to lead to instability and scattering, so invoking the anthropic principle to some degree is not totally unjustified...

A short answer is that dissipation (e.g. dust, gas interactions with planetessimals) is good at removing energy from a system, but not angular momentum. Circular orbits have the minimum energy for a given angular momentum. For short-period exoplanets, the primary form of dissipation is tidal forces of the star on the planet (similarly, the moon is on a nearly circular orbit around the earth, although it certainly didn't start that way!)

This property is likely shared by every other planetary system belonging to the same class as ours. Anthropically speaking, orbits with higher ellipticities will have more extreme environments and thus will be less likely to harbor life. As for the physical reason, I'd guess you could come up with a thermodynamic argument. Intuitively, an elliptical orbit seems to be further from equilibrium than a circular one. A collection of bodies orbiting a star, with high ellipticities initially will then relax to a state where the orbits are less and less elliptical. Of course, this is all heuristic. Edit: In response to @Marek's and @MSalters comments let me add some clarification. When a planet with some eccentricity $e$ interacts with a transient object (asteriod etc.) which process is thermodynamically more likely: that the planet eccentricity increases or that it decreases as a result of the interaction? The LRL vector is a conserved quantity but not when the object is in a noisy environment. The interactions between the planets themselves are also greater when they have orbits with high $e$. As they slowly lose $e$ by scattering and damping events, the mutual interaction will decrease as the orbits become more circular.

So far, this is an unknown question that is the subject of current research. For example: Sean N. Raymond, David P. O'Brien, Alessandro Morbidelli, Nathan A. Kaib, "Building the Terrestrial Planets: Constrained Accretion in the Inner Solar System" To date, no accretion model has succeeded in reproducing all observed constraints in the inner Solar System. These constraints include 1) the orbits, in particular the small eccentricities, and 2) the masses of the terrestrial planets -- Mars' relatively small mass in particular has not been adequately reproduced in previous simulations; 3) the formation timescales of Earth and Mars, as interpreted from Hf/W isotopes; 4) the bulk structure of the asteroid belt, in particular the lack of an imprint of planetary embryo-sized objects; and 5) Earth's relatively large water content, assuming that it was delivered in the form of water-rich primitive asteroidal material. http://arxiv.org/abs/0905.3750

The short answer is tides. Maybe there're more shaping effects for the sun, but at least that's the primary effect shaping moon's orbit to be circular. The thing is that the tidal torque on the sattelite has a strong dependence on the distance. Secondly, the orbital mechanics motion has an interesting property: if you give a bump to an orbitting body it will still pass thru the same point (but it won't get as far out). This is, qualitatively, how tides slowly shape off outer parts of the orbit. However, the effect would be the same for other types of friction strongly depending on the distance, for example friction between the bodies' athmospheres.