Is it possible to install Virtual machine inside a virtual machine using virtual box? aka nesting of virtual machines!

I cannot see an advantage in doing this from a performance perspective.  You're adding a layer of abstraction that comes at a non-zero cost to performance and memory (less visible in disk and network I/O as they operate at very different performance levels than memory and CPU -- the extra latency may go unnoticed.).  It may be useful for studying how "things" work in loosely coupled parallel environments without the overhead of adding more physical machines -- but only as a way to get around some limit on the number of virtual machines a particular virtualization environment supports. As for whether it is possible in virtualbox or other virtualization platforms -- I haven't tried it, but as other answers note, some platforms want to make use of hardware virtualization support -- and don't provide an abstraction of these features in the virtualized "hardware representation". These kinds of things, even when "possible", rarely turn out well for any situation other than an intellectual curiosity -- just because something "can" be done doesn't mean it "should" be done.

Yes, you can nest a VM inside a VM and there are lots of interesting use cases for doing this. Wikipedia describes the concept of http://en.wikipedia.org/wiki/Nested_virtualization as the act of creating an additional layer of abstraction and encapsulation in a pre-existing virtualized environment. While nesting a VM in the same hypervisor is less interesting, the ability to nest an existing VM in a different hypervisor is more useful because of the inherent differences in image formats across hypervisors. Conventional hypervisors such as http://en.wikipedia.org/wiki/VMware_ESX, http://en.wikipedia.org/wiki/Kernel_Virtual_Machine and http://en.wikipedia.org/wiki/Xen are designed to run on physical x86 hardware and use virtualization extensions offered by modern CPUs (Intel VT and AMD SVM) to virtualize the Intel architecture. On the other hand a nested hypervisor runs inside a virtual machine, where these hardware extensions are not normally available. This can cause some performance overhead which minimizes the commercial use of conventional hypervisors in a nested mode. Commercially available high performance nested hypervisors are built ground-up to be run nested in the cloud. For example, the team behind the popular KVM hypervisor commercialized http://www.ravellosystems.com/technology, a high performance nested hypervisor at http://www.ravellosystems.com/ in the year 2013.  The main advantage of nesting is the fact that the environments look exactly the same and no cloud migration is required in order to start using the public cloud for dev & test environments although the production is running in the data center. This is due to the additional layer of abstraction. The other advantage is the ability to use the public cloud for solutions that require the running of actual hypervisors such as KVM or ESXi for dev, test or training environments as described in http://www.ravellosystems.com/blog/multi-node-openstack-rdo-icehouse-aws-ec2-google/. This requires exposing the VT and SVM capabilities in the cloud.

You certainly _can_ do this with Xen(probably other hypervisors too) running Linux but the further away from the actual hardware the less predictable and efficient the performance becomes.  I specify Linux because you have to use the kernel versions that know it(the VM itself) is a virtual machine and that the CPU hooks for virtualization aren't there.   I did this several years ago just to try it and I think i got 4 deep (dom0(domU(domU(domU(domU)))) before the whole thing became non-responsive.  Other than playing I can't see any reason to do this on the current technology.