What's the smallest open OS that I can compile from source and run in a virtual environment?

I know it's not the smallest OS, but you might want to check this out, just for fun... http://www.sparrowos.com/

You can also try http://www.linuxfromscratch.org/lfs/ "Linux From Scratch (LFS) is a project that provides you with step-by-step instructions for building your own customized Linux system entirely from source." I have tried this while I'm on college for the same purpose. But it is not tiny as u expect.

You can try looking at MIT's xv6:http://%20http://pdos.csail.mit.edu/6.828/2011/xv6.html Xv6 is a teaching operating system developed in the summer of 2006 for MIT's operating systems course, http://pdos.csail.mit.edu/6.828. We hope that xv6 will be useful in other courses too.  This page collects resources to aid the use of xv6 in other courses, including a commentary on the source code itself. It's also comes with very nice booklet: http://pdos.csail.mit.edu/6.828/2012/xv6/book-rev7.pdf which explains every OS subsystem.

Not Another Completely Heuristic Operating System, or Nachos, is instructional software for teaching undergraduate, and potentially graduate level operating systems courses. It was developed at the http://en.wikipedia.org/wiki/University_of_California,_Berkeley, designed by Prof. Thomas Anderson, and is used by numerous schools. Originally written in http://en.wikipedia.org/wiki/C%2B%2B for http://en.wikipedia.org/wiki/MIPS_architecture, Nachos runs as a user-process on a host operating system. A MIPS simulator executes the code for any user programs running on top of the Nachos operating system. Ports of the Nachos code exist for a variety of architectures. http://homes.cs.washington.edu/~tom/nachos/ OS design book based on nachos:  http://www.amazon.com/Operating-Systems-Principles-Thomas-Anderson/dp/0985673516

The Baremetal OS : http://www.returninfinity.com/baremetal.html This one is written only in assembly language, have very small size comes as a virtual machine image as well. Very good for educational use. You will get lot many in this thread http://programmers.stackexchange.com/questions/117973/are-there-any-os-which-are-simple-enough-for-learning

Project XOS or eXperimental Operating System is a platform to develop a toy operating system. It is an instructional tool for students to learn and implement OS data structures and functionalities on a simulated machine called XSM (eXperimental String Machine). The OS is programmed using a custom language known as SPL (System Programmer's Language) and application programs, which run on the OS, are programmed using APL (Application Programmer's Language).  Link : http://xosnitc.github.io/index.html