Help me make tiny windows.

I always used saran wrap or waxed paper, myself. Very 'old-skool' but it worked :D .

That should be "and I don't want to". Time for more coffee.

Maybe what you're thinking of is sugar glass like what is used in movies for stunts? There are recipes online for make-your-own stunt glass, but it is sugar-based so they don't last very long.

You could get some small cheap picture frames at a craft store.

You can get small sheets of Mica at American Science and Surplus. They're like, 2x2 inches.

No information on the site or product you're looking for, but when making architectural models in school, we'd use acetate sheets (at least that's what we called them) - basically just transparency film for overhead projectors that you used to be able to get in any office supply store. If you get a heavy enough gauge, like 1/32" thick or so, they'll be fairly rigid. We'd normally frost the sheets by sanding them a little bit, but you can just leave them clear. The only downside to doing so is that glue shows up a little more, so you have to be careful, but it also helps to use a white glue (like Elmer's), which will pretty much dry clear. For mullions or frames, we'd use 1/32" basswood strips, which are available at craft or art stores.

Maybe those "shrink art plastic" sheets they make for inkjet printers? They come in clear matte as well as white. They would probably work better for stained glass windows, though, as the transparency isn't the greatest.

And http://www.garieinternational.com.sg/clay/shop/liquid_polymer2.htm of some transparent/translucent "polymer clay" liquid gels. Not marketed for dollhouses, as far as I can tell, but seems to be similar to what you're looking for.

http://www.deluxematerials.com/products_miniaturist.html a page from Deluxe Materials which, about half way down describes a liquid resin product called "Solid Water" that can be used to make clear lakes or windows.