developing a new document format: how to start?

Without more details about your goals and requirements, you might want to start by looking at ebook format standards, such as the http://idpf.org/epub/30/spec/epub30-overview.html. You could also take a look at some of the metadata formats from the library world, like http://dublincore.org/ to get a sense of what descriptors you would like your document format to support. Finally, there are several projects on Github to support ebook generation from Markdown, which may not be what you need, but may speak to what sorts of author tools you'd like to create. In that spirit, check out https://leanpub.com/. Finally, although you've already said the format doesn't exist, consider whether you need to create a new format vs using the tools you've already got (html, css, js). Since we don't know your requirements, you're best suited to answer the question, but creating a new format is often most valuable when you're looking to serve a particular community of users or creators. In that case, do they need a format, or a new set of tools to use an existing format? Good luck!

Without knowing more about what you want to put together, you might also look at http://en.wikipedia.org/wiki/Sharable_Content_Object_Reference_Model and the http://en.wikipedia.org/wiki/Tin_Can_API, both of which were developed for distributing e-learning.

The trend seems to be to take the components of your document and put 'em in a .zip file. You can use zip to explore formats like .odt and .kmz, and start from there. When you talk about file formats, you're basically asking "what's a reasonable way to serialize this data structure?" So develop the data structure first. Given that, as heliostatic mentions, you're talking about developing this with HTML and JavaScript and CSS, it could be as simple as ZIPping up the portions that vary. And the portions that don't vary may be small enough that just ZIPping up the whole darned thing could also be easy enough.

I would recommend checking out thehttp://jats.niso.org. As much as you think that what you are doing has never been done before, you could probably learn quite a bit from the work that NCBI has been doing for the last decade or so, since they are basically doing very similar things., and it is a very active community doing cool things.

So develop the data structure first. Given that, as heliostatic mentions, you're talking about developing this with HTML and JavaScript and CSS, it could be as simple as ZIPping up the portions that vary. Yes, I had thought of that as a first step: the container as a zip file with a standard file structure. The requirements go beyond that, but the links everyone has given me will be very helpful.

Take a look at browser plugins as they are basically zipped assets based around html/js.

If you want to use this data format then you are better off building on existing open formats - many standard container formats have been designed specifically to be extended. Extending an existing format will allow you to leverage standard tools and allow others to easily interact with your format. If you want to design a new format - then start from scratch and learn while building. As you optimize however keep checking back with the standard open formats to understand how others have solved similar problems - if you have a unique solution propagate it back to the standard solutions...

Yeah, sounds like you're reinventing SGML.