What is the best IP-based document sharing solution for small groups?

What about Basecamp? They have a document management solution built-in.

subversion or a wiki.

Integrated with the OS is going to be tough to find. Basecamp or wikis don't do that at all. I've never used KnowledgeTree, but the screenshots don't show explorer integration. My imperfect suggestion is subversion, with http://tortoisesvn.tigris.org/ on windows. I'm sure there's a similar client for OSX. Subversion requires a little bit of retraining and diligence to get right, you really need to be religious about checking out before you commit, but that will solve your problems.

I highly recommend http://www.atlassian.com/software/confluence/, which is web (intranet) based, but is as usable as any of the desktop apps I have ever seen, and also offers wiki-like functionality which my teams have found invaluable. Honestly, if you're going to spend time and money on a desktop/fileshare based system, it will all be wasted. Much of the web based stuff is already well beyond the fileshare based apps from a usability perspective, and they will be dead technologies in a few years.

What OS does this need to integrate with. What desktop apps are you using? The automatic merging of concurrent edits is something that is unlikely to work well for anything other than text documents unless their is a file-format aware merge tool involved, and even then, will require human intervention if there is a collision. If you are a Microsoft shop, look into hosted SharePoint Team Server (or whatever it's called now). It lets you do things like host a spreadsheet like list on the server, but let people work with it in Excel and sync the changes back on a row-by row basis. I think there are similar collaboration aids for the other Office apps. Other than that, I don't think you'll have much luck unless you loosen your requirements even further.

I have had good luck for non-tech users with Novell NetDrive. You can't get it from Novell directly because licensing issues prevent them from distributing it, but you can get it elsewhere; see http://www.engadget.com/2005/10/25/how-to-map-a-drive-to-your-ftp-server/ on Engadget for details. The upshot is that NetDrive lets Windows users map FTP shares and such as network drives, then you can use your ISP's hosted FTP space to store the files. The downside is that FTP is quite insecure, so this might not have the security you need. If only NetDrive supported SCP, it would be ideal for you (assuming you're exclusively Windows, of course.)

NetDrive also supports WebDav, if I recall correctly, including WebDAV over HTTPS, and does a better job of it than the built in windows WebDAV client (or one of them, I think there are actually two different implementations) Still doesn't solve the versioning and merge requirement. Subversion can autocommit for regular WebDAV clients, but you'll still have merge issues AND the problem with the way most desktop apps write lots of temp files, write, copy & rename, and all that jazz, which will make the version control hideous to navigate, unless NetDrive buffers and coalesces a lot of those operations.

Perhaps a hosted http://www.microsoft.com/sharepoint/default.mspx service? I'm not how well if it plays well with OS X (probably not very well). But it handles collaboration and versioning nicely (using the document library feature) and lets you use your desktop apps. I'd go with Google Docs, myself.